US20160301912A1 - Disparity value deriving device, movable apparatus, robot, disparity value producing method, and computer program - Google Patents
Disparity value deriving device, movable apparatus, robot, disparity value producing method, and computer program Download PDFInfo
- Publication number
- US20160301912A1 US20160301912A1 US15/100,895 US201415100895A US2016301912A1 US 20160301912 A1 US20160301912 A1 US 20160301912A1 US 201415100895 A US201415100895 A US 201415100895A US 2016301912 A1 US2016301912 A1 US 2016301912A1
- Authority
- US
- United States
- Prior art keywords
- matching degree
- disparity value
- threshold
- cost
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/128—Adjusting depth or disparity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
- G06T7/593—Depth or shape recovery from multiple images from stereo images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
-
- H04N13/0022—
-
- G06K9/46—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/60—Editing figures and text; Combining figures or text
-
- G06T7/0044—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
- G06T7/557—Depth or shape recovery from multiple images from light fields, e.g. from plenoptic cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
- G06T7/74—Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/271—Image signal generators wherein the generated image signals comprise depth maps or disparity maps
-
- G06K2009/4666—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30252—Vehicle exterior; Vicinity of vehicle
- G06T2207/30261—Obstacle
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N2013/0074—Stereoscopic image analysis
- H04N2013/0081—Depth or disparity estimation from stereoscopic image signals
Definitions
- the present invention relates to a disparity value deriving device, a movable apparatus, a robot, a disparity value producing method, and a computer program.
- a distant measurement method is conventionally known, in which disparity for an object is derived from a stereo camera by stereo imaging and a disparity value indicating this disparity is used to measure the distance from the stereo camera to the object based on the principle of triangulation.
- this distance measurement method for example, the distance between automobiles or the distance between an automobile and an obstacle is measured to be utilized for preventing automobile collisions.
- a stereo matching process is used for obtaining a disparity value.
- a disparity value between a reference image obtained by one of two cameras of a stereo camera and a comparison image obtained by the other camera is calculated by obtaining the position of a corresponding pixel where image signals are most similar, while successively shifting a plurality of candidates for the corresponding pixel in the comparison image relative to a reference pixel of interest in the reference image.
- the luminance values of the image signals obtained by two cameras are compared whereby the position of the pixel with the lowest one of costs (here, cost is “dissimilarity”) in terms of luminance value compared among shift amounts is obtained (see Japanese Patent Application Laid-open No. 2006-090896).
- a method of deriving more accurate disparity even for an object with weak texture is proposed (see, for example, Japanese Patent Application Laid-open No. 2012-181142). With this method, not only the cost of a reference pixel of interest in a reference image but also the costs of other pixels existing near to far positions from the reference pixel are aggregated to derive disparity for an object with weak texture.
- a disparity value deriving device for deriving a disparity value indicating the disparity of an object from a reference image obtained by a first imaging unit capturing of an image of the object and a comparison image obtained by a second imaging unit capturing another image of the object.
- the disparity value deriving device includes a calculator configured to calculate costs of candidates for a corresponding region in the comparison image that corresponds to a reference region in the reference image, based on a luminance value of the reference region and luminance values of the candidates for the corresponding region, the candidates for the corresponding region being specified in a way that the respective candidates for the corresponding region are shifted on an epipolar line based on the reference region in the comparison image by individual shift amounts; a changer configured to change a cost exceeding a threshold among the costs calculated by the calculator to a given value higher than the threshold; a synthesizer configured to synthesize, with respect to each of the shift amounts, a cost of a candidate for a corresponding region that corresponds to one reference region after change by the changer and a cost of a candidate for a corresponding region that corresponds to another reference region after change by the changer; and a deriving unit configured to derive the disparity value based on a position of the one reference region in the reference image and a position of the corresponding region in
- FIG. 1 is an illustration of the principle of deriving the distance from an imaging device to an object.
- FIG. 2 illustrates (a) an example of a reference image, (b) an example of a high density disparity image, and (c) an example of an edge disparity image.
- FIG. 3 illustrates a calculation of a shift amount while successively shifting candidates for a corresponding pixel in a comparison image relative to a reference image.
- FIG. 4 is a graph illustrating costs with respect to shift amounts.
- FIG. 5 is a conceptual diagram for deriving a synthesized cost.
- FIG. 6 is a graph illustrating synthesized costs with respect to disparity values.
- FIG. 7 illustrates an automobile equipped with a disparity value deriving device according to an embodiment of the present invention.
- FIG. 8 is a schematic diagram of the disparity value deriving device.
- FIG. 9 is a hardware configuration diagram of the disparity value deriving device as a whole.
- FIG. 10 is a hardware configuration diagram of the principal part of the disparity value deriving device.
- FIG. 11 is a flowchart illustrating the processing in a first embodiment.
- FIG. 13 illustrates a graph of costs without cost change, a graph of costs with cost change, a graph of synthesized costs without cost change, and a graph of synthesized costs with cost change.
- FIG. 14 is a conceptual diagram of a reference image including a frame indicating the size of an object recognition result recognized by an object recognition device.
- FIG. 15 is a flowchart illustrating the processing in a second embodiment.
- FIG. 16 illustrates examples of a reference image, a part of the reference image, a high density disparity image without cost change, and high density disparity images with cost change.
- FIG. 17 illustrates a graph of costs without cost change, a graph of costs with cost change, a graph of synthesized costs without cost change, and a graph of synthesized costs with cost change.
- FIG. 18 illustrates a graph identical with (b) of FIG. 17 , a graph of costs with cost change, a graph identical with (d) of FIG. 17 , and a graph of synthesized costs with cost change.
- SGM semi-global matching
- FIG. 1 is an illustration of the principle of deriving the distance from imaging devices to an object. The following description explains processes with respect to each single pixel rather than with respect to each predetermined region including a plurality of pixels, to make the explanation simpler.
- the predetermined region that includes a reference pixel is denoted as a reference region
- the predetermined region that includes a corresponding pixel is denoted as a corresponding region.
- the reference region may include a reference pixel alone and the corresponding region may include a corresponding pixel alone.
- the images captured by an imaging device 10 a and an imaging device 10 b illustrated in FIG. 1 are denoted as a reference image Ia and a comparison image Ib, respectively.
- the imaging device 10 a and the imaging device 10 b are installed parallel and at the same height.
- a point S on an object E in a three-dimensional space is captured at one position of the imaging device 10 a and also at one position of the imaging device 10 b , wherein these two positions are on the same horizontal line of the imaging device 10 a and the imaging device 10 b . That is, the point S in each image is captured at a point Sa(x,y) in the reference image Ia and a point Sb(x,y) in the comparison image Ib.
- the disparity value ⁇ is expressed as Equation (1) using Sa(x,y) in the coordinates of the imaging device 10 a and Sb(X,y) in the coordinates of the imaging device 10 b.
- the distance Z from the imaging devices 10 a , 10 b to the object E can be derived using the disparity value ⁇ .
- the distance Z is the distance from the plane including the focus position of the imaging lens 11 a and the focus position of the imaging lens 11 b to a particular point S on the object E.
- the distance Z can be calculated by Equation (2) using the focal length f of the imaging lens 11 a and the imaging lens 11 b , the base line length B that is the length between the imaging lens 11 a and the imaging lens 11 b , and the disparity value ⁇ .
- FIG. 2( a ) is a conceptual diagram illustrating a reference image
- FIG. 2( b ) is a conceptual diagram illustrating a high density disparity image for FIG. 2( a )
- FIG. 2( c ) is a conceptual diagram illustrating an edge disparity image for FIG. 2( a )
- the reference image is an image representing an object by luminance
- the high density disparity image is an image that is derived from the reference image by the SGM method and represents the disparity value at each set of coordinates in the reference image.
- the edge disparity image is an image that is derived by the conventionally used block matching method and represents the disparity values only at a part with relatively strong texture such as an edge in the reference image.
- the SGM method is a method of deriving the disparity values appropriately even for an object with weak texture and deriving the high density disparity image illustrated in FIG. 2( b ) based on the reference image illustrated in FIG. 2( a ) .
- the edge disparity image illustrated in FIG. 2( c ) is derived based on the reference image illustrated in FIG. 2( a ) .
- the high density disparity image can represent detailed information such as a road with weak texture and therefore enables distance measurement in more detail.
- a disparity value is derived by calculating a cost and thereafter further calculating a synthesized cost that is synthesized dissimilarity, rather than deriving a disparity value immediately after calculating a cost that is dissimilarity.
- a disparity image here, high density disparity image
- the block matching method is the same as the SGM method in that a cost is calculated. However, unlike the SGM method, the disparity values only at a part with relatively strong texture such as an edge are derived without synthesized costs being calculated.
- FIG. 3( a ) is a conceptual diagram illustrating a reference pixel in a reference image
- FIG. 3( b ) is a conceptual diagram of calculating shift amounts (displacement amounts) while successively shifting (displacing) candidates for the corresponding pixel in a comparison image relative to the reference pixel in (a).
- FIG. 4 is a graph illustrating costs with respect to shift amounts.
- the corresponding pixel is a pixel in the comparison image that is the most similar to the reference pixel in the reference image.
- C(p,d) represents C(x,y,d).
- the cost C(p,d) of each candidate corresponding pixel q(x+d,y) that corresponds to the reference pixel p(x,y) is calculated based on the luminance values of a predetermined reference pixel p(x,y) in the reference image and of a plurality of candidate corresponding pixels q(x+d,y) on the epipolar line in the comparison image that corresponds to the reference pixel p(x,y).
- the variable d is a shift amount (displacement amount) of each candidate q for the corresponding pixel from the reference pixel p, and a shift amount is expressed in units of pixels in the present embodiments. That is, in FIG.
- the cost C(p,d) that is dissimilarity in luminance value between a candidate corresponding pixel q(x+d,y) and the reference pixel p(x,y) is calculated while successively shifting the candidate corresponding pixels q(x+d,y) one pixel by one pixel in a predetermined range (for example, 0 ⁇ d ⁇ 25).
- a known method such as SAD (Sum of Absolute Difference) is applied as the method of calculating the cost C where the cost C denotes dissimilarity.
- the cost C(p,d) thus calculated can be expressed by a graph of a cost curve that is a set of the costs C with respect to shift amounts d.
- FIG. 5 is a conceptual diagram for deriving a synthesized cost.
- FIG. 6 is a graph of a synthesized-cost curve illustrating synthesized costs with respect to disparity values.
- the cost C(p,d) in the reference pixel p(x,y) is calculated, and costs for each one of pixels p(x n ,y n ) existing near to far positions from the reference pixel p(x,y) is also calculated as costs C for each one of another reference pixels p(x n ,y n ).
- the synthesized cost Ls(p,d) for the reference pixel p(x,y) is calculated by aggregating the calculated cost C(p,d) for the reference pixel p(x,y) and the calculated costs C for other reference pixels p(x n ,y n ) existing near to far positions from the reference pixel p(x,y).
- the synthesized cost Ls for each one of reference pixels p(x n ,y n ) is calculated by aggregating the calculated cost C for one reference pixel p(x n ,y n ) and the calculated costs C for other reference pixels existing near to far positions from the reference pixel p(x n ,y n ).
- Equation (3) is an equation for calculating the path cost Lr(p,d)
- Equation (4) is an equation for calculating the synthesized cost Ls.
- Lr ( p,d ) C ( p,d )+min ⁇ ( Lr ( p ⁇ r,d ), Lr ( p ⁇ r,d ⁇ 1)+ P 1, Lr ( p ⁇ r,d+ 1)+ P 1, Lrmin( p ⁇ r )+ p 2 ⁇ (3)
- Equation (3) r denotes a direction vector in the aggregation direction and has two components of the x direction and the y direction.
- min ⁇ is a function for obtaining the minimum value.
- Lrmin(p ⁇ r) denotes the minimum value of Lr(p ⁇ r,d) when the shift amount d is changed in the coordinates in which p is shifted by one pixel in r direction. Lr is recurrently applied as expressed in Equation (3).
- Lr(p,d) is obtained by adding the minimum value of the path cost Lr of each pixel in the pixels in r direction illustrated in FIG. 5 to the cost C in the reference pixel p(x,y).
- Lr is obtained first from the endmost pixel in r direction of the reference image p(x,y), and Lr is obtained along r direction.
- Lr 0 , Lr 45 , Lr 90 , Lr 135 , Lr 180 , Lr 225 , Lr 270 , Lr 315 in eight directions are obtained, and the synthesized cost Ls is eventually obtained based on Equation (4).
- the synthesized cost Ls(p,d) thus calculated can be represented by a graph of a synthesized-cost curve representing the synthesized cost Ls(p,d) with respect to the shift amount d, as illustrated in FIG. 6 .
- the number of r is eight in the foregoing description, the number is not limited thereto.
- the eight directions may be further divided by two into 16 directions or by three into 24 directions.
- the cost C may be expressed as “similarity” that is a reciprocal of dissimilarity.
- a known method such as NCC (Normalized Cross Correlation) is applied as the method of calculating the cost C.
- the disparity value ⁇ not with the minimum but with the “maximum” synthesized cost Ls is derived.
- the similarity and the dissimilarity may be inclusively denoted as “matching degree”.
- the object recognition system 1 may be mountable not only on an automobile as an example of a vehicle but also on a motor bicycle, a bicycle, a wheelchair, and an agricultural cultivator as other examples of a vehicle.
- the object recognition system 1 may be mountable not only on a vehicle as an example of a movable apparatus but also on a robot as another example of a movable apparatus.
- the robot may not be a movable apparatus but may be an apparatus such as an industrial robot fixedly installed in FA (Factory Automation).
- the apparatus fixedly installed may not be a robot but may be a security monitoring camera.
- FIG. 7( a ) is a schematic diagram depicting a side of an automobile equipped with the object recognition system according to an embodiment of the present invention
- FIG. 7( b ) is a schematic diagram depicting the front of the automobile.
- FIG. 8 is a schematic diagram of the object recognition system.
- the object recognition system 1 in the present embodiments includes an imaging device 10 a and an imaging device 10 b .
- the imaging device 10 a and the imaging device 10 b are installed so as to be able to image the scene ahead in the direction in which the automobile travels.
- the object recognition system 1 includes a main body 2 and a pair of cylindrical imaging device 10 a and imaging device 10 b provided on the main body 2 .
- FIG. 9 is a hardware configuration diagram of the object recognition system as a whole.
- the object recognition system 1 includes a disparity value deriving device 3 and an object recognition device 5 in the main body 2 .
- the disparity value deriving device 3 derives a disparity value ⁇ indicating disparity for an object E from a plurality of images obtained by imaging the object E and outputs a high density disparity image indicating the disparity value ⁇ in each pixel.
- the object recognition device 5 performs processing such as measuring the distance from the imaging devices 10 a , 10 b to the object E, based on the high density disparity image output from the disparity value deriving device 3 .
- the disparity value deriving device 3 includes the imaging device 10 a , the imaging device 10 b , a signal conversion device 20 a , a signal conversion device 20 b , and an image processing device 30 .
- the imaging device 10 a generates an analog signal representing an image by imaging the scene ahead and includes an imaging lens 11 a , a diaphragm 12 a , and an image sensor 13 a .
- the imaging lens 11 a is an optical element for diffracting light passing through the imaging lens 11 a to form an image of an object.
- the diaphragm 12 a cuts off part of light passing through the imaging lens 11 a to adjust the quantity of light input to the image sensor 13 a described later.
- the image sensor 13 a is a semiconductor device that converts light input from the imaging lens 11 a and the diaphragm 12 a into an electrical analog image signal and is implemented, for example, by a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS).
- the imaging device 10 b has the same configuration as the imaging device 10 a , and a description of the imaging device 10 b is omitted.
- the imaging lens 11 a and the imaging lens 11 b are installed such that their respective lens surfaces are
- the signal conversion device 20 a converts an analog signal representing the captured image into image data in digital format, and includes a correlated double sampling (CDS) 21 a , an auto gain control (AGC) 22 a , an analog digital converter (ADC) 23 a , and a frame memory 24 a .
- the CDS 21 a removes noise by correlated double sampling from the analog image signal converted by the image sensor 13 a .
- the AGC 22 a performs gain control of controlling the intensity of the analog image signal having noise removed by the CDS 21 a .
- the ADC 23 a converts the analog image signal gain-controlled by the AGC 22 a into image data in digital format.
- the frame memory 24 a stores the image data converted by the ADC 23 a.
- the signal conversion device 20 b obtains image data from the analog image signal converted by the imaging device 10 b and includes a CDS 21 b , an AGC 22 b , an ADC 23 b , and a frame memory 24 b .
- the CDS 21 b , the AGC 22 b , the ADC 23 b , and the frame memory 24 b have the same configuration as the CDS 21 a , the AGC 22 a , the ADC 23 a , and the frame memory 24 a , respectively, and a description thereof is omitted.
- the image processing device 30 is a device for processing image data converted by the signal conversion device 20 a and the signal conversion device 20 b .
- the image processing device 30 includes a field programmable gate array (FPGA) 31 , a central processing unit (CPU) 32 , a read only memory (ROM) 33 , a random access memory (RAM) 34 , an interface (I/F) 35 , and a bus line 39 such as an address bus and a data bus for electrically connecting the components 31 to 35 as illustrated in FIG. 9 .
- FPGA field programmable gate array
- CPU central processing unit
- ROM read only memory
- RAM random access memory
- I/F interface
- bus line 39 such as an address bus and a data bus for electrically connecting the components 31 to 35 as illustrated in FIG. 9 .
- the FPGA 31 is an integrated circuit and performs the process of calculating a disparity value ⁇ in the image represented by image data in accordance with an instruction from the CPU 32 .
- the CPU 32 controls each function in the disparity value deriving device 3 .
- the ROM 33 stores an image processing program for the CPU 32 to execute to control each function in the disparity value deriving device 3 .
- the RAM 34 is used as a work area for the CPU 32 .
- the I/F 35 is an interface for communicating with the I/F 55 of the object recognition device 5 described later through the bus line 4 such as an address bus and a data bus.
- the object recognition device 5 includes an FPGA 51 , a CPU 52 , a ROM 53 , a RAM 54 , an I/F 55 , a CAN I/F 58 , and a bus line 59 such as an address bus and a data bus for electrically connecting the components 51 to 55 , 58 as illustrated in FIG. 9 .
- the FPGA 51 , the CPU 52 , the ROM 53 , the RAM 54 , the I/F 55 , and the bus line 59 have the same configuration as the FPGA 31 , the CPU 32 , the ROM 33 , the RAM 34 , the I/F 35 , and the bus line 39 , respectively, in the image processing device 30 and a description thereof is omitted.
- the I/F 55 is an interface for communicating with the I/F 35 in the image processing device 30 through the bus line 4 .
- the ROM 53 stores an object recognition program for the CPU 52 to execute to control each function in the object recognition device 5 .
- the CAN I/F 58 is an interface for communicating with, for example, an external controller and can be connected to, for example, a controller area network (CAN) of the automobile.
- CAN controller area network
- the FPGA 51 calculates the distance Z from the imaging devices 10 a , 10 b to an object E according to an instruction from the CPU 52 in the object recognition device 5 .
- the FPGA 31 may calculate the distance Z under an instruction from the CPU 32 of the image processing device 30 , rather than the FPGA 51 calculating the distance Z under an instruction from the CPU 52 in the object recognition device 5 .
- the programs described above may be recorded in an installable or executable file format on a computer-readable storage medium to be distributed.
- Examples of the storage medium include a compact disc read only memory (CD-ROM) and a secure digital (SD) memory card.
- FIG. 10 is a hardware configuration diagram of the principal part of the disparity value deriving device according to the first embodiment.
- the FPGA 31 in FIG. 9 includes a cost calculator 310 , a cost synthesizer 320 , and a disparity value deriving unit 330 , as illustrated in FIG. 10 . Although these are part of the circuit of the FPGA 31 , the same processing may be performed through execution of the image processing program.
- the cost calculator 310 calculates costs C of candidate corresponding pixels q(x+d,y) that corresponds to the reference pixel p(x,y), based on the luminance value of the reference pixel p(x,y) in the reference image (see FIG. 3( a ) ) and the luminance values of candidate corresponding pixels q(x+d,y) that are shifted on the epipolar line EL based on the reference pixel p(x,y) in the comparison image (see FIG. 3( b ) ) and are specified with respect to individual shift amounts d, as illustrated in FIG. 3 .
- the cost calculator 310 also serves as a cost changer and changes a cost exceeding a threshold Th, among the calculated costs C, to a given value H higher than the threshold Th.
- the cost synthesizer 320 synthesizes, with respect to each shift amount d, a cost C of a candidate corresponding pixel q(x+d,y) that corresponds to a certain reference pixel p(x,y) after the change by the cost calculator 310 (cost changer) and a cost C of a candidate q′(x′+d,y′) for the corresponding pixel that corresponds to another reference pixel p′(x′,y′) after the change by the cost calculator 310 (cost changer), thereby outputting a synthesized cost Ls.
- This synthesizing process is a process of eventually calculating a synthesized cost Ls by calculating a path cost Lr from the cost C based on Equation (3) and thereafter further adding the path costs Lr for the respective directions based on Equation (4).
- the disparity value deriving unit 330 derives the disparity value A based on the position (x,y) in the reference image of a certain reference pixel p(x,y) and the position (x+ ⁇ ,y) of the corresponding pixel q(x+ ⁇ ,y) in the comparison image, for which the synthesized cost Ls after the synthesis by the cost synthesizer 320 is smallest, and outputs a disparity image indicating the disparity value in each pixel.
- FIG. 9 to FIG. 14 the processing or operation of the present embodiment will be described.
- the synthesized cost Ls is derived from the reference image illustrated in FIG. 12( a ) will be described.
- part of the image in FIG. 12( a ) is illustrated in FIG. 12( b ) .
- FIG. 11 is a flowchart illustrating the processing in the present embodiment.
- FIG. 12( a ) is a conceptual diagram illustrating a reference image
- FIG. 12( b ) is a conceptual diagram illustrating a part of the reference image
- FIG. 12( c ) is a conceptual diagram illustrating a high density disparity image for FIG. 12( b ) without cost change
- FIG. 12( a ) is a conceptual diagram illustrating a reference image
- FIG. 12( b ) is a conceptual diagram illustrating a part of the reference image
- FIG. 12( c ) is a conceptual diagram illustrating a high density disparity
- FIG. 13( a ) is a graph illustrating costs without cost change
- FIG. 13( b ) is a graph illustrating costs with cost change
- FIG. 13( c ) is a graph illustrating synthesized cost without cost change
- FIG. 13( d ) illustrates synthesized cost with cost change.
- the vertical axis represents costs C and the horizontal axis represents shift amounts d.
- the vertical axis represents synthesized costs Ls and the horizontal axis represents shift amounts d.
- the imaging device 10 a illustrated in FIG. 9 generates analog image data by imaging an object E (step S 1 - 1 ). Similarly, the imaging device 10 a generates analog image data by imaging the object (step S 1 - 2 ).
- the signal conversion device 20 a then converts the analog image data into digital image data (step S 2 - 1 ). Similarly, the signal conversion device 20 b converts the analog image data into digital image data (step S 2 - 2 ).
- the signal conversion device 20 a then outputs the converted digital image data to the FPGA 31 of the image processing device 30 as data of a reference image (step S 3 - 1 ).
- the conceptual diagram of this reference image is illustrated in FIG. 12( a ) .
- the signal conversion device 20 b outputs the converted digital image data to the FPGA 31 of the image processing device 30 as data of a comparison image (step S 3 - 2 ).
- This comparison image which is an image captured by the imaging device 10 b , has no significant difference from the reference image illustrated in FIG. 12( a ) and therefore a conceptual diagram of the comparison image is omitted without illustration.
- the cost calculator 310 illustrated in FIG. 10 then calculates costs C with respect to individual shift amounts d, as illustrated in the graph in FIG. 13( a ) , based on the data of the reference image and the data of the comparison image (step S 4 ).
- the cost synthesizer 320 generates and outputs a synthesized cost Ls with respect to each shift amount d, as illustrated in the graph in FIG. 13( c ) .
- the high density disparity image in this case is the image as illustrated in FIG. 12( c ) .
- Step S 5 and step S 6 may be processed in the reverse order or may be processed concurrently.
- the cost synthesizer 320 illustrated in FIG. 10 then outputs the synthesized cost Ls with respect to individual shift amounts d, as illustrated in the graph in FIG. 13( d ) (step S 7 ).
- the high density disparity image in this case is a silhouette image as illustrated in FIG. 12( d ) .
- This silhouette image is a schematic image of an object in the reference image recognized by the object recognition device 5 and decreases in size as the threshold Th is lower.
- the threshold Th approaching zero has an advantage of increasing the accuracy of the synthesized cost Ls(p,d) but has a disadvantage of increasing the circuit scale of the FPGA 31 for calculating the synthesized cost Ls(p,d).
- the disparity value ⁇ varies between when the cost calculator 310 does not change the cost and when it changes the cost.
- the high density disparity image (see FIG. 12( d ) ) in the case where the costs are changed is more faithful to the object than the high density disparity image (see FIG. 12( c ) ) in the case where the costs are not changed.
- the CPU 32 then transmits the high density disparity image represented by the disparity values ⁇ from the I/F 35 illustrated in FIG. 9 to the I/F 55 of the object recognition device 5 through the bus line 4 .
- the FPGA 51 then calculates the distance Z from the imaging devices 10 a , 10 b to the object E under an instruction from the CPU 52 in the object recognition device 5 .
- the object recognition device 5 further recognizes a frame indicating the size of the object E in the reference image as illustrated in FIG. 14 .
- FIG. 14 is a conceptual diagram of the reference image including a frame indicating the size of the object recognition result recognized by the object recognition device.
- a frame 1001 (here, denoted by the broken line) is depicted in the reference image in FIG. 14 .
- a frame 1002 (here, denoted by the dashed line) is depicted in the reference image in FIG. 14 .
- the size of the object recognition result may not be indicated by a frame but may be indicated by a longitudinal or oval form, or another form that masks the frame.
- the cost calculator 310 when the cost C calculated by the cost calculator 310 exceeds the threshold Th, the cost calculator 310 changes the cost C exceeding the threshold Th to a first given value H higher than the threshold Th. As a result, the cost C greater than the first given value H is reduced to the first given value H, so that the cost in an object with weak texture is less susceptible to the costs in the neighboring object with strong texture. This processing therefore achieves the advantageous effect of deriving a more accurate cost.
- the cost calculator 310 further changes the cost equal to or smaller than the threshold Th to a second given value, zero, lower than the threshold Th.
- the costs C not narrowed down by the first given value H are changed to the second predetermine value zero. Therefore, the shift amount d can be derived through calculation with only two values, namely, the first given value H and the second given value zero, thereby achieving the advantageous effect of further reducing the circuit scale.
- the calculation with only two values, namely, the first given value H and the second given value zero can reduce the excessive effects brought about by the magnitude of the costs C in calculating the synthesized cost Ls and enables derivation of a disparity value even more faithful to the actual object.
- the hardware configuration of the principal part of the present embodiment is the same as the hardware configuration of the principal part of the first embodiment and a description thereof is omitted.
- FIG. 15 to FIG. 18 the processing or operation of the present embodiment will be described.
- the synthesized cost Ls is derived from the reference image illustrated in FIG. 16( a ) will be described.
- FIG. 16( b ) a part of the image in FIG. 16( a ) is illustrated in FIG. 16( b ) .
- FIG. 15 is a flowchart illustrating the processing in the second example embodiment.
- FIG. 16( a ) is an example of a reference image
- FIG. 16( b ) is a part of the reference image
- FIG. 16( c ) is a conceptual diagram illustrating a high density disparity image for FIG. 16( b ) without cost change
- FIG. 16( d ) is a conceptual diagram illustrating a high density disparity image for FIG. 16( b ) with cost change
- FIG. 17( a ) is a graph illustrating costs without cost change
- FIG. 17( b ) is a graph illustrating costs with cost change
- FIG. 17( c ) is a graph illustrating synthesized costs without cost change
- FIGS. 18( a ) and 18( c ) are the same graphs as FIGS. 17( b ) and 17( d ) , respectively.
- FIGS. 18( a ) and 18( c ) are provided to facilitate comparison thereof with FIGS. 18( b ) and 18( d ) .
- FIG. 18( b ) is a graph illustrating costs with further cost change
- FIG. 18( d ) is a graph illustrating synthesized costs with further cost change.
- Steps S 11 - 1 to S 14 illustrated in FIG. 15 are the same as steps S 1 to S 4 illustrated in FIG. 11 and a description thereof is omitted. Step S 15 and subsequent steps are described below.
- the cost synthesizer 320 generates and outputs a synthesized cost Ls with respect to each shift amount d, as expressed by the graph as in FIG. 17 ( d ).
- the high density disparity image in this case is the image as illustrated in FIG. 16( d ) .
- the cost calculator 310 further changes the costs as a cost changer and changes the cost C smaller than the threshold Th 2 to zero, as illustrated in the graph in FIG. 18( b ) (step S 17 ).
- the cost synthesizer 320 generates and outputs a synthesized cost Ls with respect to each shift amount d, as expressed by the graph as in FIG. 18( d ) .
- the processing in steps S 15 to S 17 may be performed in any order or may be processed concurrently.
- the cost synthesizer 320 illustrated in FIG. 10 then outputs a synthesized cost Ls with respect to each shift amount d, as illustrated in the graph in FIG. 18( d ) (step S 18 ).
- the high density disparity image in this case is the image as illustrated in FIG. 16( e ) .
- the disparity value ⁇ thus varies between when the cost calculator 310 does not change the costs and when it changes the costs.
- the high density disparity image see FIG.
- the high density disparity image (see FIG. 16( e ) ) in the case where the costs C are binarized is an image even more faithful to the object than the high density disparity image (see FIG. 16( d ) ) in the case where the costs C are not binarized.
- the CPU 32 thereafter transmits the high density disparity image represented by the disparity value ⁇ from the I/F 35 illustrated in FIG. 9 to the I/F 55 of the object recognition device 5 through the bus line 4 .
- the FPGA 51 then calculates the distance Z from the imaging devices 10 a , 10 b to the object E under an instruction from the CPU 52 in the object recognition device 5 .
- the cost C greater than the threshold Th 1 is thus reduced to a value equal to or smaller than the threshold Th 1 , so that the cost in an object with weak texture is less susceptible to the effects of the cost in the neighboring object with strong texture. This processing therefore achieves the advantageous effect of deriving a more accurate cost.
- the cost calculator 310 serves as the cost changer.
- the cost synthesizer 320 may serve as the cost changer. In this case, the cost synthesizer 320 performs cost synthesized after cost change.
- the second given value is zero.
- the second given value may be any value such as 10, 20, or 30 as long as it is smaller than the threshold Th.
- the cost calculated by the calculation unit exceeds a threshold
- the cost exceeding the threshold is changed to a given value higher than the threshold.
- the cost C greater than the given value is thus reduced to the given value, so that the cost in an object with weak texture is less susceptible to the effects of the cost in a neighboring object with strong texture.
- the present invention therefore achieves the advantageous effect of deriving a more accurate cost.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
- Measurement Of Optical Distance (AREA)
Abstract
A disparity value deriving device includes a calculator configured to calculate costs of candidates for a corresponding region in a comparison image that corresponds to a reference region in a reference image, based on luminance values of the regions. The device also includes a changer configured to change a cost exceeding a threshold to a value higher than the threshold; a synthesizer configured to synthesize a cost of a candidate for a corresponding region for one reference region after the change and a cost of a candidate for a corresponding region for another reference region after the change; and a deriving unit configured to derive a disparity value based on a position of the one reference region and a position of the corresponding region in the comparison image for which the cost after the synthesis is smallest.
Description
- The present invention relates to a disparity value deriving device, a movable apparatus, a robot, a disparity value producing method, and a computer program.
- A distant measurement method is conventionally known, in which disparity for an object is derived from a stereo camera by stereo imaging and a disparity value indicating this disparity is used to measure the distance from the stereo camera to the object based on the principle of triangulation. With this distance measurement method, for example, the distance between automobiles or the distance between an automobile and an obstacle is measured to be utilized for preventing automobile collisions.
- A stereo matching process is used for obtaining a disparity value. In this stereo matching process, a disparity value between a reference image obtained by one of two cameras of a stereo camera and a comparison image obtained by the other camera is calculated by obtaining the position of a corresponding pixel where image signals are most similar, while successively shifting a plurality of candidates for the corresponding pixel in the comparison image relative to a reference pixel of interest in the reference image. In general, the luminance values of the image signals obtained by two cameras are compared whereby the position of the pixel with the lowest one of costs (here, cost is “dissimilarity”) in terms of luminance value compared among shift amounts is obtained (see Japanese Patent Application Laid-open No. 2006-090896).
- However, in a region where texture indicating the magnitude of a luminance change of an object is weak and in which features to be extracted are scarce, a sufficient effect cannot be obtained even by performing edge detection.
- A method of deriving more accurate disparity even for an object with weak texture is proposed (see, for example, Japanese Patent Application Laid-open No. 2012-181142). With this method, not only the cost of a reference pixel of interest in a reference image but also the costs of other pixels existing near to far positions from the reference pixel are aggregated to derive disparity for an object with weak texture.
- However, when the distance between automobiles or other distance is measured using the method described in Japanese Patent Application Laid-open No. 2012-181142, costs for a road or other objects with weak texture are affected by costs for the surroundings with strong texture such as the vehicle running ahead. This causes failure to derive accurate costs.
- It is an object of the present invention to at least partially solve the problems in the conventional technology.
- According to an embodiment, there is provided a disparity value deriving device for deriving a disparity value indicating the disparity of an object from a reference image obtained by a first imaging unit capturing of an image of the object and a comparison image obtained by a second imaging unit capturing another image of the object. The disparity value deriving device includes a calculator configured to calculate costs of candidates for a corresponding region in the comparison image that corresponds to a reference region in the reference image, based on a luminance value of the reference region and luminance values of the candidates for the corresponding region, the candidates for the corresponding region being specified in a way that the respective candidates for the corresponding region are shifted on an epipolar line based on the reference region in the comparison image by individual shift amounts; a changer configured to change a cost exceeding a threshold among the costs calculated by the calculator to a given value higher than the threshold; a synthesizer configured to synthesize, with respect to each of the shift amounts, a cost of a candidate for a corresponding region that corresponds to one reference region after change by the changer and a cost of a candidate for a corresponding region that corresponds to another reference region after change by the changer; and a deriving unit configured to derive the disparity value based on a position of the one reference region in the reference image and a position of the corresponding region in the comparison image for which the cost after synthesis by the synthesizer is smallest.
- The above and other objects, features, advantages and technical and industrial significance of this invention will be better understood by reading the following detailed description of presently preferred embodiments of the invention, when considered in connection with the accompanying drawings.
-
FIG. 1 is an illustration of the principle of deriving the distance from an imaging device to an object. -
FIG. 2 illustrates (a) an example of a reference image, (b) an example of a high density disparity image, and (c) an example of an edge disparity image. -
FIG. 3 illustrates a calculation of a shift amount while successively shifting candidates for a corresponding pixel in a comparison image relative to a reference image. -
FIG. 4 is a graph illustrating costs with respect to shift amounts. -
FIG. 5 is a conceptual diagram for deriving a synthesized cost. -
FIG. 6 is a graph illustrating synthesized costs with respect to disparity values. -
FIG. 7 illustrates an automobile equipped with a disparity value deriving device according to an embodiment of the present invention. -
FIG. 8 is a schematic diagram of the disparity value deriving device. -
FIG. 9 is a hardware configuration diagram of the disparity value deriving device as a whole. -
FIG. 10 is a hardware configuration diagram of the principal part of the disparity value deriving device. -
FIG. 11 is a flowchart illustrating the processing in a first embodiment. -
FIG. 12 illustrates examples of a reference image, a part of the reference image, a high density disparity image without cost change, a high density disparity image with cost change with a threshold Th=40, and a high density disparity image with cost change with a threshold Th=20. -
FIG. 13 illustrates a graph of costs without cost change, a graph of costs with cost change, a graph of synthesized costs without cost change, and a graph of synthesized costs with cost change. -
FIG. 14 is a conceptual diagram of a reference image including a frame indicating the size of an object recognition result recognized by an object recognition device. -
FIG. 15 is a flowchart illustrating the processing in a second embodiment. -
FIG. 16 illustrates examples of a reference image, a part of the reference image, a high density disparity image without cost change, and high density disparity images with cost change. -
FIG. 17 illustrates a graph of costs without cost change, a graph of costs with cost change, a graph of synthesized costs without cost change, and a graph of synthesized costs with cost change. -
FIG. 18 illustrates a graph identical with (b) ofFIG. 17 , a graph of costs with cost change, a graph identical with (d) ofFIG. 17 , and a graph of synthesized costs with cost change. - Embodiments of the present invention will be described below with reference to the accompanying drawings.
- Overview of Distance Measurement Method Using SGM Method
- Referring first to
FIG. 1 toFIG. 6 , an overview of a distance measurement method using the semi-global matching (SGM) method will be described. The SGM method is disclosed in Accurate and Efficient Stereo Processing by Semi-Global Matching and Mutual Information) and a brief explanation is given below. - Principle of Distance Measurement
- Referring to
FIG. 1 , the principle of measuring the distance from a stereo camera to an object will be described, in which disparity of the object is derived as a disparity value by conducting stereo imaging using the stereo camera, and the disparity value is used to measure the distance from the stereo camera to the object.FIG. 1 is an illustration of the principle of deriving the distance from imaging devices to an object. The following description explains processes with respect to each single pixel rather than with respect to each predetermined region including a plurality of pixels, to make the explanation simpler. - When the process is performed not in units of single pixels but in units of predetermined regions each including a plurality of pixels, the predetermined region that includes a reference pixel is denoted as a reference region, and the predetermined region that includes a corresponding pixel is denoted as a corresponding region. The reference region may include a reference pixel alone and the corresponding region may include a corresponding pixel alone.
- Disparity Value Calculation
- The images captured by an
imaging device 10 a and animaging device 10 b illustrated inFIG. 1 are denoted as a reference image Ia and a comparison image Ib, respectively. InFIG. 1 , theimaging device 10 a and theimaging device 10 b are installed parallel and at the same height. InFIG. 1 , a point S on an object E in a three-dimensional space is captured at one position of theimaging device 10 a and also at one position of theimaging device 10 b, wherein these two positions are on the same horizontal line of theimaging device 10 a and theimaging device 10 b. That is, the point S in each image is captured at a point Sa(x,y) in the reference image Ia and a point Sb(x,y) in the comparison image Ib. Here, the disparity value Δ is expressed as Equation (1) using Sa(x,y) in the coordinates of theimaging device 10 a and Sb(X,y) in the coordinates of theimaging device 10 b. -
Δ=X−x (1) - Here, as in the case of
FIG. 1 , the disparity value is written as Δ=Δa+Δb, where Δa is the distance between the point Sa(x,y) in the reference image Ia and the point of intersection of the normal extending from theimaging lens 11 a to the imaging surface, and Δb is the distance between the point Sb(X,y) in the comparison image Ib and the point of intersection of the normal extending from theimaging lens 11 b to the imaging surface. - Distance Calculation
- The distance Z from the
imaging devices imaging lens 11 a and the focus position of theimaging lens 11 b to a particular point S on the object E. As illustrated inFIG. 1 , the distance Z can be calculated by Equation (2) using the focal length f of theimaging lens 11 a and theimaging lens 11 b, the base line length B that is the length between theimaging lens 11 a and theimaging lens 11 b, and the disparity value Δ. -
Z=(B×f)/Δ (2) - From Equation (2), the greater the disparity value Δ is, the smaller the distance Z is, and the smaller the disparity value Δ is, the greater the distance Z is.
- SGM Method
- Referring now to
FIG. 2 toFIG. 6 , a distance measurement method using the SGM method will be described.FIG. 2(a) is a conceptual diagram illustrating a reference image,FIG. 2(b) is a conceptual diagram illustrating a high density disparity image forFIG. 2(a) , andFIG. 2(c) is a conceptual diagram illustrating an edge disparity image forFIG. 2(a) . Here, the reference image is an image representing an object by luminance. The high density disparity image is an image that is derived from the reference image by the SGM method and represents the disparity value at each set of coordinates in the reference image. The edge disparity image is an image that is derived by the conventionally used block matching method and represents the disparity values only at a part with relatively strong texture such as an edge in the reference image. - The SGM method is a method of deriving the disparity values appropriately even for an object with weak texture and deriving the high density disparity image illustrated in
FIG. 2(b) based on the reference image illustrated inFIG. 2(a) . When the block matching method is used, the edge disparity image illustrated inFIG. 2(c) is derived based on the reference image illustrated inFIG. 2(a) . As can be understood by comparing the ovals enclosed by the broken lines inFIG. 2(b) andFIG. 2(c) , compared with the edge disparity image, the high density disparity image can represent detailed information such as a road with weak texture and therefore enables distance measurement in more detail. - In the SGM method, a disparity value is derived by calculating a cost and thereafter further calculating a synthesized cost that is synthesized dissimilarity, rather than deriving a disparity value immediately after calculating a cost that is dissimilarity. In this method, a disparity image (here, high density disparity image) representing disparity values in almost all the pixels is finally derived.
- The block matching method is the same as the SGM method in that a cost is calculated. However, unlike the SGM method, the disparity values only at a part with relatively strong texture such as an edge are derived without synthesized costs being calculated.
- Calculation of Cost
- Referring first to
FIG. 3 andFIG. 4 , a method of calculating a cost C(p,d) will be described.FIG. 3(a) is a conceptual diagram illustrating a reference pixel in a reference image andFIG. 3(b) is a conceptual diagram of calculating shift amounts (displacement amounts) while successively shifting (displacing) candidates for the corresponding pixel in a comparison image relative to the reference pixel in (a).FIG. 4 is a graph illustrating costs with respect to shift amounts. Here, the corresponding pixel is a pixel in the comparison image that is the most similar to the reference pixel in the reference image. In the following description, it is assumed that C(p,d) represents C(x,y,d). - As illustrated in
FIG. 3(a) , the cost C(p,d) of each candidate corresponding pixel q(x+d,y) that corresponds to the reference pixel p(x,y) is calculated based on the luminance values of a predetermined reference pixel p(x,y) in the reference image and of a plurality of candidate corresponding pixels q(x+d,y) on the epipolar line in the comparison image that corresponds to the reference pixel p(x,y). The variable d is a shift amount (displacement amount) of each candidate q for the corresponding pixel from the reference pixel p, and a shift amount is expressed in units of pixels in the present embodiments. That is, inFIG. 3 , the cost C(p,d) that is dissimilarity in luminance value between a candidate corresponding pixel q(x+d,y) and the reference pixel p(x,y) is calculated while successively shifting the candidate corresponding pixels q(x+d,y) one pixel by one pixel in a predetermined range (for example, 0<d<25). A known method such as SAD (Sum of Absolute Difference) is applied as the method of calculating the cost C where the cost C denotes dissimilarity. - As illustrated in
FIG. 4 , the cost C(p,d) thus calculated can be expressed by a graph of a cost curve that is a set of the costs C with respect to shift amounts d. InFIG. 4 , since the cost C is zero when the shift amount d=5, 12, 19, the minimum value cannot be obtained. In the case of an object with weak texture, it is thus difficult to obtain the minimum value of the cost C. - Calculation of Synthesized Cost
- Referring now to
FIG. 5 andFIG. 6 , the method of calculating a synthesized cost Ls(p,d) will be described.FIG. 5 is a conceptual diagram for deriving a synthesized cost.FIG. 6 is a graph of a synthesized-cost curve illustrating synthesized costs with respect to disparity values. In the method of calculating a synthesized cost in the present embodiments, the cost C(p,d) in the reference pixel p(x,y) is calculated, and costs for each one of pixels p(xn,yn) existing near to far positions from the reference pixel p(x,y) is also calculated as costs C for each one of another reference pixels p(xn,yn). With this configuration, the synthesized cost Ls(p,d) for the reference pixel p(x,y) is calculated by aggregating the calculated cost C(p,d) for the reference pixel p(x,y) and the calculated costs C for other reference pixels p(xn,yn) existing near to far positions from the reference pixel p(x,y). Similarly, the synthesized cost Ls for each one of reference pixels p(xn,yn) is calculated by aggregating the calculated cost C for one reference pixel p(xn,yn) and the calculated costs C for other reference pixels existing near to far positions from the reference pixel p(xn,yn). - The method of calculating a synthesized cost will now be described in more detail. In order to calculate the synthesized cost Ls(p,d), first, it is necessary to calculate a path cost Lr(p,d). Equation (3) is an equation for calculating the path cost Lr(p,d), and Equation (4) is an equation for calculating the synthesized cost Ls.
-
Lr(p,d)=C(p,d)+min{(Lr(p−r,d), Lr(p−r,d−1)+P1, Lr(p−r,d+1)+P1, Lrmin(p−r)+p2} (3) - Here, in Equation (3), r denotes a direction vector in the aggregation direction and has two components of the x direction and the y direction. The term min{} is a function for obtaining the minimum value. Lrmin(p−r) denotes the minimum value of Lr(p−r,d) when the shift amount d is changed in the coordinates in which p is shifted by one pixel in r direction. Lr is recurrently applied as expressed in Equation (3). P1 and P2 are fixed parameters set by experiment in advance such that the disparity values Δ of adjacent reference pixels on the path are likely to be continuous. For example, P1=48, P2=96.
- As expressed in Equation (3), Lr(p,d) is obtained by adding the minimum value of the path cost Lr of each pixel in the pixels in r direction illustrated in
FIG. 5 to the cost C in the reference pixel p(x,y). As described above, in order to obtain Lr at each pixel in r direction, Lr is obtained first from the endmost pixel in r direction of the reference image p(x,y), and Lr is obtained along r direction. - As illustrated in
FIG. 5 , Lr0, Lr45, Lr90, Lr135, Lr180, Lr225, Lr270, Lr315 in eight directions are obtained, and the synthesized cost Ls is eventually obtained based on Equation (4). -
- The synthesized cost Ls(p,d) thus calculated can be represented by a graph of a synthesized-cost curve representing the synthesized cost Ls(p,d) with respect to the shift amount d, as illustrated in
FIG. 6 . InFIG. 6 , the synthesized cost Ls has the minimum value when the shift amount d=3 and therefore is calculated as the disparity value Δ=3. - Although the number of r is eight in the foregoing description, the number is not limited thereto. For example, the eight directions may be further divided by two into 16 directions or by three into 24 directions.
- Although being expressed as “dissimilarity”, the cost C may be expressed as “similarity” that is a reciprocal of dissimilarity. In this case, a known method such as NCC (Normalized Cross Correlation) is applied as the method of calculating the cost C. In this case, the disparity value Δ not with the minimum but with the “maximum” synthesized cost Ls is derived. The similarity and the dissimilarity may be inclusively denoted as “matching degree”.
- Specific Description of Present Embodiments
- Specific descriptions of the present embodiments are given below with reference to the drawings. Here, an
object recognition system 1 mounted on an automobile will be described. - The
object recognition system 1 may be mountable not only on an automobile as an example of a vehicle but also on a motor bicycle, a bicycle, a wheelchair, and an agricultural cultivator as other examples of a vehicle. Theobject recognition system 1 may be mountable not only on a vehicle as an example of a movable apparatus but also on a robot as another example of a movable apparatus. The robot may not be a movable apparatus but may be an apparatus such as an industrial robot fixedly installed in FA (Factory Automation). The apparatus fixedly installed may not be a robot but may be a security monitoring camera. - Configuration of Embodiment
- First, the overall configuration of each of the present embodiments will be described with reference to
FIG. 7 toFIG. 9 . - External Configuration
- With reference to
FIG. 7 andFIG. 8 , an external configuration of theobject recognition system 1 in the present embodiments will be described.FIG. 7(a) is a schematic diagram depicting a side of an automobile equipped with the object recognition system according to an embodiment of the present invention andFIG. 7(b) is a schematic diagram depicting the front of the automobile.FIG. 8 is a schematic diagram of the object recognition system. - As illustrated in
FIGS. 7(a) and 7(b) , theobject recognition system 1 in the present embodiments includes animaging device 10 a and animaging device 10 b. Theimaging device 10 a and theimaging device 10 b are installed so as to be able to image the scene ahead in the direction in which the automobile travels. - As illustrated in
FIG. 8 , theobject recognition system 1 includes amain body 2 and a pair ofcylindrical imaging device 10 a andimaging device 10 b provided on themain body 2. - Overall Hardware Configuration
- Referring now to
FIG. 9 , an overall hardware configuration of theobject recognition system 1 will be described.FIG. 9 is a hardware configuration diagram of the object recognition system as a whole. - As illustrated in
FIG. 9 , theobject recognition system 1 includes a disparityvalue deriving device 3 and anobject recognition device 5 in themain body 2. The disparityvalue deriving device 3 derives a disparity value Δ indicating disparity for an object E from a plurality of images obtained by imaging the object E and outputs a high density disparity image indicating the disparity value Δ in each pixel. Theobject recognition device 5 performs processing such as measuring the distance from theimaging devices value deriving device 3. - Here, a hardware configuration of the disparity
value deriving device 3 will be described first. As illustrated inFIG. 9 , the disparityvalue deriving device 3 includes theimaging device 10 a, theimaging device 10 b, a signal conversion device 20 a, asignal conversion device 20 b, and animage processing device 30. - The
imaging device 10 a generates an analog signal representing an image by imaging the scene ahead and includes animaging lens 11 a, adiaphragm 12 a, and animage sensor 13 a. Theimaging lens 11 a is an optical element for diffracting light passing through theimaging lens 11 a to form an image of an object. Thediaphragm 12 a cuts off part of light passing through theimaging lens 11 a to adjust the quantity of light input to theimage sensor 13 a described later. Theimage sensor 13 a is a semiconductor device that converts light input from theimaging lens 11 a and thediaphragm 12 a into an electrical analog image signal and is implemented, for example, by a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). Theimaging device 10 b has the same configuration as theimaging device 10 a, and a description of theimaging device 10 b is omitted. Theimaging lens 11 a and theimaging lens 11 b are installed such that their respective lens surfaces are on the same plane. - The signal conversion device 20 a converts an analog signal representing the captured image into image data in digital format, and includes a correlated double sampling (CDS) 21 a, an auto gain control (AGC) 22 a, an analog digital converter (ADC) 23 a, and a
frame memory 24 a. The CDS 21 a removes noise by correlated double sampling from the analog image signal converted by theimage sensor 13 a. TheAGC 22 a performs gain control of controlling the intensity of the analog image signal having noise removed by the CDS 21 a. TheADC 23 a converts the analog image signal gain-controlled by theAGC 22 a into image data in digital format. Theframe memory 24 a stores the image data converted by theADC 23 a. - Similarly, the
signal conversion device 20 b obtains image data from the analog image signal converted by theimaging device 10 b and includes aCDS 21 b, anAGC 22 b, anADC 23 b, and aframe memory 24 b. TheCDS 21 b, theAGC 22 b, theADC 23 b, and theframe memory 24 b have the same configuration as the CDS 21 a, theAGC 22 a, theADC 23 a, and theframe memory 24 a, respectively, and a description thereof is omitted. - The
image processing device 30 is a device for processing image data converted by the signal conversion device 20 a and thesignal conversion device 20 b. Theimage processing device 30 includes a field programmable gate array (FPGA) 31, a central processing unit (CPU) 32, a read only memory (ROM) 33, a random access memory (RAM) 34, an interface (I/F) 35, and abus line 39 such as an address bus and a data bus for electrically connecting thecomponents 31 to 35 as illustrated inFIG. 9 . - The
FPGA 31 is an integrated circuit and performs the process of calculating a disparity value Δ in the image represented by image data in accordance with an instruction from theCPU 32. TheCPU 32 controls each function in the disparityvalue deriving device 3. TheROM 33 stores an image processing program for theCPU 32 to execute to control each function in the disparityvalue deriving device 3. TheRAM 34 is used as a work area for theCPU 32. - The I/
F 35 is an interface for communicating with the I/F 55 of theobject recognition device 5 described later through thebus line 4 such as an address bus and a data bus. - A hardware configuration of the
object recognition device 5 will now be described. As illustrated inFIG. 9 , theobject recognition device 5 includes anFPGA 51, aCPU 52, aROM 53, aRAM 54, an I/F 55, a CAN I/F 58, and abus line 59 such as an address bus and a data bus for electrically connecting thecomponents 51 to 55, 58 as illustrated inFIG. 9 . - The
FPGA 51, theCPU 52, theROM 53, theRAM 54, the I/F 55, and thebus line 59 have the same configuration as theFPGA 31, theCPU 32, theROM 33, theRAM 34, the I/F 35, and thebus line 39, respectively, in theimage processing device 30 and a description thereof is omitted. The I/F 55 is an interface for communicating with the I/F 35 in theimage processing device 30 through thebus line 4. TheROM 53 stores an object recognition program for theCPU 52 to execute to control each function in theobject recognition device 5. - The CAN I/
F 58 is an interface for communicating with, for example, an external controller and can be connected to, for example, a controller area network (CAN) of the automobile. - In such a configuration, when a high density disparity image is transmitted from the I/
F 35 of theimage processing device 30 to theobject recognition device 5 through thebus line 4, theFPGA 51 calculates the distance Z from theimaging devices CPU 52 in theobject recognition device 5. TheFPGA 31 may calculate the distance Z under an instruction from theCPU 32 of theimage processing device 30, rather than theFPGA 51 calculating the distance Z under an instruction from theCPU 52 in theobject recognition device 5. - The programs described above may be recorded in an installable or executable file format on a computer-readable storage medium to be distributed. Examples of the storage medium include a compact disc read only memory (CD-ROM) and a secure digital (SD) memory card.
- Specific Description of Embodiments
- A specific description of embodiments will be described below with reference to the accompanying drawings.
- First, a first embodiment of the present invention will be described.
- Hardware Configuration of Principal Part
- Referring to
FIG. 3 ,FIG. 9 andFIG. 10 , a hardware configuration of the principal part of the disparityvalue deriving device 3 according to the first embodiment will be described.FIG. 10 is a hardware configuration diagram of the principal part of the disparity value deriving device according to the first embodiment. - The
FPGA 31 inFIG. 9 includes acost calculator 310, acost synthesizer 320, and a disparityvalue deriving unit 330, as illustrated inFIG. 10 . Although these are part of the circuit of theFPGA 31, the same processing may be performed through execution of the image processing program. - The
cost calculator 310 calculates costs C of candidate corresponding pixels q(x+d,y) that corresponds to the reference pixel p(x,y), based on the luminance value of the reference pixel p(x,y) in the reference image (seeFIG. 3(a) ) and the luminance values of candidate corresponding pixels q(x+d,y) that are shifted on the epipolar line EL based on the reference pixel p(x,y) in the comparison image (seeFIG. 3(b) ) and are specified with respect to individual shift amounts d, as illustrated inFIG. 3 . Thecost calculator 310 also serves as a cost changer and changes a cost exceeding a threshold Th, among the calculated costs C, to a given value H higher than the threshold Th. - The
cost synthesizer 320 synthesizes, with respect to each shift amount d, a cost C of a candidate corresponding pixel q(x+d,y) that corresponds to a certain reference pixel p(x,y) after the change by the cost calculator 310 (cost changer) and a cost C of a candidate q′(x′+d,y′) for the corresponding pixel that corresponds to another reference pixel p′(x′,y′) after the change by the cost calculator 310 (cost changer), thereby outputting a synthesized cost Ls. This synthesizing process is a process of eventually calculating a synthesized cost Ls by calculating a path cost Lr from the cost C based on Equation (3) and thereafter further adding the path costs Lr for the respective directions based on Equation (4). - The disparity
value deriving unit 330 derives the disparity value A based on the position (x,y) in the reference image of a certain reference pixel p(x,y) and the position (x+Δ,y) of the corresponding pixel q(x+Δ,y) in the comparison image, for which the synthesized cost Ls after the synthesis by thecost synthesizer 320 is smallest, and outputs a disparity image indicating the disparity value in each pixel. - Processing or Operation of Embodiment
- Referring now to
FIG. 9 toFIG. 14 , the processing or operation of the present embodiment will be described. In the present embodiment, a case where the synthesized cost Ls is derived from the reference image illustrated inFIG. 12(a) will be described. For easy understanding of the drawing, part of the image inFIG. 12(a) is illustrated inFIG. 12(b) . -
FIG. 11 is a flowchart illustrating the processing in the present embodiment.FIG. 12(a) is a conceptual diagram illustrating a reference image,FIG. 12(b) is a conceptual diagram illustrating a part of the reference image,FIG. 12(c) is a conceptual diagram illustrating a high density disparity image forFIG. 12(b) without cost change,FIG. 12(d) is a conceptual diagram illustrating a high density disparity image forFIG. 12(b) with cost change with a threshold Th=40, andFIG. 12(e) is a conceptual diagram illustrating a high density disparity image forFIG. 12(b) with cost change with a threshold Th=20.FIG. 13(a) is a graph illustrating costs without cost change,FIG. 13(b) is a graph illustrating costs with cost change,FIG. 13(c) is a graph illustrating synthesized cost without cost change, andFIG. 13(d) illustrates synthesized cost with cost change. In the graphs inFIGS. 13(a) and 13(b) , the vertical axis represents costs C and the horizontal axis represents shift amounts d. In the graphs inFIGS. 13(c) and 13(d) , the vertical axis represents synthesized costs Ls and the horizontal axis represents shift amounts d. - First, the
imaging device 10 a illustrated inFIG. 9 generates analog image data by imaging an object E (step S1-1). Similarly, theimaging device 10 a generates analog image data by imaging the object (step S1-2). - The signal conversion device 20 a then converts the analog image data into digital image data (step S2-1). Similarly, the
signal conversion device 20 b converts the analog image data into digital image data (step S2-2). - The signal conversion device 20 a then outputs the converted digital image data to the
FPGA 31 of theimage processing device 30 as data of a reference image (step S3-1). The conceptual diagram of this reference image is illustrated inFIG. 12(a) . Similarly, thesignal conversion device 20 b outputs the converted digital image data to theFPGA 31 of theimage processing device 30 as data of a comparison image (step S3-2). This comparison image, which is an image captured by theimaging device 10 b, has no significant difference from the reference image illustrated inFIG. 12(a) and therefore a conceptual diagram of the comparison image is omitted without illustration. - The
cost calculator 310 illustrated inFIG. 10 then calculates costs C with respect to individual shift amounts d, as illustrated in the graph inFIG. 13(a) , based on the data of the reference image and the data of the comparison image (step S4). Here, supposing that thecost calculator 310 does not change the costs as a cost changer, thecost synthesizer 320 generates and outputs a synthesized cost Ls with respect to each shift amount d, as illustrated in the graph inFIG. 13(c) . In this case, since the shift amount d=13 gives the smallest synthesized cost Ls (Ls1), the disparityvalue deriving unit 330 outputs a disparity value Δ=13. The high density disparity image in this case is the image as illustrated inFIG. 12(c) . - By contrast, in the present embodiment, the
cost calculator 310 changes the costs as a cost changer and changes the cost C exceeding a threshold Th (for example, C=40) to a given value (for example, C=80), as illustrated in the graph inFIG. 13(b) (step S5). Thecost calculator 310, as a cost calculator, further changes the cost C equal to or smaller than the threshold Th (for example, C=40) to a given value (for example, C=0), as illustrated in the graph inFIG. 13(b) (step S6). Step S5 and step S6 may be processed in the reverse order or may be processed concurrently. - The
cost synthesizer 320 illustrated inFIG. 10 then outputs the synthesized cost Ls with respect to individual shift amounts d, as illustrated in the graph inFIG. 13(d) (step S7). In this case, since the shift amount d=3 gives the smallest synthesized cost Ls (Ls2), the disparityvalue deriving unit 330 outputs a disparity value Δ=3. The high density disparity image in this case is a silhouette image as illustrated inFIG. 12(d) . This silhouette image is a schematic image of an object in the reference image recognized by theobject recognition device 5 and decreases in size as the threshold Th is lower. The threshold Th approaching zero has an advantage of increasing the accuracy of the synthesized cost Ls(p,d) but has a disadvantage of increasing the circuit scale of theFPGA 31 for calculating the synthesized cost Ls(p,d). - As described above, the disparity value Δ varies between when the
cost calculator 310 does not change the cost and when it changes the cost. The high density disparity image (seeFIG. 12(d) ) in the case where the costs are changed is more faithful to the object than the high density disparity image (seeFIG. 12(c) ) in the case where the costs are not changed. - The
CPU 32 then transmits the high density disparity image represented by the disparity values Δ from the I/F 35 illustrated inFIG. 9 to the I/F 55 of theobject recognition device 5 through thebus line 4. TheFPGA 51 then calculates the distance Z from theimaging devices CPU 52 in theobject recognition device 5. - The
object recognition device 5 further recognizes a frame indicating the size of the object E in the reference image as illustrated inFIG. 14 .FIG. 14 is a conceptual diagram of the reference image including a frame indicating the size of the object recognition result recognized by the object recognition device. In this case, since the silhouette image is large when the threshold Th=40 as illustrated inFIG. 12(d) , a frame 1001 (here, denoted by the broken line) is depicted in the reference image inFIG. 14 . On the other hand, since the silhouette image is small when the threshold Th=20 as illustrated inFIG. 12(e) , a frame 1002 (here, denoted by the dashed line) is depicted in the reference image inFIG. 14 . The size of the object recognition result may not be indicated by a frame but may be indicated by a longitudinal or oval form, or another form that masks the frame. - Main Advantageous Effects of Embodiment
- As described above, in the present embodiment, when the cost C calculated by the
cost calculator 310 exceeds the threshold Th, thecost calculator 310 changes the cost C exceeding the threshold Th to a first given value H higher than the threshold Th. As a result, the cost C greater than the first given value H is reduced to the first given value H, so that the cost in an object with weak texture is less susceptible to the costs in the neighboring object with strong texture. This processing therefore achieves the advantageous effect of deriving a more accurate cost. - Changing various costs C to the first given value H also achieves the advantageous effect of reducing the circuit scale of the RAM or other units for deriving a shift amount d.
- The
cost calculator 310 further changes the cost equal to or smaller than the threshold Th to a second given value, zero, lower than the threshold Th. As described above, the costs C not narrowed down by the first given value H are changed to the second predetermine value zero. Therefore, the shift amount d can be derived through calculation with only two values, namely, the first given value H and the second given value zero, thereby achieving the advantageous effect of further reducing the circuit scale. The calculation with only two values, namely, the first given value H and the second given value zero can reduce the excessive effects brought about by the magnitude of the costs C in calculating the synthesized cost Ls and enables derivation of a disparity value even more faithful to the actual object. - A second embodiment of the present invention will now be described. The hardware configuration of the principal part of the present embodiment is the same as the hardware configuration of the principal part of the first embodiment and a description thereof is omitted.
- Processing or Operation of Embodiment
- Referring now to
FIG. 15 toFIG. 18 , the processing or operation of the present embodiment will be described. In the present embodiment, a case where the synthesized cost Ls is derived from the reference image illustrated inFIG. 16(a) will be described. For easy understanding of the drawing, a part of the image inFIG. 16(a) is illustrated inFIG. 16(b) . -
FIG. 15 is a flowchart illustrating the processing in the second example embodiment.FIG. 16(a) is an example of a reference image,FIG. 16(b) is a part of the reference image,FIG. 16(c) is a conceptual diagram illustrating a high density disparity image forFIG. 16(b) without cost change, andFIG. 16(d) is a conceptual diagram illustrating a high density disparity image forFIG. 16(b) with cost change.FIG. 17(a) is a graph illustrating costs without cost change,FIG. 17(b) is a graph illustrating costs with cost change,FIG. 17(c) is a graph illustrating synthesized costs without cost change, andFIG. 17(d) illustrates synthesized costs with cost change.FIGS. 18(a) and 18(c) are the same graphs asFIGS. 17(b) and 17(d) , respectively.FIGS. 18(a) and 18(c) are provided to facilitate comparison thereof withFIGS. 18(b) and 18(d) .FIG. 18(b) is a graph illustrating costs with further cost change, andFIG. 18(d) is a graph illustrating synthesized costs with further cost change. - In the graphs in
FIGS. 17(a) and 17(b) andFIGS. 18(a) and 18(b) , the vertical axis represents costs C and the horizontal axis represents shift amounts d. In the graphs inFIGS. 17(c) and 17(d) andFIGS. 18(c) and 18(d) , the vertical axis represents synthesized costs Ls and the horizontal axis represents shift amounts d. Steps S11-1 to S14 illustrated inFIG. 15 are the same as steps S1 to S4 illustrated inFIG. 11 and a description thereof is omitted. Step S15 and subsequent steps are described below. - After the processing in step S14, the
cost calculator 310 changes the costs as a cost changer and changes the cost C exceeding a threshold Th1 (for example, C=80) to the threshold Th1, as illustrated in the graph inFIG. 17(b) (step S15). In this state, thecost synthesizer 320 generates and outputs a synthesized cost Ls with respect to each shift amount d, as expressed by the graph as in FIG. 17(d). In this case, since the shift amount d=3 gives the smallest synthesized cost Ls (Ls3), the disparityvalue deriving unit 330 outputs a disparity value Δ=3. The high density disparity image in this case is the image as illustrated inFIG. 16(d) . - The
cost calculator 310 then changes the costs as a cost changer and changes the cost C smaller than the threshold Th1 and equal to or greater than a threshold Th2 (for example, C=40) to the threshold Th1, as illustrated in the graph inFIG. 18(b) (step S16). Thecost calculator 310 further changes the costs as a cost changer and changes the cost C smaller than the threshold Th2 to zero, as illustrated in the graph inFIG. 18(b) (step S17). In this state, thecost synthesizer 320 generates and outputs a synthesized cost Ls with respect to each shift amount d, as expressed by the graph as inFIG. 18(d) . In this case, because the shift amount d=3 becomes the smallest synthesized cost Ls (Ls3), the disparityvalue deriving unit 330 outputs a disparity value Δ=3. In this case, the shift amount d=13 corresponds to a local minimum synthesized cost Ls4, which is a synthesized cost greater than the smallest local minimum synthesized cost (i.e., the smallest synthesized cost Ls3) as illustrated inFIG. 18(d) . Because the synthesized cost of the local minimum synthesized cost Ls4 ofFIG. 18(d) is higher compared toFIG. 18(c) , the lowness of the synthesized cost Ls in the vicinity of the shift amount d=3 ofFIG. 18(d) becomes prominent. The processing in steps S15 to S17 may be performed in any order or may be processed concurrently. - The
cost synthesizer 320 illustrated inFIG. 10 then outputs a synthesized cost Ls with respect to each shift amount d, as illustrated in the graph inFIG. 18(d) (step S18). In this case, since the shift amount d=3 gives the smallest synthesized cost Ls (Ls3), the disparityvalue deriving unit 330 outputs a disparity value Δ=3. The high density disparity image in this case is the image as illustrated inFIG. 16(e) . The disparity value Δ thus varies between when thecost calculator 310 does not change the costs and when it changes the costs. The high density disparity image (seeFIG. 16(d) ) in the case where the costs are changed is an image more faithful to the object than the high density disparity image (seeFIG. 16(c) ) in the case where the costs are not changed. The high density disparity image (seeFIG. 16(e) ) in the case where the costs C are binarized is an image even more faithful to the object than the high density disparity image (seeFIG. 16(d) ) in the case where the costs C are not binarized. - The
CPU 32 thereafter transmits the high density disparity image represented by the disparity value Δ from the I/F 35 illustrated inFIG. 9 to the I/F 55 of theobject recognition device 5 through thebus line 4. TheFPGA 51 then calculates the distance Z from theimaging devices CPU 52 in theobject recognition device 5. - Since the silhouette image is large when the threshold Th2=40, a
frame 1001 is depicted in the reference image as illustrated inFIG. 14 , in the same manner as in the first embodiment. On the other hand, since the silhouette image is small when the threshold Th2=20, aframe 1002 is depicted in the reference image as illustrated inFIG. 14 . - Main Advantageous Effects of Embodiment
- As described above, in the present embodiment, when the cost C calculated by the
cost calculator 310 exceeds the threshold Th1 (for example, C=80), the cost calculator 310 (cost changer) changes the cost C exceeding the threshold Th1 to a value equal to or smaller than the threshold Th1. The cost C greater than the threshold Th1 is thus reduced to a value equal to or smaller than the threshold Th1, so that the cost in an object with weak texture is less susceptible to the effects of the cost in the neighboring object with strong texture. This processing therefore achieves the advantageous effect of deriving a more accurate cost. - The cost calculator 310 (cost changer) further changes the cost C smaller than the threshold Th1 and equal to or greater than the threshold Th2 (for example, C=40) to the threshold Th1. The cost calculator 310 (cost changer) changes the cost smaller than the threshold Th2 to a given value smaller than the threshold (for example, C=0). As described above, the costs C are narrowed down as much as possible and eventually controlled by two values (C=0, 80) as illustrated in
FIG. 18(b) . This processing therefore achieves the advantageous effect of reducing the circuit scale of the RAM or the like for deriving the shift amount d. - Supplementary Remarks on Embodiments
- In the foregoing embodiments, the
cost calculator 310 serves as the cost changer. However, embodiments are not limited thereto and thecost synthesizer 320 may serve as the cost changer. In this case, thecost synthesizer 320 performs cost synthesized after cost change. - In the foregoing second embodiment, the second given value is zero. However, embodiments are not limited thereto and the second given value may be any value such as 10, 20, or 30 as long as it is smaller than the threshold Th.
- As described above, to synthesize the costs, if the cost calculated by the calculation unit exceeds a threshold, the cost exceeding the threshold is changed to a given value higher than the threshold. The cost C greater than the given value is thus reduced to the given value, so that the cost in an object with weak texture is less susceptible to the effects of the cost in a neighboring object with strong texture. The present invention therefore achieves the advantageous effect of deriving a more accurate cost.
- Although the invention has been described with respect to specific embodiments for a complete and clear disclosure, the appended claims are not to be thus limited but are to be construed as embodying all modifications and alternative constructions that may occur to one skilled in the art that fairly fall within the basic teaching herein set forth.
Claims (19)
1-17. (canceled)
18. A disparity value deriving device comprising:
an obtaining unit configured to obtain a matching degree of a reference region in a reference image captured at a first imaging position with each of regions in a comparison image captured at a second imaging position, the regions in the comparison image being within a certain range and including a region corresponding to the reference region;
a changer configured to change the matching degree based on the matching degree;
a synthesizer configured to synthesize a matching degree for one reference region in the reference image after change by the changer and a matching degree for another reference region near the one reference region in the reference image after change by the changer; and
a deriving unit configured to derive, based on the synthesized matching degree, a disparity value for an object whose image is captured in the one reference region.
19. The disparity value deriving device according to claim 18 , wherein
the changer changes a matching degree exceeding a threshold among the matching degrees obtained by the obtaining unit to a given value higher than the threshold, and
the deriving unit derives the disparity value based on a minimum matching degree of the matching degrees after synthesis by the synthesizer.
20. The disparity value deriving device according to claim 19 , wherein
the given value is a first given value, and
the obtaining unit changes a matching degree equal to or smaller than the threshold to a second given value lower than the threshold.
21. The disparity value deriving device according to claim 20 , wherein the second given value is zero.
22. The disparity value deriving device according to claim 18 , wherein
the changer changes a matching degree falling below a threshold among the matching degrees obtained by the obtaining unit to a given value lower than the threshold, and
the deriving unit derives the disparity value based on a maximum matching degree of the matching degrees after synthesis by the synthesizer.
23. The disparity value deriving device according to claim 22 , wherein
the given value is a first given value, and
the obtaining unit changes a matching degree equal to or greater than the threshold to a second given value higher than the threshold.
24. The disparity value deriving device according to claim 18 , wherein
the changer changes a matching degree exceeding a threshold among the matching degrees obtained by the obtaining unit to a value equal to or smaller than the threshold, and
the deriving unit derives the disparity value based on a minimum matching degree of the matching degrees after synthesis by the synthesizer.
25. The disparity value deriving device according to claim 18 , wherein
the changer changes a matching degree falling below a threshold among the matching degrees obtained by the obtaining unit to a value equal to or greater than the threshold, and
the deriving unit derives the disparity value based on a maximum matching degree of the matching degrees after synthesis by the synthesizer.
26. The disparity value deriving device according to claim 18 , wherein
the changer binarizes the matching degree obtained by the obtaining unit based on a threshold.
27. The disparity value deriving device according to claim 26 , wherein
the changer changes a matching degree exceeding a threshold to a given value, and changes a matching degree falling below the threshold to zero.
28. The disparity value deriving device according to claim 18 , wherein
the changer changes a matching degree exceeding a threshold among the matching degrees obtained by the obtaining unit to a given value,
the deriving unit derives the disparity value based on a minimum matching degree of the matching degrees after synthesis by the synthesizer, and outputs the disparity value to an object recognition device for recognizing the object in the reference image, and
when the threshold is lower, the size of an object recognition result recognized by the object recognition device is smaller.
29. The disparity value deriving device according to claim 18 , wherein
the changer changes a matching degree falling below a threshold among the matching degrees obtained by the obtaining unit to a given value,
the deriving unit derives the disparity value based on a maximum matching degree of the matching degrees after synthesis by the synthesizer, and outputs the disparity value to an object recognition device for recognizing the object in the reference image, and
when the threshold is higher, the size of an object recognition result recognized by the object recognition device is smaller.
30. A movable apparatus comprising the disparity value deriving device according to claim 18 .
31. The movable apparatus according to claim 30 , wherein the movable apparatus is a vehicle or a robot.
32. A robot comprising the disparity value deriving device according to claim 18 .
33. The robot according to claim 32 , wherein the robot is an industrial robot fixedly installed.
34. A disparity value producing method comprising:
obtaining a matching degree of a reference region in a reference image captured at a first imaging position with each of regions in a comparison image captured at a second imaging position, the regions in the comparison image being within a certain range and including a region corresponding to the reference region;
changing the matching degree based on the matching degree;
synthesizing a matching degree for one reference region in the reference image after the changing and a matching degree for another reference region near the one reference region in the reference image after the changing; and
deriving, based on the synthesized matching degree, a disparity value for an object whose image is captured in the one reference region.
35. A non-transitory computer-readable storage medium with an executable program stored thereon and executed by a computer, wherein the program instructs the computer to perform:
obtaining a matching degree of a reference region in a reference image captured at a first imaging position with each of regions in a comparison image captured at a second imaging position, the regions in the comparison image being within a certain range and including a region corresponding to the reference region;
changing the matching degree based on the matching degree;
synthesizing a matching degree for one reference region in the reference image after the changing and a matching degree for another reference region near the one reference region in the reference image after the changing; and
deriving, based on the synthesized matching degree, a disparity value for an object whose image is captured in the one reference region.
Applications Claiming Priority (7)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2013-257217 | 2013-12-12 | ||
JP2013257217 | 2013-12-12 | ||
JP2013-268468 | 2013-12-26 | ||
JP2013268468 | 2013-12-26 | ||
JP2014-234777 | 2014-11-19 | ||
JP2014234777A JP6417886B2 (en) | 2013-12-12 | 2014-11-19 | Parallax value deriving device, moving body, robot, parallax value production method, and program |
PCT/JP2014/083626 WO2015088044A1 (en) | 2013-12-12 | 2014-12-12 | Disparity value deriving device, movable apparatus, robot, disparity value producing method, and computer program |
Publications (2)
Publication Number | Publication Date |
---|---|
US20160301912A1 true US20160301912A1 (en) | 2016-10-13 |
US10104359B2 US10104359B2 (en) | 2018-10-16 |
Family
ID=53371338
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/100,895 Active US10104359B2 (en) | 2013-12-12 | 2014-12-12 | Disparity value deriving device, movable apparatus, robot, disparity value producing method, and computer program |
Country Status (6)
Country | Link |
---|---|
US (1) | US10104359B2 (en) |
EP (1) | EP3080549B1 (en) |
JP (1) | JP6417886B2 (en) |
KR (1) | KR101820259B1 (en) |
CN (1) | CN105814403B (en) |
WO (1) | WO2015088044A1 (en) |
Cited By (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20190180461A1 (en) * | 2016-07-06 | 2019-06-13 | SZ DJI Technology Co., Ltd. | Systems and methods for stereoscopic imaging |
US10733459B2 (en) | 2016-03-18 | 2020-08-04 | Ricoh Company, Ltd. | Image processing device, image processing method, computer-readable recording medium, object recognition device, device control system, and movable body |
US10748014B2 (en) | 2015-12-28 | 2020-08-18 | Ricoh Company, Ltd. | Processing device, object recognition apparatus, device control system, processing method, and computer-readable recording medium |
US10853963B2 (en) | 2016-02-05 | 2020-12-01 | Ricoh Company, Ltd. | Object detection device, device control system, and medium |
US10937181B2 (en) | 2016-03-15 | 2021-03-02 | Ricoh Company, Ltd. | Information processing apparatus, object recognition apparatus, device control system, movable body, image processing method, and computer-readable recording medium |
US10984509B2 (en) | 2016-01-28 | 2021-04-20 | Ricoh Company, Ltd. | Image processing apparatus, imaging device, moving body device control system, image information processing method, and program product |
US10992879B2 (en) | 2017-03-17 | 2021-04-27 | Ricoh Company, Ltd. | Imaging system with multiple wide-angle optical elements arranged on a straight line and movable along the straight line |
US11039120B2 (en) | 2017-01-31 | 2021-06-15 | Ricoh Company, Ltd. | Imaging apparatus |
US11054245B2 (en) | 2016-03-10 | 2021-07-06 | Ricoh Company, Ltd. | Image processing apparatus, device control system, imaging apparatus, image processing method, and recording medium |
US20220094933A1 (en) * | 2015-01-30 | 2022-03-24 | Texas Instruments Incorporated | Semi-global matching (sgm) cost compression |
US11443410B2 (en) | 2019-10-10 | 2022-09-13 | Canon Kabushiki Kaisha | Image processing method and image processing apparatus |
Families Citing this family (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP6417886B2 (en) | 2013-12-12 | 2018-11-07 | 株式会社リコー | Parallax value deriving device, moving body, robot, parallax value production method, and program |
JP6565188B2 (en) | 2014-02-28 | 2019-08-28 | 株式会社リコー | Parallax value deriving apparatus, device control system, moving body, robot, parallax value deriving method, and program |
JP7206583B2 (en) * | 2016-11-25 | 2023-01-18 | 株式会社リコー | Information processing device, imaging device, device control system, moving object, information processing method and program |
Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120014590A1 (en) * | 2010-06-25 | 2012-01-19 | Qualcomm Incorporated | Multi-resolution, multi-window disparity estimation in 3d video processing |
Family Cites Families (17)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5602653A (en) * | 1994-11-08 | 1997-02-11 | Xerox Corporation | Pixel pair grid halftoning for a hyperacuity printer |
US6215898B1 (en) | 1997-04-15 | 2001-04-10 | Interval Research Corporation | Data processing system and method |
US7697749B2 (en) | 2004-08-09 | 2010-04-13 | Fuji Jukogyo Kabushiki Kaisha | Stereo image processing device |
JP4554316B2 (en) | 2004-09-24 | 2010-09-29 | 富士重工業株式会社 | Stereo image processing device |
WO2008044911A1 (en) | 2006-10-09 | 2008-04-17 | Tele Atlas B.V. | Method and apparatus for generating an orthorectified tile |
JP5371725B2 (en) | 2009-12-16 | 2013-12-18 | 富士重工業株式会社 | Object detection device |
JP2011163900A (en) | 2010-02-09 | 2011-08-25 | Casio Computer Co Ltd | Image processor, image processing program, and image processing method |
KR101694292B1 (en) * | 2010-12-17 | 2017-01-09 | 한국전자통신연구원 | Apparatus for matching stereo image and method thereof |
JP2012177676A (en) * | 2011-01-31 | 2012-09-13 | Sony Corp | Information processor and method, and program |
JP5682065B2 (en) * | 2011-03-02 | 2015-03-11 | 独立行政法人産業技術総合研究所 | Stereo image processing apparatus and stereo image processing method |
EP2511875A1 (en) | 2011-04-14 | 2012-10-17 | Thomson Licensing | Apparatus and method for refining a value of a similarity measure |
JP2014096062A (en) | 2012-11-09 | 2014-05-22 | Yamaguchi Univ | Image processing method and image processing apparatus |
JP2014115978A (en) | 2012-11-19 | 2014-06-26 | Ricoh Co Ltd | Mobile object recognition device, notification apparatus using the device, mobile object recognition program for use in the mobile object recognition device, and mobile object with the mobile object recognition device |
CN103198314B (en) * | 2013-02-20 | 2015-11-25 | 北京农业信息技术研究中心 | Remote sensing images radiation correction method |
CN103247033A (en) | 2013-04-26 | 2013-08-14 | 中国科学院东北地理与农业生态研究所 | Vector data correction method for image-based control point selection |
JP6202367B2 (en) | 2013-05-14 | 2017-09-27 | 株式会社リコー | Image processing device, distance measurement device, mobile device control system, mobile device, and image processing program |
JP6417886B2 (en) | 2013-12-12 | 2018-11-07 | 株式会社リコー | Parallax value deriving device, moving body, robot, parallax value production method, and program |
-
2014
- 2014-11-19 JP JP2014234777A patent/JP6417886B2/en active Active
- 2014-12-12 US US15/100,895 patent/US10104359B2/en active Active
- 2014-12-12 KR KR1020167015408A patent/KR101820259B1/en active IP Right Grant
- 2014-12-12 WO PCT/JP2014/083626 patent/WO2015088044A1/en active Application Filing
- 2014-12-12 EP EP14869791.5A patent/EP3080549B1/en active Active
- 2014-12-12 CN CN201480067342.3A patent/CN105814403B/en active Active
Patent Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120014590A1 (en) * | 2010-06-25 | 2012-01-19 | Qualcomm Incorporated | Multi-resolution, multi-window disparity estimation in 3d video processing |
Cited By (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11831874B2 (en) * | 2015-01-30 | 2023-11-28 | Texas Instruments Incorporated | Semi-global matching (SGM) cost compression |
US20220094933A1 (en) * | 2015-01-30 | 2022-03-24 | Texas Instruments Incorporated | Semi-global matching (sgm) cost compression |
US10748014B2 (en) | 2015-12-28 | 2020-08-18 | Ricoh Company, Ltd. | Processing device, object recognition apparatus, device control system, processing method, and computer-readable recording medium |
US10984509B2 (en) | 2016-01-28 | 2021-04-20 | Ricoh Company, Ltd. | Image processing apparatus, imaging device, moving body device control system, image information processing method, and program product |
US10853963B2 (en) | 2016-02-05 | 2020-12-01 | Ricoh Company, Ltd. | Object detection device, device control system, and medium |
US11054245B2 (en) | 2016-03-10 | 2021-07-06 | Ricoh Company, Ltd. | Image processing apparatus, device control system, imaging apparatus, image processing method, and recording medium |
US10937181B2 (en) | 2016-03-15 | 2021-03-02 | Ricoh Company, Ltd. | Information processing apparatus, object recognition apparatus, device control system, movable body, image processing method, and computer-readable recording medium |
US10733459B2 (en) | 2016-03-18 | 2020-08-04 | Ricoh Company, Ltd. | Image processing device, image processing method, computer-readable recording medium, object recognition device, device control system, and movable body |
US10896519B2 (en) * | 2016-07-06 | 2021-01-19 | SZ DJI Technology Co., Ltd. | Systems and methods for stereoscopic imaging |
US20190180461A1 (en) * | 2016-07-06 | 2019-06-13 | SZ DJI Technology Co., Ltd. | Systems and methods for stereoscopic imaging |
US11039120B2 (en) | 2017-01-31 | 2021-06-15 | Ricoh Company, Ltd. | Imaging apparatus |
US10992879B2 (en) | 2017-03-17 | 2021-04-27 | Ricoh Company, Ltd. | Imaging system with multiple wide-angle optical elements arranged on a straight line and movable along the straight line |
US11443410B2 (en) | 2019-10-10 | 2022-09-13 | Canon Kabushiki Kaisha | Image processing method and image processing apparatus |
Also Published As
Publication number | Publication date |
---|---|
WO2015088044A1 (en) | 2015-06-18 |
KR20160086875A (en) | 2016-07-20 |
EP3080549A4 (en) | 2016-12-21 |
KR101820259B1 (en) | 2018-01-18 |
JP6417886B2 (en) | 2018-11-07 |
CN105814403A (en) | 2016-07-27 |
US10104359B2 (en) | 2018-10-16 |
JP2015143677A (en) | 2015-08-06 |
EP3080549A1 (en) | 2016-10-19 |
CN105814403B (en) | 2018-11-09 |
EP3080549B1 (en) | 2019-03-06 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10104359B2 (en) | Disparity value deriving device, movable apparatus, robot, disparity value producing method, and computer program | |
US9747524B2 (en) | Disparity value deriving device, equipment control system, movable apparatus, and robot | |
US9898823B2 (en) | Disparity deriving apparatus, movable apparatus, robot, method of deriving disparity, method of producing disparity, and storage medium | |
US8971578B2 (en) | Driving support apparatus | |
WO2017138245A1 (en) | Image processing device, object recognition device, device control system, and image processing method and program | |
JP2009176090A (en) | Environment recognition system | |
US9813694B2 (en) | Disparity value deriving device, equipment control system, movable apparatus, robot, and disparity value deriving method | |
JP2016152027A (en) | Image processing device, image processing method and program | |
JP2006318060A (en) | Apparatus, method, and program for image processing | |
JP2006322795A (en) | Image processing device, image processing method and image processing program | |
JP2006318062A (en) | Image processor, image processing method and image processing program | |
EP2913998B1 (en) | Disparity value deriving device, equipment control system, movable apparatus, robot, disparity value deriving method, and computer-readable storage medium | |
EP2913999A1 (en) | Disparity value deriving device, equipment control system, movable apparatus, robot, disparity value deriving method, and computer-readable storage medium | |
JP2006318059A (en) | Apparatus, method, and program for image processing | |
JP6519138B2 (en) | PARALLEL VALUE DERIVING DEVICE, MOBILE OBJECT, ROBOT, PARALLEL VALUE PRODUCTION METHOD, AND PROGRAM | |
JP6561511B2 (en) | Parallax value deriving device, moving body, robot, parallax value production deriving method, parallax value producing method and program | |
JP2006318061A (en) | Apparatus, method, and program for image processing | |
EP2919191B1 (en) | Disparity value deriving device, equipment control system, movable apparatus, robot, and disparity value producing method | |
JP6442911B2 (en) | Parallax value deriving device, moving object, robot, parallax value deriving method, and program | |
JP2016095229A (en) | Parallax value derivation device, movable body, robot, parallax value production method, and program | |
JP2015195019A (en) | Disparity deriving apparatus, movable apparatus, robot, method of deriving disparity, method of producing disparity, and program |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: RICOH COMPANY, LIMITED, JAPAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:SAITOH, KIICHIROH;WATANABE, YOSHIKAZU;YOKOTA, SOICHIRO;AND OTHERS;SIGNING DATES FROM 20160418 TO 20160507;REEL/FRAME:038765/0657 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |