CN109741385A - A kind of image processing system, method, apparatus, electronic equipment and storage medium - Google Patents
A kind of image processing system, method, apparatus, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN109741385A CN109741385A CN201811583936.6A CN201811583936A CN109741385A CN 109741385 A CN109741385 A CN 109741385A CN 201811583936 A CN201811583936 A CN 201811583936A CN 109741385 A CN109741385 A CN 109741385A
- Authority
- CN
- China
- Prior art keywords
- image
- pretreated
- disparity map
- pixel
- dsp
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
The invention discloses a kind of image processing system, method, apparatus, electronic equipment and storage mediums, and the system comprises binocular camera, Digital Signal Processing DSP and on-site programmable gate array FPGAs；The binocular camera, for sending the first image and the second image of acquisition to the DSP；The DSP, for being pre-processed to the first image and the second image；The FPGA carries out Stereo matching to pretreated first image and the second image in the parallax directions parallel, generates the disparity map in the parallax directions for being directed to each parallax directions；According to each disparity map, depth map is determined.Since in embodiments of the present invention, FPGA determines the disparity map in each parallax directions parallel；And then according to each disparity map, depth map is determined.The disparity map first successively determined in each parallax directions is avoided, depth map bring latency issue is then determined again, it is thus determined that the real-time of depth map is preferable.
Description
Technical field
The present invention relates to technical field of image processing more particularly to a kind of image processing system, method, apparatus, electronics to set
Standby and storage medium.
Background technique
Stereoscopic vision is one of widest field of computer vision research, is widely used in automatic Pilot, Three-dimensional Gravity
It builds, the fields such as obstacle detection.Stereoscopic vision is the picture that Same Scene different perspectives is obtained by left and right camera, is then led to again
Cross the process that Stereo Matching Algorithm obtains the depth map of scene.
In the prior art when determining depth map, Digital Signal Processing (Digital Signal is generallyd use
Processing, DSP) mode, as shown in Figure 1, DSP receives left and right two video cameras sensor0 and sensor1 transmission
Then image carries out depth nomography to two images.
Problem of the existing technology is, when determining depth map, needs to calculate multidirectional disparity map, but DSP is only
There is single-pass process ability, can only successively determine that the disparity map in each direction, the disparity map to each direction calculate completion
Afterwards, then final depth map is determined, therefore the prior art determines that the real-time of depth map is poor.
Summary of the invention
The embodiment of the invention provides a kind of image processing system, method, apparatus, electronic equipment and storage medium, to
It solves the problems, such as to determine that the real-time of depth map is poor in the prior art.
The embodiment of the invention provides a kind of image processing systems, and the system comprises at binocular camera, digital signal
Manage DSP and on-site programmable gate array FPGA；
The binocular camera, for sending the first image and the second image of acquisition to the DSP；
The DSP, for being pre-processed to the first image and the second image；
The FPGA, for being directed to each parallax directions, parallel to by pretreated first figure in the parallax directions
Picture and the second image carry out Stereo matching, generate the disparity map in the parallax directions；According to each disparity map, depth map is determined.
Further, the DSP, specifically for according to preset pixel in the first image and the second image
Mark information, judge the first image and the second image whether be identical frame number left figure and right figure, if so, will be described
First image and the second image are sent to the FPGA.
It further, include three pieces of cachings in the FPGA；
The DSP is successively sent in the FPGA specifically for that will pass through pretreated first image and the second image
First piece caching, second piece caching and third block caching；Second piece of caching, is used for data buffering；
The FPGA is also used to obtain from third block caching described by pretreated first image and the second image.
Further, the FPGA is also used to carry out polar curve by pretreated first image and the second image to described
Correction.
Further, the FPGA, it is described by the first picture of each of pretreated first image specifically for determination
Vegetarian refreshments is relative to the cost value by each second pixel in pretreated second image；According to each first pixel
The cost value of point determines the disparity map in five directions in left, upper, right, upper left and upper right respectively；
Wherein, the FPGA stores determining cost value in sequence, for determine it is left, upper left and upper right four
The disparity map in direction；Determining cost value is stored according to inverted order, for determining the disparity map of right direction.
Further, the FPGA is specifically used for determining parallel described by pretreated the according to preset quantity
The first pixel of each of one image is relative to the cost value by each second pixel in pretreated second image.
Further, the FPGA, be also used to by every a line disparity map of left, upper, upper left and upper right four direction according to
Sequence is stored；The odd-numbered line disparity map of the right direction is stored according to inverted order, even number line disparity map is in sequence
It is stored.
Further, the FPGA, is also used to carry out the depth map median filter process, and will that treated is deep
Degree figure is sent to the DSP.
On the other hand, the embodiment of the invention provides a kind of image processing methods, which comprises
Receive DSP send by pretreated first image and the second image；Wherein, the first image and second
Image is that binocular camera acquires and is sent to the DSP；The DSP carries out the first image and the second image pre-
Processing；
For each parallax directions, pass through pretreated first image and second to described in the parallax directions parallel
Image carries out Stereo matching, generates the disparity map in the parallax directions；According to each disparity map, depth map is determined.
Further, it is described receive DSP send after pretreated first image and the second image, for every
A parallax directions carry out three-dimensional by pretreated first image and the second image to described in the parallax directions parallel
Match, before generating the disparity map in the parallax directions, the method also includes:
Polar curve correction is carried out by pretreated first image and the second image to described.
Further, described to be directed to each parallax directions, parallel in the parallax directions to described by pretreated the
One image and the second image carry out Stereo matching, and the disparity map generated in the parallax directions includes:
Determine it is described by the first pixel of each of pretreated first image relative to pass through pretreated second
The cost value of each second pixel in image；According to the cost value of each first pixel, determine respectively it is left, upper,
Right, five directions in upper left and upper right disparity map；
Wherein, the FPGA stores determining cost value in sequence, for determine it is left, upper left and upper right four
The disparity map in direction；Determining cost value is stored according to inverted order, for determining the disparity map of right direction.
Further, the determination it is described by the first pixel of each of pretreated first image relative to warp
The cost value for crossing each second pixel in pretreated second image includes:
According to preset quantity, determine parallel described by the first pixel phase of each of pretreated first image
For the cost value by each second pixel in pretreated second image.
Further, after the disparity map for determining five directions in left, upper, right, upper left and upper right respectively, the method is also
Include:
Every a line disparity map of left, upper, upper left and upper right four direction is stored in sequence；By the right
To odd-numbered line disparity map stored according to inverted order, even number line disparity map is stored in sequence.
Further, after the determining depth map, the method also includes:
Median filter process is carried out to the depth map, and depth map is sent to the DSP by treated.
The embodiment of the invention provides a kind of image processing apparatus, described device includes:
Receiving module, for receive DSP transmission by pretreated first image and the second image；Wherein, described
One image and the second image are that binocular camera acquires and is sent to the DSP；The DSP is to the first image and
Two images are pre-processed；
Determining module, for being directed to each parallax directions, parallel in the parallax directions to described by pretreated the
One image and the second image carry out Stereo matching, generate the disparity map in the parallax directions；According to each disparity map, determine deep
Degree figure.
The embodiment of the invention provides a kind of electronic equipment, including processor, communication interface, memory and communication bus,
Wherein, processor, communication interface, memory complete mutual communication by communication bus；
Memory, for storing computer program；
Processor when for executing the program stored on memory, realizes method and step described in any of the above embodiments.
The embodiment of the invention provides a kind of computer readable storage medium, the computer readable storage medium memory
Computer program is contained, the computer program realizes method and step described in any of the above embodiments when being executed by processor.
It is described the embodiment of the invention provides a kind of image processing system, method, apparatus, electronic equipment and storage medium
System includes binocular camera, Digital Signal Processing DSP and on-site programmable gate array FPGA；The binocular camera, is used for
The first image and the second image of acquisition are sent to the DSP；The DSP, for the first image and the second image into
Row pretreatment；The FPGA, for being directed to each parallax directions, parallel to by pretreated first in the parallax directions
Image and the second image carry out Stereo matching, generate the disparity map in the parallax directions；According to each disparity map, depth is determined
Figure.
Since in embodiments of the present invention, FPGA determines the disparity map in each parallax directions parallel；And then according to each
Disparity map determines depth map.The disparity map first successively determined in each parallax directions is avoided, then determines depth map band again
The latency issue come, it is thus determined that the real-time of depth map is preferable.
Detailed description of the invention
To describe the technical solutions in the embodiments of the present invention more clearly, required in being described below to embodiment
The attached drawing used is briefly introduced, it should be apparent that, drawings in the following description are only some embodiments of the invention, right
For those skilled in the art, without creative efforts, it can also be obtained according to these attached drawings
Other attached drawings.
Fig. 1 is image processing system schematic diagram in the prior art；
Fig. 2 is image processing system schematic diagram provided in an embodiment of the present invention；
Fig. 3 is the schematic diagram of mark information in image provided in an embodiment of the present invention；
Fig. 4 is that three pieces of cachings provided in an embodiment of the present invention read and write schematic diagram；
Fig. 5 is five direction schematic diagrams of each pixel provided in an embodiment of the present invention；
Fig. 6 is the process schematic of the final cost value of determination provided in an embodiment of the present invention；
Fig. 7 is the cost value schematic diagram of the first image provided in an embodiment of the present invention and the second image；
Fig. 8 is that cost value provided in an embodiment of the present invention stores schematic diagram；
Fig. 9 is the circuit diagram of parallel computation cost value provided in an embodiment of the present invention；
Figure 10 is that the disparity map of the first row provided in an embodiment of the present invention stores schematic diagram；
Figure 11 is that the disparity map of the second row provided in an embodiment of the present invention stores schematic diagram；
Figure 12 is image processing system flow chart provided in an embodiment of the present invention；
Figure 13 is image processing process schematic diagram provided in an embodiment of the present invention；
Figure 14 is image processing apparatus structural schematic diagram provided in an embodiment of the present invention；
Figure 15 is electronic devices structure schematic diagram provided in an embodiment of the present invention.
Specific embodiment
The present invention will be describe below in further detail with reference to the accompanying drawings, it is clear that described embodiment is only this
Invention a part of the embodiment, instead of all the embodiments.Based on the embodiments of the present invention, those of ordinary skill in the art
All other embodiment obtained without making creative work, shall fall within the protection scope of the present invention.
Fig. 2 is image processing system schematic diagram provided in an embodiment of the present invention, and the system comprises binocular cameras
11, Digital Signal Processing DSP12 and on-site programmable gate array FPGA 13；
The binocular camera 11, for sending the first image and the second image of acquisition to the DSP12；
The DSP12, for being pre-processed to the first image and the second image；
The FPGA13, for being directed to each parallax directions, parallel to by pretreated first in the parallax directions
Image and the second image carry out Stereo matching, generate the disparity map in the parallax directions；According to each disparity map, depth is determined
Figure.
As shown in Fig. 2, image processing system includes binocular camera 11, binocular camera 11 includes the camera shooting of left and right two
Head, two cameras in left and right acquire the first image and the second image respectively.Wherein it is possible to which the image that left camera is acquired is made
For the first image, the image that right camera is acquired is as the second image, and the image that left camera can also be acquired is as
Two images, the image that right camera is acquired is as the first image.
Image processing system further includes DSP12 and FPGA13, and the first image and the second image of binocular camera acquisition can
To be sent to DSP12, DSP12 pre-processes the first image and the second image, then will pass through pretreated first image
FPGA13 is sent to the second image.DSP12 to the first image and the second image carry out pretreatment can be to the first image and
Second image carries out hot-tempered processing, ISP processing etc..
FPGA13 is received after pretreated first image and the second image, while right in each parallax directions
First image and the second image carry out Stereo matching, determine the disparity map in each parallax directions, and then according to each parallax
Figure, determines depth map.Wherein, each parallax directions can be direction, upper left, left direction, upper right and right direction
Deng.
Since in embodiments of the present invention, FPGA determines the disparity map in each parallax directions parallel；And then according to each
Disparity map determines depth map.The disparity map first successively determined in each parallax directions is avoided, then determines depth map band again
The latency issue come, it is thus determined that the real-time of depth map is preferable.
In order to accurately determine depth map, on the basis of the above embodiments, in embodiments of the present invention, the DSP12,
Specifically for the mark information according to preset pixel in the first image and the second image, the first image is judged
Whether it is the left figure and right figure of identical frame number with the second image, if so, the first image and the second image are sent to institute
State FPGA13.
Since the speed of 11 two thecamera head images of binocular camera may be variant, what DSP12 was received
First image and the second image be possible to be not same frame number two field pictures, in order to which the depth map for guaranteeing determining is accurate, first
It needs when determining disparity map, is determined for the first image of same frame and the second image.Therefore in the embodiment of the present invention
In, the first image and the second image are pre-processed other than carrying out hot-tempered processing, ISP processing, can also be judgement the
One image and the second image whether be same frame left image and right image.
Specifically, preset pixel can carry mark in the first image and the second image that binocular camera 11 acquires
Remember that information, the mark information are used to indicate the frame number and left and right figure of image.
Specifically, binocular camera 11 can in the first image and the second image from the upper left corner first pixel
The upper mark information for carrying mark information first frame image is 1, and the mark information of the second frame image is 2, and so on.DSP12
After receiving the first image and the second image, according to the label carried on first pixel in the first image and the second image
Information, judge the first image and the second image whether be identical frame number image.
In addition, the mark information that preset pixel carries in the first image and the second image also can indicate that image is
Left figure or right figure.For example, mark information is a, image is left figure, and mark information b, image is right figure.
It should be noted that being used to indicate the mark information of image frame number and being used to indicate image is left figure or right figure
Mark information can be same information, for example, the mark information carry with first pixel, then, mark information a1
Image be first frame left figure, the image that mark information is a2 is second frame left figure, etc.；The image that mark information is b1 is
First frame right figure, the image that mark information is b2 are second frame right figure, etc..Certainly, it is used to indicate the label letter of image frame number
It ceases and is used to indicate image and be also possible to carry two labels letter in different pixels point for the mark information of left figure or right figure
Breath, for example, first pixel of image carries the mark information that mark image is left figure or right figure, second of image
Pixel carries the mark information of mark image frame number.As long as DSP12 is according to preset picture in the first image and the second image
The mark information of vegetarian refreshments, can judge the first image and the second image whether be identical frame number left figure and right figure.Only
Have when DSP12 judges the first image and the second image for the left figure of identical frame number and right figure, just by the first image and the second image
It is sent to the FPGA13, does not otherwise send image, until the first image received and the second image meet condition and just send out
It send.
Fig. 3 is the schematic diagram of mark information in image, as shown in figure 3, the mark information of first pixel is used to indicate
The mark information of left figure or right figure, second pixel point is used to indicate the frame number of image.
Due in embodiments of the present invention, DSP12 is according to the label of preset pixel in the first image and the second image
Information when judging the first image and the second image for the left figure of identical frame number and right figure, is just sent out the first image and the second image
It send to FPGA13.Therefore it can guarantee the determining disparity map of FPGA13 and depth map is left figure and the right side for identical frame number
Figure is determining, it is thus determined that depth map it is more acurrate.
In order to further increase the timeliness of image procossing, on the basis of the various embodiments described above, in the embodiment of the present invention
In, it include three pieces of cachings in the FPGA13；
The DSP12, specifically for will pass through pretreated first image and the second image be successively sent to it is described
First piece of caching, second piece of caching and third block caching in FPGA13；Second piece of caching, is used for data buffering；
The FPGA13 is also used to obtain from third block caching described by pretreated first image and the second figure
Picture.
It in embodiments of the present invention, include three pieces of cachings in FPGA13, DSP12 is specifically used for will be by pretreated the
One image and the second image are successively sent to first piece of caching, second piece of caching and third block caching in FPGA13.FPGA13
In every piece cache and will access the left figure and right figure of same frame number, if can not find the identical left figure of frame number and right figure, with regard to one
The caching of direct write current block just writes one piece of caching until finding the identical left figure of frame number and right figure.FPGA13 is used for from the
It is obtained in three pieces of cachings and passes through pretreated first image and the second image, and carry out the process of subsequent determining disparity map.
It can guarantee that first piece of caching passes through pretreated first image and the second image for DSP12 write-in in this way, the
Three pieces of cachings are read for FPGA13 passes through pretreated first image and the second image, and second piece of caching is for by pre- place
First image of reason and the buffering of the second image transmitting.For a video flowing, use it is provided in an embodiment of the present invention for
FPGA13 opens up the mode of three pieces of cachings, may be implemented to be written and read while image, so as to further increase image
The timeliness of processing.
Fig. 4 is that three pieces of cachings read and write schematic diagrames, as shown in figure 4, include in FPGA13 three pieces cache be respectively buff0,
Buff1 and buff2, for a video flowing, buff0 is write-in point write_point, and buff2 is to read point read_
point。
In order to further make determining depth map more acurrate, on the basis of the various embodiments described above, in the embodiment of the present invention
In, the FPGA13 is also used to carry out polar curve correction by pretreated first image and the second image to described.
The parameter of the available binocular camera 11 obtained by calibrating in advance of FPGA13, reads simultaneously by pretreated
After first image and the second image, polar curve school is carried out to by pretreated first image and the second image using the parameter
Just.Then the first image of polar curve correction completion and the second image are write back into external memory.It is same after polar curve corrects
The ordinate phase of environment o'clock ordinate of corresponding pixel and pixel corresponding in the second image in the first image
Together.Ordinate herein is the ordinate in image coordinate system.
By the first image and the second image that polar curve corrects, the parallax in determining each parallax directions can be made
Scheme it is more acurrate, further such that determine depth map it is more acurrate.Wherein, polar curve correction is carried out to the first image and the second image
Process belong to the prior art, no longer the process is repeated herein.
In order to make to determine that the disparity map in each parallax directions is more acurrate, on the basis of the various embodiments described above, in this hair
In bright embodiment, the FPGA13 is described by the first pixel of each of pretreated first image specifically for determination
Relative to the cost value by each second pixel in pretreated second image；According to each first pixel
Cost value determines the disparity map in five directions in left, upper, right, upper left and upper right respectively；
Wherein, the FPGA stores determining cost value in sequence, for determine it is left, upper left and upper right four
The disparity map in direction；Determining cost value is stored according to inverted order, for determining the disparity map of right direction.
In embodiments of the present invention, to be illustrated for determining the disparity map in five directions, as shown in figure 5, each picture
Five directions of vegetarian refreshments are left, upper, right, upper left and upper right.The disparity map in each direction is mutually indepedent parallel processing
, it can be the disparity map for determining the quantity directions such as 6,7,8 with Quick Extended on the basis of the embodiment of the present invention.
Before determining the disparity map in each parallax directions, it is necessary first to determine by pretreated first image
Each of the first pixel relative to the cost value by each second pixel in pretreated second image.
Specifically, it is first determined by the corresponding first gradient image of pretreated first image and by pretreated
Corresponding second gradient image of second image, when determining gradient image, for each pixel in each original image, meter
The difference for calculating the pixel value for the pixel being located on the right side of the pixel and the pixel value of the pixel, obtains the pixel right
The pixel value in gradient image answered, the pixel value in gradient image corresponding for the pixel of the rightmost side can all be set
It is set to 0, it can also be using the pixel value of the pixel of the rightmost side as the corresponding pixel value in gradient image.
Fig. 6 is the process schematic for determining final cost value, is being determined by each of pretreated first image the
One pixel is relative to by when the cost value of each second pixel, needing to determine the in pretreated second image respectively
Cost value of each pixel relative to each pixel in the second gradient image in one gradient image, and by pre- place
The first pixel of each of first image of reason is relative to the generation by each second pixel in pretreated second image
Corresponding cost value, is then added, obtains final cost value by value.In order to improve real-time, first gradient image is calculated
In cost value of each pixel relative to each pixel in the second gradient image, and pass through pretreated first figure
The first pixel of each of picture is relative to the process by the cost value of each second pixel in pretreated second image
It is parallel processing.
The processing of cost value is to be calculated according to row, and the process of gradient image and original image calculating cost value is complete
It is exactly the same, it introduces and is calculated by each of pretreated first image the first pixel relative to by pre-processing at this
The second image in each second pixel cost value process.
Assuming that being M*N by the resolution ratio of pretreated first image and the second image, according to beginning match point parameter
Min_disparity takes out the pixel U by pretreated first image, while adjacent two that take out the pixel or so
A pixel UrAnd Ul, U and U are then calculated simultaneouslyr, U and UlAverage value obtain U0And U1.Then compare U, U0、U1Obtain three
Between maximum value and minimum value Umax、Umin.Same processing is done to by the pixel V in pretreated second image, is asked
Maximum value and minimum value V outmax、 Vmin.Then simultaneously according to formula max (Vmin-Umin,max(0,Umin-Vmax))、 max
(Vmin-Umax,max(0,Umax-Vmax)) find out C0And C1.Finally take C0And C1Cost value of the minimum value as current pixel point,
Need to match num_disparit y pixel of right figure by each pixel in pretreated first image simultaneously, so through
The cost value for crossing each pixel in pretreated first image has num_disparity, by pretreated first image
It is as shown in Figure 7 with the cost value of the second image.
In addition, since image data is all to be sent by row, and every row is direction from left to right.In order to just
In the calculating of subsequent disparity map, as shown in figure 8, FPGA13 stores determining cost value in sequence, left for determination,
The disparity map of upper left and upper right four direction；Determining cost value is stored according to inverted order, for determining the parallax of right direction
Figure.
There is corresponding parallax formula in five directions in left, upper, right, upper left and upper right, are determining by pretreated
The first pixel of each of one image relative to the cost value by each second pixel in pretreated second image it
Afterwards, cost value is substituted into the corresponding parallax formula in each direction respectively, can determines left, upper, right, upper left and upper right five
Then the disparity map in each direction in direction can be added the disparity map in each direction, determine by pretreated the
The disparity map of one image and the second image.Then according to the focal length of binocular camera, binocular camera two camera lens optical centers it
Between distance and the disparity map of the first image and the second image determined, can determine by pretreated first image and
The depth map of second image.Specifically, formula depth=(f*baseline)/disp can be used；Determine depth map, formula
In, f is the focal length of binocular camera, and baseline is the distance between two camera lens optical centers of binocular camera, disp the
The disparity map of one image and the second image, depth are the depth map of the first image and the second image.
Since the cost value of each pixel in the first image has num_disparity, if using serial processing
Mode is difficult to meet requirement of real-time.Therefore, in order to further increase the real-time of image procossing, in the various embodiments described above
On the basis of, in embodiments of the present invention, the FPGA13 is specifically used for determining parallel described by pre- according to preset quantity
The first pixel of each of first image of processing is relative to by each second pixel in pretreated second image
Cost value.
Parallel processing quantity can be preset in FPGA13, it is parallel true according to preset quantity when carrying out cost value calculating
Surely by the first pixel of each of pretreated first image relative to by pretreated second image each second
The cost value of pixel.It is serial process on the time during entire determining cost value if preset quantity is K
1/K。
Fig. 9 is the circuit diagram of parallel computation cost value, as shown in figure 9, preset quantity is K, for each pixel,
K cost value of the pixel can be calculated simultaneously.Specifically, the pixel U by pretreated first image is taken out,
Two adjacent pixel U of the pixel or so are taken out simultaneouslyrAnd Ul, U and U are then calculated simultaneouslyr, U and UlAverage value obtain
To U0And U1.Then compare U, U0、U1Obtain the maximum value and minimum value U between threemax、Umin.To by pretreated the
Pixel V in two images does same processing, finds out maximum value and minimum value Vmax、Vmin.Then simultaneously according to formula max
(Vmin-Umin, max (0, Umin-Vmax))、max(Vmin-Umax,max(0,Umax-Vmax)) find out C0And C1.Finally take C0And C1Most
Cost value of the small value as current pixel point.The treatment process of pixel V1 to Vk is identical.
Since in embodiments of the present invention, FPGA passes through pretreated first image according to preset quantity, parallel determine
Each of the first pixel relative to the cost value by each second pixel in pretreated second image.It therefore can
To further increase the real-time of image procossing.
It is in embodiments of the present invention, described on the basis of the various embodiments described above in order to save memory resource
FPGA13 is also used in sequence store every a line disparity map of left, upper, upper left and upper right four direction；It will be described
The odd-numbered line disparity map of right direction is stored according to inverted order, and even number line disparity map is stored in sequence.
In embodiments of the present invention, the left side, upper left of the first row, upper, upper right four direction are the matching costs according to sequence
Value carries out disparity computation, is then added storage into row buffer；And right direction is carried out according to the matching cost value of inverted order
Disparity map calculates, and the result of calculating also is stored in row caching.Therefore, the disparity map storage of the first row is as shown in Figure 10, left,
Upper left, upper, upper right four direction parallax can store in the same row buffer, but the parallax of right direction needs are deposited
Storage is in other row buffer, also, the parallax of right direction is inverted order storage.
In the disparity map for calculating the second row, the first row disparity map in right disparity map, buffer base address etc. are first read at this time
The second row disparity map SPN is stored in current address simultaneously in (N-1) * num_disparity/, K1..k.Value, then according to
Secondary base address (N-P) * num_disparity/K for reading other pixels (P) of the first row, while the second row is stored in the address
SP (N-P)1...kValue.The disparity map of second row is stored as shown in Figure 11, a left side, upper left, upper, upper right four direction parallax
It is stored in the same row buffer, but the parallax of right direction needs to be stored in other row buffer, also, right
To parallax be sequential storage.
In the disparity map for calculating the third line, the second row disparity map in right disparity map, buffer base address etc. are first read at this time
In (1-1) * num_disparity/K, at the same in current address store the third line disparity map SPN1...k value, then
Base address (P-1) * num_disparity/K of other pixels (P) of the second row is successively read, while storing third in the address
Capable SP (P-1)1...kValue.The disparity map storage of the third line is identical as Figure 10.
From reasoning above it is also seen that a left side, upper left, upper, upper right four direction disparity map an and shared storage
Device, while right direction disparity map odd-numbered line and even number line are alternately to store, odd-numbered line disparity map is stored according to inverted order,
Even number line disparity map is stored in sequence, in this way for right direction disparity map, a memory is also only needed, thus pole
Memory resource has been saved greatly.
In order to improve the quality of determining depth map, on the basis of the various embodiments described above, in embodiments of the present invention,
FPGA13 is also used to carry out the depth map median filter process, and depth map is sent to the DSP12 by treated.
After FPGA13 determines depth map, median filter process is carried out to depth map according to preset window size, in
Value filtering processing is a centrally located preset window namely for each pixel in depth map with the pixel, then
Sequence sequence by pixel in window according to from big to small, or from small to large, take median to the pixel value of the pixel into
Row updates.Preset window size can be 3x3 window, 5x5 window etc..Finally by treated, depth map is sent to
DSP12 carries out subsequent transmission or processing to depth map by DSP12.
Due in embodiments of the present invention, FPGA carries out median filter process to depth map, and will treated depth map
It is sent to DSP, so as to improve the quality of depth map.
Figure 12 is image processing system flow chart provided in an embodiment of the present invention, and as shown in figure 12, DSP receives binocular
After the first image and the second image that video camera is sent, pre-process then deposit FPGA to the first image and the second image
In caching DDR, the first image and the second image cached in DDR be stored in external memory, and FPGA obtains from external memory again
The first image and the second image are taken, carries out polar curve correction respectively, the first image and the deposit of the second image after polar curve correction are external
Memory, FPGA obtain the first image and the second image after polar curve correction from external memory again, by Stereo matching, really
Determine depth map.Wherein, which can be half global Stereo matching.Depth map is stored in external memory by FPGA, warp
DSP is sent to by external memory.
Figure 13 be image processing process schematic diagram provided in an embodiment of the present invention, the process the following steps are included:
S101: receive DSP send by pretreated first image and the second image；Wherein, the first image and
Second image is that binocular camera acquires and is sent to the DSP；The DSP to the first image and the second image into
Row pretreatment.
S102: being directed to each parallax directions, parallel in the parallax directions to described by pretreated first image and
Second image carries out Stereo matching, generates the disparity map in the parallax directions；According to each disparity map, depth map is determined.
Image processing system includes binocular camera, and binocular camera includes two cameras in left and right, the camera shooting of left and right two
Head acquires the first image and the second image respectively.Wherein it is possible to the image that left camera is acquired is as the first image, it will be right
The image of camera acquisition is as the second image, and the image that left camera can also be acquired is as the second image, by right camera shooting
The image of head acquisition is as the first image.
Image processing system further includes DSP and FPGA, and the first image and the second image of binocular camera acquisition can be sent out
It send to DSP, DSP12 and the first image and the second image is pre-processed, then will pass through pretreated first image and second
Image is sent to FPGA13.DSP12, which carries out pretreatment to the first image and the second image, can be to the first image and the second figure
As carrying out hot-tempered processing, ISP processing etc..
FPGA is received by after pretreated first image and the second image, while to the in each parallax directions
One image and the second image carry out Stereo matching, determine the disparity map in each parallax directions, and then according to each parallax
Figure, determines depth map.Wherein, each parallax directions can be direction, upper left, left direction, upper right and right direction
Deng.
Since in embodiments of the present invention, FPGA determines the disparity map in each parallax directions parallel；And then according to each
Disparity map determines depth map.The disparity map first successively determined in each parallax directions is avoided, then determines depth map band again
The latency issue come, it is thus determined that the real-time of depth map is preferable.
In order to accurately determine depth map, on the basis of the above embodiments, in embodiments of the present invention, the reception DSP
Transmission after pretreated first image and the second image, for each parallax directions, parallel in the parallax directions
Carry out Stereo matching by pretreated first image and the second image to described, generate disparity map in the parallax directions it
Before, the method also includes:
Polar curve correction is carried out by pretreated first image and the second image to described.
The parameter of the available binocular camera obtained by calibrating in advance of FPGA, reads simultaneously by pretreated first
After image and the second image, polar curve correction is carried out to by pretreated first image and the second image using the parameter.So
The first image of polar curve correction completion and the second image are write back into external memory afterwards.After polar curve corrects, same environment point
The ordinate of corresponding pixel is identical as the ordinate of pixel corresponding in the second image in the first image.Herein
Ordinate be image coordinate system in ordinate.
By the first image and the second image that polar curve corrects, the parallax in determining each parallax directions can be made
Scheme it is more acurrate, further such that determine depth map it is more acurrate.Wherein, polar curve correction is carried out to the first image and the second image
Process belong to the prior art, no longer the process is repeated herein.
In order to make to determine that the disparity map in each parallax directions is more acurrate, on the basis of the various embodiments described above, in this hair
It is described to be directed to each parallax directions in bright embodiment, pass through pretreated first image to described in the parallax directions parallel
Stereo matching is carried out with the second image, the disparity map generated in the parallax directions includes:
Determine it is described by the first pixel of each of pretreated first image relative to pass through pretreated second
The cost value of each second pixel in image；According to the cost value of each first pixel, determine respectively it is left, upper,
Right, five directions in upper left and upper right disparity map；
Wherein, the FPGA stores determining cost value in sequence, for determine it is left, upper left and upper right four
The disparity map in direction；Determining cost value is stored according to inverted order, for determining the disparity map of right direction.
In embodiments of the present invention, to be illustrated for determining the disparity map in five directions, as shown in figure 5, each picture
Five directions of vegetarian refreshments are left, upper, right, upper left and upper right.The disparity map in each direction is mutually indepedent parallel processing
, it can be the disparity map for determining the quantity directions such as 6,7,8 with Quick Extended on the basis of the embodiment of the present invention.
Before determining the disparity map in each parallax directions, it is necessary first to determine by pretreated first image
Each of the first pixel relative to the cost value by each second pixel in pretreated second image.
Specifically, it is first determined by the corresponding first gradient image of pretreated first image and by pretreated
Corresponding second gradient image of second image, when determining gradient image, for each pixel in each original image, meter
The difference for calculating the pixel value for the pixel being located on the right side of the pixel and the pixel value of the pixel, obtains the pixel right
The pixel value in gradient image answered, the pixel value in gradient image corresponding for the pixel of the rightmost side can all be set
It is set to 0, it can also be using the pixel value of the pixel of the rightmost side as the corresponding pixel value in gradient image.
Fig. 6 is the process schematic for determining final cost value, is being determined by each of pretreated first image the
One pixel is relative to by when the cost value of each second pixel, needing to determine the in pretreated second image respectively
Cost value of each pixel relative to each pixel in the second gradient image in one gradient image, and by pre- place
The first pixel of each of first image of reason is relative to the generation by each second pixel in pretreated second image
Corresponding cost value, is then added, obtains final cost value by value.In order to improve real-time, first gradient image is calculated
In cost value of each pixel relative to each pixel in the second gradient image, and pass through pretreated first figure
The first pixel of each of picture is relative to the process by the cost value of each second pixel in pretreated second image
It is parallel processing.
The processing of cost value is to be calculated according to row, and the process of gradient image and original image calculating cost value is complete
It is exactly the same, it introduces and is calculated by each of pretreated first image the first pixel relative to by pre-processing at this
The second image in each second pixel cost value process.
Assuming that being M*N by the resolution ratio of pretreated first image and the second image, according to beginning match point parameter
Min_disparity takes out the pixel U by pretreated first image, while adjacent two that take out the pixel or so
A pixel UrAnd Ul, U and U are then calculated simultaneouslyr, U and UlAverage value obtain U0And U1.Then compare U, U0、U1Obtain three
Between maximum value and minimum value Umax、Umin.Same processing is done to by the pixel V in pretreated second image, is asked
Maximum value and minimum value V outmax、 Vmin.Then simultaneously according to formula max (Vmin-Umin,max(0,Umin-Vmax))、 max
(Vmin-Umax,max(0,Umax-Vmax)) find out C0And C1.Finally take C0And C1Cost value of the minimum value as current pixel point,
Need to match num_disparit y pixel of right figure by each pixel in pretreated first image simultaneously, so through
The cost value for crossing each pixel in pretreated first image has num_disparity, by pretreated first image
It is as shown in Figure 7 with the cost value of the second image.
In addition, since image data is all to be sent by row, and every row is direction from left to right.In order to just
In the calculating of subsequent disparity map, as shown in figure 8, FPGA13 stores determining cost value in sequence, left for determination,
The disparity map of upper left and upper right four direction；Determining cost value is stored according to inverted order, for determining the parallax of right direction
Figure.
There is corresponding parallax formula in five directions in left, upper, right, upper left and upper right, are determining by pretreated
The first pixel of each of one image relative to the cost value by each second pixel in pretreated second image it
Afterwards, cost value is substituted into the corresponding parallax formula in each direction respectively, can determines left, upper, right, upper left and upper right five
Then the disparity map in each direction in direction can be added the disparity map in each direction, determine by pretreated the
The disparity map of one image and the second image.Then according to the focal length of binocular camera, binocular camera two camera lens optical centers it
Between distance and the disparity map of the first image and the second image determined, can determine by pretreated first image and
The depth map of second image.Specifically, formula depth=(f*baseline)/disp can be used；Determine depth map, formula
In, f is the focal length of binocular camera, and baseline is the distance between two camera lens optical centers of binocular camera, disp the
The disparity map of one image and the second image, depth are the depth map of the first image and the second image.
Since the cost value of each pixel in the first image has num_disparity, if using serial processing
Mode is difficult to meet requirement of real-time.Therefore, in order to further increase the real-time of image procossing, in the various embodiments described above
On the basis of, in embodiments of the present invention, the determination is described by the first pixel phase of each of pretreated first image
Include: for the cost value by each second pixel in pretreated second image
According to preset quantity, determine parallel described by the first pixel phase of each of pretreated first image
For the cost value by each second pixel in pretreated second image.
Parallel processing quantity can be preset in FPGA, it is parallel to determine according to preset quantity when carrying out cost value calculating
By the first pixel of each of pretreated first image relative to by each second picture in pretreated second image
The cost value of vegetarian refreshments.It is the 1/ of serial process on the time during entire determining cost value if preset quantity is K
K。
Fig. 9 is the circuit diagram of parallel computation cost value, as shown in figure 9, preset quantity is K, for each pixel,
K cost value of the pixel can be calculated simultaneously.Specifically, the pixel U by pretreated first image is taken out,
Two adjacent pixel U of the pixel or so are taken out simultaneouslyrAnd Ul, U and U are then calculated simultaneouslyr, U and UlAverage value obtain
To U0And U1.Then compare U, U0、U1Obtain the maximum value and minimum value U between threemax、Umin.To by pretreated the
Pixel V in two images does same processing, finds out maximum value and minimum value Vmax、Vmin.Then simultaneously according to formula max
(Vmin-Umin, max (0, Umin-Vmax))、max(Vmin-Umax,max(0,Umax-Vmax)) find out C0And C1.Finally take C0And C1Most
Cost value of the small value as current pixel point.The treatment process of pixel V1 to Vk is identical.
Since in embodiments of the present invention, FPGA passes through pretreated first image according to preset quantity, parallel determine
Each of the first pixel relative to the cost value by each second pixel in pretreated second image.It therefore can
To further increase the real-time of image procossing.
In order to save memory resource, on the basis of the various embodiments described above, in embodiments of the present invention, determine respectively
After the disparity map in five directions in left, upper, right, upper left and upper right, the method also includes:
Every a line disparity map of left, upper, upper left and upper right four direction is stored in sequence；By the right
To odd-numbered line disparity map stored according to inverted order, even number line disparity map is stored in sequence.
In embodiments of the present invention, the left side, upper left of the first row, upper, upper right four direction are the matching costs according to sequence
Value carries out disparity computation, is then added storage into row buffer；And right direction is carried out according to the matching cost value of inverted order
Disparity map calculates, and the result of calculating also is stored in row caching.Therefore, the disparity map storage of the first row is as shown in Figure 10, left,
Upper left, upper, upper right four direction parallax can store in the same row buffer, but the parallax of right direction needs are deposited
Storage is in other row buffer, also, the parallax of right direction is inverted order storage.
In the disparity map for calculating the second row, the first row disparity map in right disparity map, buffer base address etc. are first read at this time
In (N-1) * num_disparity/K, while the second row disparity map SPN is stored in current address1...kValue, then successively
Base address (N-P) * num_disparity/K of other pixels (P) of the first row is read, while storing the second row in the address
SP(N-P)1...kValue.As shown in figure 11, a left side, upper left, upper, upper right four direction parallax are deposited for the disparity map storage of second row
Storage is in the same row buffer, but the parallax of right direction needs to be stored in other row buffer, also, right direction
Parallax be sequential storage.
In the disparity map for calculating the third line, the second row disparity map in right disparity map, buffer base address etc. are first read at this time
In (1-1) * num_disparity/K, while the third line disparity map SPN is stored in current address1...kValue, then successively
Base address (P-1) * num_disparity/K of other pixels (P) of the second row is read, while storing the third line in the address
SP(P-1)1...kValue.The disparity map storage of the third line is identical as Fig. 9.
From reasoning above it is also seen that a left side, upper left, upper, upper right four direction disparity map an and shared storage
Device, while right direction disparity map odd-numbered line and even number line are alternately to store, odd-numbered line disparity map is stored according to inverted order,
Even number line disparity map is stored in sequence, in this way for right direction disparity map, a memory is also only needed, thus pole
Memory resource has been saved greatly.
In order to improve the quality of determining depth map, on the basis of the various embodiments described above, in embodiments of the present invention,
After the determining depth map, the method also includes:
Median filter process is carried out to the depth map, and depth map is sent to the DSP by treated.
After FPGA determines depth map, median filter process, intermediate value are carried out to depth map according to preset window size
Filtering processing namely for each pixel in depth map, is a centrally located preset window with the pixel, then will
Pixel is according to from big to small in window, or sequence sequence from small to large, and median is taken to carry out the pixel value of the pixel
It updates.Preset window size can be 3x3 window, 5x5 window etc..Finally by treated, depth map is sent to DSP12,
Subsequent transmission or processing are carried out to depth map by DSP12.
Due in embodiments of the present invention, FPGA carries out median filter process to depth map, and will treated depth map
It is sent to DSP, so as to improve the quality of depth map.
Figure 14 is image processing apparatus structural schematic diagram provided in an embodiment of the present invention, and described device includes:
Receiving module 141, for receive DSP transmission by pretreated first image and the second image；Wherein, institute
It states the first image and the second image is that binocular camera acquires and is sent to the DSP；The DSP is to the first image
It is pre-processed with the second image；
Determining module 142 passes through pretreatment to described in the parallax directions parallel for being directed to each parallax directions
The first image and the second image carry out Stereo matching, generate the disparity map in the parallax directions；According to each disparity map, really
Determine depth map.
On the basis of the various embodiments described above, a kind of electronic equipment is additionally provided in the embodiment of the present invention, such as Figure 15 institute
Show, comprising: processor 151, communication interface 152, memory 153 and communication bus 154, wherein processor 151, communication interface
152, memory 153 completes mutual communication by communication bus 154；
It is stored with computer program in the memory 153, when described program is executed by the processor 151, so that
The processor 151 executes following steps:
Receive DSP send by pretreated first image and the second image；Wherein, the first image and second
Image is that binocular camera acquires and is sent to the DSP；The DSP carries out the first image and the second image pre-
Processing；
For each parallax directions, pass through pretreated first image and second to described in the parallax directions parallel
Image carries out Stereo matching, generates the disparity map in the parallax directions；According to each disparity map, depth map is determined.
Based on the same inventive concept, a kind of electronic equipment is additionally provided in the embodiment of the present invention, due to above-mentioned electronic equipment
The principle solved the problems, such as is similar to image processing method, therefore the implementation of above-mentioned electronic equipment may refer to the implementation of method, weight
Multiple place repeats no more.
Electronic equipment provided in an embodiment of the present invention is specifically as follows desktop computer, portable computer, intelligent hand
Machine, tablet computer, personal digital assistant (Personal Digital Assistant, PDA), network side equipment etc..
The communication bus that above-mentioned electronic equipment is mentioned can be Peripheral Component Interconnect standard (Peripheral Component
Interconnect, PCI) bus or expanding the industrial standard structure (Extended Industry Standard
Architecture, EISA) bus etc..The communication bus can be divided into address bus, data/address bus, control bus etc..For just
It is only indicated with a thick line in expression, figure, it is not intended that an only bus or a type of bus.
Communication interface 152 is for the communication between above-mentioned electronic equipment and other equipment.
Memory may include random access memory (Random Access Memory, RAM), also may include non-
Volatile memory (Non-Volatile Memory, NVM), for example, at least a magnetic disk storage.Optionally, memory is also
It can be at least one storage device for being located remotely from aforementioned processor.
Above-mentioned processor can be general processor, including central processing unit, network processing unit (Network
Processor, NP) etc.；It can also be digital signal processor (Digital Signal Processing, DSP), dedicated collection
At circuit, field programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete
Hardware component etc..
When processor executes the program stored on memory in embodiments of the present invention, realizes and receive the warp that DSP is sent
Cross pretreated first image and the second image；Wherein, the first image and the second image are that binocular camera acquisition is concurrent
It send to the DSP's；The DSP pre-processes the first image and the second image；For each parallax directions, and
Row carries out Stereo matching by pretreated first image and the second image to described in the parallax directions, generates the parallax
Disparity map on direction；According to each disparity map, depth map is determined.Since in embodiments of the present invention, FPGA determines every parallel
Disparity map in a parallax directions；And then according to each disparity map, depth map is determined.It avoids and first successively determines each parallax
Then disparity map on direction determines depth map bring latency issue, it is thus determined that the real-time of depth map is preferable again.
On the basis of the various embodiments described above, the embodiment of the invention also provides a kind of computers to store readable storage medium
Matter is stored with the computer program that can be executed by electronic equipment in the computer readable storage medium, when described program is in institute
It states when being run on electronic equipment, so that the electronic equipment realizes following steps when executing:
Receive DSP send by pretreated first image and the second image；Wherein, the first image and second
Image is that binocular camera acquires and is sent to the DSP；The DSP carries out the first image and the second image pre-
Processing；
For each parallax directions, pass through pretreated first image and second to described in the parallax directions parallel
Image carries out Stereo matching, generates the disparity map in the parallax directions；According to each disparity map, depth map is determined.
Based on the same inventive concept, a kind of computer readable storage medium is additionally provided in the embodiment of the present invention, due to place
The principle and image procossing that reason device is solved the problems, such as in the computer program stored on executing above-mentioned computer readable storage medium
Method is similar, therefore processor can join in the implementation for the computer program for executing above-mentioned computer-readable recording medium storage
The implementation of square method, overlaps will not be repeated.
Above-mentioned computer readable storage medium can be any usable medium that the processor in electronic equipment can access
Or data storage device, including but not limited to magnetic storage such as floppy disk, hard disk, tape, magneto-optic disk (MO) etc., optical storage
Device such as CD, DVD, BD, HVD etc. and semiconductor memory such as ROM, EPROM, EEPROM, nonvolatile memory (NAND
FLASH), solid state hard disk (SSD) etc..
Computer program, computer program are provided in the computer readable storage medium provided in embodiments of the present invention
When being executed by processor realize receive DSP send by pretreated first image and the second image；Wherein, described first
Image and the second image are that binocular camera acquires and is sent to the DSP；The DSP is to the first image and second
Image is pre-processed；For each parallax directions, pass through pretreated first image to described in the parallax directions parallel
Stereo matching is carried out with the second image, generates the disparity map in the parallax directions；According to each disparity map, depth map is determined.By
In in embodiments of the present invention, FPGA determines the disparity map in each parallax directions parallel；And then according to each disparity map, determine
Depth map.The disparity map first successively determined in each parallax directions is avoided, then determines that the delay of depth map bring is asked again
Topic, it is thus determined that the real-time of depth map is preferable.
It is described the embodiment of the invention provides a kind of image processing system, method, apparatus, electronic equipment and storage medium
System includes binocular camera and Digital Signal Processing DSP, the system also includes: on-site programmable gate array FPGA；It is described
Binocular camera, for sending the first image and the second image of acquisition to the DSP；The DSP is used for described first
Image and the second image are sent to the FPGA；The FPGA, for being directed to each parallax directions, parallel in the parallax directions
On Stereo matching is carried out to the first image and the second image, generate the disparity map in the parallax directions；To each disparity map
It is registrated, determines depth map.
Since in embodiments of the present invention, FPGA determines the disparity map in each parallax directions parallel；And then to each view
Poor figure is registrated, and determines depth map.The disparity map first successively determined in each parallax directions is avoided, is then determined again deep
Figure bring latency issue is spent, it is thus determined that the real-time of depth map is preferable.
The present invention be referring to according to the method for the embodiment of the present invention, the process of equipment (system) and computer program product
Figure and/or block diagram describe.It should be understood that can be realized by computer program instructions each in flowchart and/or the block diagram
The combination of process and/or box in process and/or box and flowchart and/or the block diagram.It can provide these computers
Processor of the program instruction to general purpose computer, special purpose computer, Embedded Processor or other programmable data processing devices
To generate a machine, so that being generated by the instruction that computer or the processor of other programmable data processing devices execute
For realizing the function of being specified in one or more flows of the flowchart and/or one or more blocks of the block diagram
Device.
These computer program instructions, which may also be stored in, is able to guide computer or other programmable data processing devices with spy
Determine in the computer-readable memory that mode works, so that instruction stored in the computer readable memory generation includes
The manufacture of command device, the command device are realized in one box of one or more flows of the flowchart and/or block diagram
Or the function of being specified in multiple boxes.
These computer program instructions also can be loaded onto a computer or other programmable data processing device, so that
Series of operation steps are executed on computer or other programmable devices to generate computer implemented processing, thus calculating
The instruction executed on machine or other programmable devices is provided for realizing in one or more flows of the flowchart and/or side
The step of function of being specified in block diagram one box or multiple boxes.
Although preferred embodiments of the present invention have been described, once a person skilled in the art knows basic wounds
The property made concept, then additional changes and modifications may be made to these embodiments.It is wrapped so the following claims are intended to be interpreted as
It includes preferred embodiment and falls into all change and modification of the scope of the invention.
Obviously, various changes and modifications can be made to the invention without departing from of the invention by those skilled in the art
Spirit and scope.In this way, if these modifications and changes of the present invention belongs to the model of the claims in the present invention and its equivalent technologies
Within enclosing, then the present invention is also intended to include these modifications and variations.
Claims (17)
1. a kind of image processing system, which is characterized in that the system comprises binocular camera, Digital Signal Processing DSP and show
Field programmable gate array FPGA；
The binocular camera, for sending the first image and the second image of acquisition to the DSP；
The DSP, for being pre-processed to the first image and the second image；
The FPGA, for being directed to each parallax directions, parallel in the parallax directions to by pretreated first image and
Second image carries out Stereo matching, generates the disparity map in the parallax directions；According to each disparity map, depth map is determined.
2. the system as claimed in claim 1, which is characterized in that the DSP is specifically used for according to the first image and second
The mark information of preset pixel in image, judge the first image and the second image whether be identical frame number left figure and
Right figure, if so, the first image and the second image are sent to the FPGA.
3. the system as claimed in claim 1, which is characterized in that include three pieces of cachings in the FPGA；
The DSP, specifically for will pass through pretreated first image and the second image be successively sent in the FPGA
One piece of caching, second piece of caching and third block caching；Second piece of caching, is used for data buffering；
The FPGA is also used to obtain from third block caching described by pretreated first image and the second image.
4. the system as claimed in claim 1, which is characterized in that the FPGA is also used to pass through pretreated first to described
Image and the second image carry out polar curve correction.
5. the system as claimed in claim 1, which is characterized in that the FPGA is specifically used for determining described by pretreatment the
The first pixel of each of one image is relative to the cost value by each second pixel in the second image of pretreatment；According to
The cost value of each first pixel determines the disparity map in five directions in left, upper, right, upper left and upper right respectively；
Wherein, the FPGA stores determining cost value in sequence, for determine it is left, upper left and upper right four direction
Disparity map；Determining cost value is stored according to inverted order, for determining the disparity map of right direction.
6. system as claimed in claim 5, which is characterized in that the FPGA is specifically used for according to preset quantity, parallel true
It is fixed described by pre-processing the first pixel of each of the first image relative to by the second image of pretreatment each second
The cost value of pixel.
7. system as claimed in claim 5, which is characterized in that the FPGA is also used to left, upper, upper left and upper right four
Every a line disparity map in direction is stored in sequence；The odd-numbered line disparity map of the right direction is deposited according to inverted order
Storage, even number line disparity map are stored in sequence.
8. the system as claimed in claim 1, which is characterized in that the FPGA is also used to carry out intermediate value filter to the depth map
Wave processing, and depth map is sent to the DSP by treated.
9. a kind of image processing method, which is characterized in that the described method includes:
Receive DSP send by pretreated first image and the second image；Wherein, the first image and the second image are
Binocular camera acquires and is sent to the DSP's；The DSP pre-processes the first image and the second image；
For each parallax directions, parallel in the parallax directions to it is described by pretreated first image and the second image into
Row Stereo matching generates the disparity map in the parallax directions；According to each disparity map, depth map is determined.
10. method as claimed in claim 9, which is characterized in that it is described receive DSP send by pretreated first image
After the second image, for each parallax directions, pass through pretreated first image to described in the parallax directions parallel
Stereo matching is carried out with the second image, before generating the disparity map in the parallax directions, the method also includes:
Polar curve correction is carried out by pretreated first image and the second image to described.
11. method as claimed in claim 9, which is characterized in that it is described to be directed to each parallax directions, parallel in the parallax directions
On carry out Stereo matching by pretreated first image and the second image to described, generate the disparity map packet in the parallax directions
It includes:
Determine it is described by the first pixel of each of pretreated first image relative to pass through pretreated second image
In each second pixel cost value；According to the cost value of each first pixel, left, upper, right, upper left is determined respectively
With the disparity map in five directions of upper right；
Wherein, the FPGA stores determining cost value in sequence, for determine it is left, upper left and upper right four direction
Disparity map；Determining cost value is stored according to inverted order, for determining the disparity map of right direction.
12. method as claimed in claim 11, which is characterized in that the determination is described by pretreated first image
Each first pixel includes: relative to the cost value by each second pixel in pretreated second image
According to preset quantity, determine parallel it is described by the first pixel of each of pretreated first image relative to warp
Cross the cost value of each second pixel in pretreated second image.
13. method as claimed in claim 11, which is characterized in that determine five directions in left, upper, right, upper left and upper right respectively
Disparity map after, the method also includes:
Every a line disparity map of left, upper, upper left and upper right four direction is stored in sequence；By the surprise of the right direction
Several rows of disparity maps are stored according to inverted order, and even number line disparity map is stored in sequence.
14. method as claimed in claim 9, which is characterized in that after the determining depth map, the method also includes:
Median filter process is carried out to the depth map, and depth map is sent to the DSP by treated.
15. a kind of image processing apparatus, which is characterized in that described device includes:
Receiving module, for receive DSP transmission by pretreated first image and the second image；Wherein, first figure
As and the second image be that binocular camera acquires and is sent to the DSP；The DSP is to the first image and the second image
It is pre-processed；
Determining module passes through pretreated first figure to described in the parallax directions parallel for being directed to each parallax directions
Picture and the second image carry out Stereo matching, generate the disparity map in the parallax directions；According to each disparity map, depth map is determined.
16. a kind of electronic equipment, which is characterized in that including processor, communication interface, memory and communication bus, wherein processing
Device, communication interface, memory complete mutual communication by communication bus；
Memory, for storing computer program；
Processor when for executing the program stored on memory, realizes the described in any item method steps of claim 9-14
Suddenly.
17. a kind of computer readable storage medium, which is characterized in that be stored with computer in the computer readable storage medium
Program realizes claim 9-14 described in any item method and steps when the computer program is executed by processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811583936.6A CN109741385A (en) | 2018-12-24 | 2018-12-24 | A kind of image processing system, method, apparatus, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811583936.6A CN109741385A (en) | 2018-12-24 | 2018-12-24 | A kind of image processing system, method, apparatus, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109741385A true CN109741385A (en) | 2019-05-10 |
Family
ID=66359640
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811583936.6A Pending CN109741385A (en) | 2018-12-24 | 2018-12-24 | A kind of image processing system, method, apparatus, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109741385A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111832494A (en) * | 2020-07-17 | 2020-10-27 | 中国联合网络通信集团有限公司 | Information storage method and device |
| CN113345001A (en) * | 2021-05-19 | 2021-09-03 | 智车优行科技（北京）有限公司 | Disparity map determination method and device, computer-readable storage medium and electronic device |
| WO2021185036A1 (en) * | 2020-03-18 | 2021-09-23 | 广州极飞科技有限公司 | Point cloud data generation and real-time display method and apparatus, device, and medium |
Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1175848A (en) * | 1996-06-29 | 1998-03-11 | 三星电子株式会社 | Video data shuffling method and apparatus |
| CN101185580A (en) * | 2006-11-15 | 2008-05-28 | 深圳迈瑞生物医疗电子股份有限公司 | Method and apparatus for gathering ultrasonic diagnosis system high-speed radio-frequency echo wave data |
| CN101308697A (en) * | 2008-07-10 | 2008-11-19 | 哈尔滨工业大学 | FIFO burst buffer with large capacity based on SDRAM and data storage method |
| CN101841730A (en) * | 2010-05-28 | 2010-09-22 | 浙江大学 | Real-time stereoscopic vision implementation method based on FPGA |
| CN102395958A (en) * | 2011-08-26 | 2012-03-28 | 华为技术有限公司 | Concurrent processing method and device for data packet |
| CN103167306A (en) * | 2013-03-22 | 2013-06-19 | 上海大学 | Device and method for extracting high-resolution depth map in real time based on image matching |
| US20140002441A1 (en) * | 2012-06-29 | 2014-01-02 | Hong Kong Applied Science and Technology Research Institute Company Limited | Temporally consistent depth estimation from binocular videos |
| CN103561255A (en) * | 2013-10-24 | 2014-02-05 | 洪丹 | Nakedness-yet stereoscopic display method |
| CN106525004A (en) * | 2016-11-09 | 2017-03-22 | 人加智能机器人技术（北京）有限公司 | Binocular stereo vision system and depth measuring method |
| CN106570901A (en) * | 2016-11-11 | 2017-04-19 | 浙江大学 | CUDA-based binocular depth information recovery acceleration method |
| CN206557758U (en) * | 2017-03-06 | 2017-10-13 | 扬州大学 | A kind of NAND FLASH storage chip array control unit expansible based on FPGA |
| CN107437273A (en) * | 2017-09-06 | 2017-12-05 | 深圳岚锋创视网络科技有限公司 | Six degree of freedom three-dimensional reconstruction method, system and the portable terminal of a kind of virtual reality |
| CN108257165A (en) * | 2018-01-03 | 2018-07-06 | 上海兴芯微电子科技有限公司 | Image solid matching method, binocular vision equipment |
| CN108520537A (en) * | 2018-03-29 | 2018-09-11 | 电子科技大学 | A kind of binocular depth acquisition methods based on photometric parallax |
| CN108629763A (en) * | 2018-04-16 | 2018-10-09 | 海信集团有限公司 | A kind of evaluation method of disparity map, device and terminal |
| CN108682026A (en) * | 2018-03-22 | 2018-10-19 | 辽宁工业大学 | A kind of binocular vision solid matching method based on the fusion of more Matching units |
| CN108742495A (en) * | 2018-03-26 | 2018-11-06 | 天津大学 | A kind of three-dimensional stereo laparoscope system and solid matching method for medical field |
-
2018
- 2018-12-24 CN CN201811583936.6A patent/CN109741385A/en active Pending
Patent Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1175848A (en) * | 1996-06-29 | 1998-03-11 | 三星电子株式会社 | Video data shuffling method and apparatus |
| CN101185580A (en) * | 2006-11-15 | 2008-05-28 | 深圳迈瑞生物医疗电子股份有限公司 | Method and apparatus for gathering ultrasonic diagnosis system high-speed radio-frequency echo wave data |
| CN101308697A (en) * | 2008-07-10 | 2008-11-19 | 哈尔滨工业大学 | FIFO burst buffer with large capacity based on SDRAM and data storage method |
| CN101841730A (en) * | 2010-05-28 | 2010-09-22 | 浙江大学 | Real-time stereoscopic vision implementation method based on FPGA |
| CN102395958A (en) * | 2011-08-26 | 2012-03-28 | 华为技术有限公司 | Concurrent processing method and device for data packet |
| US20140002441A1 (en) * | 2012-06-29 | 2014-01-02 | Hong Kong Applied Science and Technology Research Institute Company Limited | Temporally consistent depth estimation from binocular videos |
| CN103167306A (en) * | 2013-03-22 | 2013-06-19 | 上海大学 | Device and method for extracting high-resolution depth map in real time based on image matching |
| CN103561255A (en) * | 2013-10-24 | 2014-02-05 | 洪丹 | Nakedness-yet stereoscopic display method |
| CN106525004A (en) * | 2016-11-09 | 2017-03-22 | 人加智能机器人技术（北京）有限公司 | Binocular stereo vision system and depth measuring method |
| CN106570901A (en) * | 2016-11-11 | 2017-04-19 | 浙江大学 | CUDA-based binocular depth information recovery acceleration method |
| CN206557758U (en) * | 2017-03-06 | 2017-10-13 | 扬州大学 | A kind of NAND FLASH storage chip array control unit expansible based on FPGA |
| CN107437273A (en) * | 2017-09-06 | 2017-12-05 | 深圳岚锋创视网络科技有限公司 | Six degree of freedom three-dimensional reconstruction method, system and the portable terminal of a kind of virtual reality |
| CN108257165A (en) * | 2018-01-03 | 2018-07-06 | 上海兴芯微电子科技有限公司 | Image solid matching method, binocular vision equipment |
| CN108682026A (en) * | 2018-03-22 | 2018-10-19 | 辽宁工业大学 | A kind of binocular vision solid matching method based on the fusion of more Matching units |
| CN108742495A (en) * | 2018-03-26 | 2018-11-06 | 天津大学 | A kind of three-dimensional stereo laparoscope system and solid matching method for medical field |
| CN108520537A (en) * | 2018-03-29 | 2018-09-11 | 电子科技大学 | A kind of binocular depth acquisition methods based on photometric parallax |
| CN108629763A (en) * | 2018-04-16 | 2018-10-09 | 海信集团有限公司 | A kind of evaluation method of disparity map, device and terminal |
Non-Patent Citations (4)
| Title |
|---|
| 冯彬彬等: "基于FPGA的实时SGM匹配算法研究与实现", 《计算机工程与应用》 * |
| 熊静等著: "《物流信息管理》", 31 May 2017, 国防工业出版社 * |
| 纪坤: "基于FPGA的双目视觉立体匹配算法的研究与实现", 《中国优秀硕士学位论文全文数据库信息科技辑》 * |
| 顾晓菁等: "《运动人体机能研究的理论与方法》", 31 January 2012, 西南交通大学出版社 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021185036A1 (en) * | 2020-03-18 | 2021-09-23 | 广州极飞科技有限公司 | Point cloud data generation and real-time display method and apparatus, device, and medium |
| CN113496503A (en) * | 2020-03-18 | 2021-10-12 | 广州极飞科技股份有限公司 | Point cloud data generation and real-time display method, device, equipment and medium |
| CN111832494A (en) * | 2020-07-17 | 2020-10-27 | 中国联合网络通信集团有限公司 | Information storage method and device |
| CN111832494B (en) * | 2020-07-17 | 2024-03-05 | 中国联合网络通信集团有限公司 | Information storage method and device |
| CN113345001A (en) * | 2021-05-19 | 2021-09-03 | 智车优行科技（北京）有限公司 | Disparity map determination method and device, computer-readable storage medium and electronic device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3248374B1 (en) | Method and apparatus for multiple technology depth map acquisition and fusion | |
| CN104881666B (en) | A kind of real-time bianry image connected component labeling implementation method based on FPGA | |
| US10621446B2 (en) | Handling perspective magnification in optical flow processing | |
| CN109741385A (en) | A kind of image processing system, method, apparatus, electronic equipment and storage medium | |
| WO2020177480A1 (en) | Vehicle accident identification method and apparatus, and electronic device | |
| US10769849B2 (en) | Use of temporal motion vectors for 3D reconstruction | |
| CN105635588B (en) | A kind of digital image stabilization method and device | |
| CN106251403A (en) | A kind of methods, devices and systems of virtual three-dimensional Scene realization | |
| CN105760836A (en) | Multi-angle face alignment method based on deep learning and system thereof and photographing terminal | |
| CN106529517A (en) | Image processing method and image processing device | |
| CN105513083B (en) | A kind of PTAM video camera tracking method and device | |
| CN110602474B (en) | Method, device and equipment for determining image parallax | |
| CN110096993A (en) | The object detection apparatus and method of binocular stereo vision | |
| CN103533313A (en) | Geographical position based panoramic electronic map video synthesis display method and system | |
| Komorkiewicz et al. | FPGA-based hardware-in-the-loop environment using video injection concept for camera-based systems in automotive applications | |
| CN107749069A (en) | Image processing method, electronic equipment and image processing system | |
| CN115240043A (en) | Data processing method and device, electronic equipment and readable storage medium | |
| CN113378867A (en) | Asynchronous data fusion method and device, storage medium and electronic equipment | |
| CN112529006B (en) | Panoramic picture detection method, device, terminal and storage medium | |
| WO2022041257A1 (en) | Method and apparatus for processing three-dimensional map, and movable platform and storage medium | |
| CN116760937B (en) | Video stitching method, device, equipment and storage medium based on multiple machine positions | |
| CN108803658A (en) | Cruising inspection system based on unmanned plane | |
| CN108898617A (en) | A kind of tracking and device of target object | |
| CN106954029A (en) | A kind of panoramic video joining method and system | |
| CN109816791B (en) | Method and apparatus for generating information |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20200922 Address after: Room 1201, building a, 1181 Bin'an Road, Changhe street, Binjiang District, Hangzhou City, Zhejiang Province Applicant after: Zhejiang Xinsheng Electronic Technology Co., Ltd Address before: Hangzhou City, Zhejiang province Binjiang District 310053 shore road 1187 Applicant before: ZHEJIANG DAHUA TECHNOLOGY Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190510 |
|
| RJ01 | Rejection of invention patent application after publication |