CN107749069B - Image processing method, electronic device and image processing system - Google Patents

Image processing method, electronic device and image processing system Download PDF

Info

Publication number
CN107749069B
CN107749069B CN201710903510.3A CN201710903510A CN107749069B CN 107749069 B CN107749069 B CN 107749069B CN 201710903510 A CN201710903510 A CN 201710903510A CN 107749069 B CN107749069 B CN 107749069B
Authority
CN
China
Prior art keywords
image
frames
images
feature point
acquisition
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201710903510.3A
Other languages
Chinese (zh)
Other versions
CN107749069A (en
Inventor
王东
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Lenovo Beijing Ltd
Original Assignee
Lenovo Beijing Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Lenovo Beijing Ltd filed Critical Lenovo Beijing Ltd
Priority to CN201710903510.3A priority Critical patent/CN107749069B/en
Publication of CN107749069A publication Critical patent/CN107749069A/en
Application granted granted Critical
Publication of CN107749069B publication Critical patent/CN107749069B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/64—Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/95—Computational photography systems, e.g. light-field imaging systems
    • H04N23/951—Computational photography systems, e.g. light-field imaging systems by using two or more images to influence resolution, frame rate or aspect ratio

Abstract

The disclosure provides an image processing method, which includes acquiring at least two frames of images at different moments through a single camera, wherein at least partial contents of the at least two frames of images correspond to each other; and determining the depth of the feature point of the part corresponding to the content in the two frames of images based on the acquisition parameters when the two frames of images are acquired, wherein the acquisition parameters comprise position change parameters, and the position change parameters are used for determining the position change of the camera when the two frames of images are acquired. The present disclosure also provides an electronic device and an image processing system.

Description

Image processing method, electronic device and image processing system
Technical Field
The present disclosure relates to an image processing method, an electronic device, and an image processing system.
Background
At present, in the field of artificial intelligence, three-dimensional space-based applications are increasing on mobile phones and computers, and meanwhile, schemes for assisting image or video processing by using image depth data are also increasing. However, the number of devices capable of acquiring depth data in time is still small, which greatly hinders the popularization and application of the depth data acquisition and depth data acquisition. Using TOF sensors or using dual cameras to obtain the depth of a picture in real time requires improvements in the hardware of the capture device, increases cost, and is not available on many existing devices.
Disclosure of Invention
One aspect of the present disclosure provides an image processing method, including acquiring at least two frames of images at different times by a single camera, where at least part of contents of the at least two frames of images correspond to each other, and determining a depth of a feature point of a portion of the two frames of images corresponding to each other in content based on an acquisition parameter when the two frames of images are acquired, where the acquisition parameter includes a position change parameter, and the position change parameter is used to determine a position change of the camera when the two images are acquired.
Optionally, the determining the depth of the feature point of the portion corresponding to the content in the two frames of images based on the acquisition parameter when the two frames of images are acquired includes determining a rotation vector between an acquisition direction of a first image and an acquisition direction of a second image in the two frames of images based on the acquisition parameter when the two frames of images are acquired, wherein the acquisition parameter further includes a focal length of the camera, processing the feature point of the portion corresponding to the content in the first image based on the rotation vector, and determining the depth of the feature point of the portion corresponding to the content in the two frames of images based on the processed feature point, the feature point in the second image corresponding to the feature point, and the acquisition parameter.
Optionally, the determining, based on the acquisition parameters when the two frames of images are acquired, a rotation vector between the acquisition direction of the first image and the acquisition direction of the second image in the two frames of images includes selecting multiple sets of corresponding feature points from the two frames of images, and determining an optimal solution of the rotation vector based on a common intersection point of connection lines between feature points of a portion corresponding to content in the first image and corresponding feature points in the second image, the feature points being obtained after processing the feature points of the portion corresponding to the content in the first image based on the rotation vector between the acquisition direction of the first image and the acquisition direction of the second image in the two frames of images.
Optionally, the determining the depth of the feature point of the portion corresponding to the content in the two frames of images based on the processed feature point, the feature point in the second image corresponding to the feature point, and the acquisition parameter includes determining a first feature point, wherein the position of the first characteristic point after being processed by the rotation vector in the first image is the same as the position of the first characteristic point in the second image, and a second characteristic point is determined, wherein the position of the second feature point in the first image after being processed by the rotation vector is different from the position of the second feature point in the second image, determining the position change of the camera in two image acquisition according to the acquisition parameters, and determining the positions of the first characteristic point and the second characteristic point in two frames of images according to the positions of the first characteristic point and the second characteristic point, and the moving distance and direction of the camera determine the depth of the second characteristic point in the two frames of images.
Optionally, the method further comprises clustering the feature points based on depths of the feature points in the image, identifying objects at different depths in the image.
Another aspect of the present disclosure provides an electronic device, including a processor, and a memory, on which computer readable instructions are stored, and when executed by the processor, the instructions cause the processor to acquire at least two frames of images at different times through a single camera, where at least part of the contents of the at least two frames of images correspond to each other, and determine a depth of a feature point of a portion of the two frames of images where the contents correspond to each other based on an acquisition parameter at the time of acquiring the two frames of images, where the acquisition parameter includes a position change parameter for determining a position change of the camera at the time of two image acquisitions.
Optionally, the processor determines the depth of the feature point of the portion corresponding to the content in the two frames of images based on the acquisition parameters when the two frames of images are acquired, and includes determining a rotation vector between the acquisition direction of the first image and the acquisition direction of the second image in the two frames of images based on the acquisition parameters when the two frames of images are acquired, wherein the acquisition parameters further include a focal length of the camera, processing the feature point of the portion corresponding to the content in the first image based on the rotation vector, and determining the depth of the feature point of the portion corresponding to the content in the two frames of images based on the processed feature point, the feature point in the second image corresponding to the feature point, and the acquisition parameters.
Optionally, the processor determines, based on the acquisition parameters when the two frames of images are acquired, a rotation vector between the acquisition direction of the first image and the acquisition direction of the second image in the two frames of images, where the rotation vector includes a plurality of groups of corresponding feature points selected from the two frames of images, and determines an optimal solution of the rotation vector based on a common intersection point of connection lines between feature points of a portion corresponding to content in the first image and corresponding feature points in the second image, the feature points being obtained after processing the feature points of the portion corresponding to the content in the first image based on the rotation vector between the acquisition direction of the first image and the acquisition direction of the second image in the two frames of images.
Optionally, the processor determines the depth of the feature point of the portion corresponding to the content in the two frames of images based on the processed feature point, the feature point in the second image corresponding to the feature point, and the acquisition parameter includes determining a first feature point, wherein the position of the first characteristic point after being processed by the rotation vector in the first image is the same as the position of the first characteristic point in the second image, and a second characteristic point is determined, wherein the position of the second feature point in the first image after being processed by the rotation vector is different from the position of the second feature point in the second image, determining the position change of the camera in two image acquisition according to the acquisition parameters, and determining the positions of the first characteristic point and the second characteristic point in two frames of images according to the positions of the first characteristic point and the second characteristic point, and the moving distance and direction of the camera determine the depth of the second characteristic point in the two frames of images.
Another aspect of the present disclosure provides an image processing system, including an obtaining module, configured to obtain at least two frames of images at different times through a single camera, where at least part of contents of the at least two frames of images correspond to each other, and a processing module, configured to determine a depth of a feature point of a portion of the two frames of images corresponding to each other in terms of collecting parameters when the two frames of images are collected, where the collecting parameters include a position change parameter, and the position change parameter is used to determine a position change of the camera when the two images are collected.
Another aspect of the disclosure provides a non-volatile storage medium storing computer-executable instructions for implementing the method as described above when executed.
Another aspect of the disclosure provides a computer program comprising computer executable instructions for implementing the method as described above when executed.
Drawings
For a more complete understanding of the present disclosure and the advantages thereof, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, in which:
fig. 1 schematically shows an application scenario of an image processing method, an electronic device and an image processing system according to an embodiment of the present disclosure;
FIG. 2 schematically shows a flow chart of an image processing method according to an embodiment of the present disclosure;
fig. 3 schematically shows a flowchart of determining depths of feature points of a portion corresponding to contents in the two frames of images based on acquisition parameters when the two frames of images are acquired according to an embodiment of the present disclosure;
FIG. 4A schematically illustrates a diagram of selecting multiple sets of corresponding feature points according to an embodiment of the disclosure;
fig. 4B schematically illustrates a schematic view of a camera capturing an image according to an embodiment of the disclosure;
FIG. 4C schematically shows a schematic diagram of continuous extensions imaged at two times by points in space intersecting at one point when only translation, and no rotation, occurs at different times of the camera head according to an embodiment of the disclosure;
fig. 5A schematically shows a flowchart of determining depths of feature points of a portion corresponding to contents in the two frames of images based on the processed feature points, the feature points in the second image corresponding to the feature points, and the acquisition parameters according to an embodiment of the present disclosure;
fig. 5B schematically shows a schematic diagram of a method for determining the depth of the second feature point in two frames of images according to the positions of the first feature point and the second feature point in the two frames of images, and the distance and the direction in which the camera moves, according to an embodiment of the present disclosure;
FIG. 6 schematically shows a flow chart of an image processing method according to another embodiment of the present disclosure;
FIG. 7 schematically shows a block diagram of an image processing system according to an embodiment of the present disclosure;
FIG. 8 schematically shows a block diagram of a processing module according to an embodiment of the disclosure;
FIG. 9 schematically illustrates a block diagram of a second determination submodule according to an embodiment of the present disclosure;
FIG. 10 schematically shows a block diagram of an image processing system according to another embodiment of the present disclosure; and
FIG. 11 schematically shows a block diagram of an electronic device according to an embodiment of the disclosure.
Detailed Description
Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is illustrative only and is not intended to limit the scope of the present disclosure. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present disclosure.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.
All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It is noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of this specification and should not be interpreted in an idealized or overly formal sense.
Where a convention analogous to "at least one of A, B and C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B and C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.). Where a convention analogous to "A, B or at least one of C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B or C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.). It will be further understood by those within the art that virtually any disjunctive word and/or phrase presenting two or more alternative terms, whether in the description, claims, or drawings, should be understood to contemplate the possibilities of including one of the terms, either of the terms, or both terms. For example, the phrase "a or B" should be understood to include the possibility of "a" or "B", or "a and B".
Some block diagrams and/or flow diagrams are shown in the figures. It will be understood that some blocks of the block diagrams and/or flowchart illustrations, or combinations thereof, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the instructions, which execute via the processor, create means for implementing the functions/acts specified in the block diagrams and/or flowchart block or blocks.
Accordingly, the techniques of this disclosure may be implemented in hardware and/or software (including firmware, microcode, etc.). In addition, the techniques of this disclosure may take the form of a computer program product on a computer-readable medium having instructions stored thereon for use by or in connection with an instruction execution system. In the context of this disclosure, a computer-readable medium may be any medium that can contain, store, communicate, propagate, or transport the instructions. For example, the computer readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium. Specific examples of the computer readable medium include: magnetic storage devices, such as magnetic tape or Hard Disk Drives (HDDs); optical storage devices, such as compact disks (CD-ROMs); a memory, such as a Random Access Memory (RAM) or a flash memory; and/or wired/wireless communication links.
The embodiment of the disclosure provides an image processing method, an image processing system and an electronic device for calculating image depth. The method comprises an image acquisition process and a feature point depth determination process. In the image acquisition process, at least two frames of images at different moments are acquired through a single camera, wherein at least partial contents of the at least two frames of images correspond to each other. In the process of determining the depth of the feature point, the depth of the feature point of a part corresponding to the content in the two frames of images is determined based on the acquisition parameters of the two frames of images. The method is used for calculating the image depth, the limitation of the motion mode of the camera is reduced, the solving precision of the image depth is within an acceptable range, and the method is suitable for being deployed on portable equipment. In addition, on one hand, the method does not need to improve the hardware condition of the acquired image, so that the application equipment is not limited, and on the other hand, the method does not need to acquire many continuous frame images, so that the calculation amount is reduced.
Fig. 1 schematically shows an application scenario of an image processing method, an electronic device, and an image processing system according to embodiments of the present disclosure.
As shown in fig. 1, at a first time, the photographing apparatus is located at a position 10, and at a second time, the photographing apparatus is located at a position 20, wherein the photographing angle and the photographing position of the photographing apparatus are different between the position 10 and the position 20. For example, in the example of fig. 1, at position 10, the bottom side of the photographing apparatus is parallel to the X-axis direction, and at position 20, the distance from position 10 in the X-direction is s, and for various reasons, the angle of the photographing apparatus may be changed, for example, rotated by a certain angle in the direction as shown in the figure.
The image acquired by the camera at position 10 is image 11 and the image acquired at position 20 is image 21, where image 11 corresponds to a portion of image 21, such as the example of fig. 1, and where image 11 corresponds to a kiosk in image 21.
According to the embodiment of the present disclosure, the image depth of the corresponding content is calculated by the image 11 and the image 21, for example, the image depth of a kiosk in a scene is calculated, and the image depth of the kiosk may be the image depth in the image 11 or the image depth in the image 21.
It is understood that the photographing device may be any device having a function of capturing images, such as a mobile phone, a tablet computer, and a camera. The position 10 and the position 20 are only schematically shown, the position 10 may be any position, and the shooting device may select any shooting angle, and likewise, the position 20 may be any position different from the position 10, and the posture of the shooting device may be the same as or different from that of the position 10. The at least two images may be at least two images captured by the shooting device, at least two images obtained from a video captured by the shooting device, or at least two images obtained in other manners. According to the embodiment of the disclosure, the two captured images only need to contain at least corresponding partial contents
According to the embodiment of the disclosure, the image processing method for acquiring at least two frames of images at different moments, wherein at least part of contents of the at least two frames of images correspond to each other, and determining the depth of the feature point of the part corresponding to the contents in the two frames of images based on different acquisition parameters, such as a position change parameter and a rotation vector, when the two frames of images are acquired, at least partially solves the problems that the requirement for hardware equipment is high, the amount of calculation is large, and the motion mode of a camera is more limited in the prior art when the image depth is calculated.
Fig. 2 schematically shows a flow chart of an image processing method according to an embodiment of the present disclosure.
As shown in fig. 2, the method includes operations S210 and S220.
In operation S210, at least two frames of images at different times are acquired by a single camera, where at least part of the contents of the at least two frames of images correspond to each other.
In operation S220, based on the acquisition parameters obtained when the two frames of images are acquired, the depth of the feature point of the portion corresponding to the content in the two frames of images is determined, where the acquisition parameters include a position variation parameter, and the position variation parameter is used to determine the position variation of the camera when the two frames of images are acquired.
The image depth is calculated by the method without improving hardware conditions for collecting images or limiting the movement mode of the camera, the calculated amount is reduced, and the calculation accuracy of the image depth is better.
According to the embodiment of the disclosure, a single camera indicates that the image processing method only needs one camera, and does not need two cameras or a plurality of cameras as in the method for calculating the image depth in the prior art. The single camera may be any device with image capturing capability, such as a cell phone, a camera, or a video camera.
According to the embodiment of the disclosure, at least two frames of images at different moments are acquired by a single camera, wherein the two frames of images are images shot by the single camera at different positions, or images acquired from videos shot at different positions, or images acquired at different positions in other manners, and no matter what manner is the image acquired, part of the image corresponds to the content of the image.
According to the embodiment of the disclosure, when the two frames of images are acquired, the camera is provided with acquisition parameters, the acquisition parameters comprise position change parameters of the camera, and the acquisition parameters are used for determining position change of the camera when the two frames of images are acquired. For example, in the case of two image acquisitions of the camera in fig. 1, the position of the camera changes from the position 10 to the position 20, and the position change parameters include a moving distance s and a moving direction, or a position change parameter determined by coordinates of the position 10 and the position 20.
According to the embodiment of the disclosure, the image processing method can determine the depth of the feature point of the corresponding part of the content in the two frames of images according to the two frames of acquired images and the acquisition parameters when the two frames of images are acquired.
A specific embodiment of determining the depth of the feature point of the corresponding portion of the content in the two frames of images according to the two frames of images and the acquisition parameters when the two frames of images are acquired will be described below with reference to fig. 3. It should be noted that fig. 3 may better illustrate the image processing method of the present disclosure, but the present disclosure is not limited to the embodiment illustrated in fig. 3.
Fig. 3 schematically shows a flowchart of determining the depth of the feature point of the portion corresponding to the content in the two frames of images based on the acquisition parameters when the two frames of images are acquired according to the embodiment of the disclosure.
As shown in fig. 3, the method includes operations S310, S320, and S330.
In operation S310, a rotation vector between a capturing direction of a first image and a capturing direction of a second image in the two frames of images is determined based on a capturing parameter when the two frames of images are captured, wherein the capturing parameter further includes a focal length of the camera.
In operation S320, feature points of a portion of the first image corresponding to the content are processed based on the rotation vector.
In operation S330, based on the processed feature points, the feature points in the second image corresponding to the feature points, and the acquisition parameters, the depths of the feature points of the portion corresponding to the content in the two frames of images are determined.
By the method, the depth data of the image is calculated, the limitation on the motion mode is reduced, the calculation amount is reduced, and the depth data can be acquired in real time.
According to an embodiment of the present disclosure, in operation S310, determining a rotation vector between the capturing direction of the first image and the capturing direction of the second image in the two frames of images includes determining a rotation vector between the capturing direction of the first image and the capturing direction of the second image in the two frames of images based on the capturing parameters when the two frames of images are captured. Specifically, the coordinate of the camera in the space when the camera acquires the first image is (x)1，y1，z1) And the camera coordinates when the second image is acquired are (x)2，y2，z2) The rotation vector (a, β, γ) of the camera is determined based on the above coordinates.
According to the embodiment of the present disclosure, the determining, based on the acquisition parameters when the two images are acquired, a rotation vector between the acquisition direction of the first image and the acquisition direction of the second image in the two images includes selecting multiple sets of corresponding feature points from the two images, and determining an optimal solution of the rotation vector based on a common intersection point of connection lines between feature points of a portion corresponding to the content in the first image and corresponding feature points in the second image, the feature points being obtained after processing the feature points of the portion corresponding to the content in the first image based on the rotation vector between the acquisition direction of the first image and the acquisition direction of the second image in the two images. According to the method, the optimal solution of the rotation vector is determined through calculation, so that the accurate rotation vector can be obtained without measurement, and the calculation precision of the image depth is improved. The following description will be made with reference to the embodiments illustrated in fig. 4A, 4B, and 4C.
According to the embodiment of the disclosure, the two frames of images are respectively a first image acquired by the camera in the first acquisition direction and a second image acquired in the second acquisition direction at different moments, wherein the first image and the second image have partial contents corresponding to each other. And selecting a plurality of groups of corresponding characteristic points from the corresponding contents in the two frames of images.
Fig. 4A schematically illustrates a schematic diagram of selecting a plurality of corresponding sets of feature points according to an embodiment of the present disclosure.
As shown in fig. 4A, three corresponding sets of feature points m1 and m2, n1 and n2, and p1 and p2 in the first image and the second image, where m1 and m2 correspond, n1 and n2 correspond, and p1 and p2 correspond, in practical operation, multiple sets of corresponding feature points may be selected according to different situations, and the number of the selected sets is not limited in the present disclosure.
For example, the second acquisition direction is relative to the first acquisition direction, the camera rotates vector R (α, γ).
According to the embodiment of the present disclosure, a specific implementation manner of processing the feature points of the portion corresponding to the content in the first image based on the rotation vector between the capturing direction of the first image and the capturing direction of the second image in the two frames of images is as follows.
Fig. 4B schematically shows a schematic diagram of a camera acquiring an image according to an embodiment of the disclosure.
As shown in fig. 4B, the plane of the camera is an XY plane, and at the first time, the camera acquires the first image M in the first acquisition direction, the image point corresponding to the object point a in the first image M is the image point a1, and the coordinate of the image point a1 in the coordinate system of the camera plane is (x) x1，y1). At the second moment, the camera moves a certain position, and for some reason, the camera rotates a certain angle, the camera acquires a second image N in the second acquisition direction, the image point corresponding to the object point a in the second image N is the image point a2, and the coordinate of the image point a2 in the coordinate system of the camera plane is (x) x2，y2) The camera rotation vector is R (α, γ).
According to an embodiment of the disclosure, the processing of the feature points of the portion of the first image (M-image) corresponding to the content based on the rotation vector comprises rotating the feature points of the portion of the first image (M-image) corresponding to the content, for example, the point a1, by the rotation vector R (α, γ),so as to compensate the displacement of the image point caused by the rotation of the camera, and the characteristic point of the part of the image point A1 corresponding to the content in the image point A2 has only the displacement caused by the translation of the camera and has no displacement caused by the rotation of the camera. Image point A1 (x)1，y1) The coordinates after rotating the vector R (α, γ) become A1' (x)2’，y2') of which one or more,
Figure GDA0002131487400000111
Figure GDA0002131487400000112
then
Figure GDA0002131487400000113
Figure GDA0002131487400000114
And processing the characteristic points of the part corresponding to the content in the first image based on the rotation vector between the acquisition direction of the first image and the acquisition direction of the second image in the two frames of images.
When the camera only translates at different moments without any rotation, continuous extension lines formed by all points in the space at two moments intersect at one point.
Fig. 4C schematically shows a schematic diagram of continuous extended lines imaged at two times by each point in space intersecting at one point when only translation occurs and no rotation occurs at different times of the camera according to an embodiment of the disclosure.
As shown in fig. 4C, a1 ' a2, B1 ' B2, and C1 ' C2 intersect at point P. As described above, the feature points in the first image after processing the feature points of the portion corresponding to the content in the first image based on the rotation vector between the capturing direction of the first image and the capturing direction of the second image in the two frames of images are only translated without any rotation with respect to the corresponding feature points in the second image, and therefore the processed feature points a1 ' a2, B1 ' B2 and C1 ' C2 intersect at one point, and it is set that the intersection pointIs P (x)0，y0)。
According to the embodiment of the disclosure, certain errors exist due to the influence of factors such as measurement accuracy. The error can be determined by
Figure GDA0002131487400000115
The minimum error is calculated by nonlinear optimization, and α, β, gamma and x corresponding to the minimum error are obtained0And y0R (α, γ) at this time is the optimal solution for the rotation vector.
According to the embodiment of the present disclosure, in operation S320, the rotation vector is the rotation vector determined in operation S310, for example, the rotation vector R (α, γ) determined according to the optimal solution, based on which processing the feature points of the portion corresponding to the content in the first image is understood as that the coordinates of the feature points of the portion corresponding to the content in the first image are rotated by the rotation vector to compensate for the displacement of the image points of the portion corresponding to the content in the first image due to the rotation of the camera, so that the feature points of the portion corresponding to the content in the first image and the second image are only translated and not rotated.
According to the embodiment of the present disclosure, in operation S330, based on the processed feature points, the feature points in the second image corresponding to the feature points, and the acquisition parameters, the depths of the feature points of the portion corresponding to the content in the two frames of images are determined.
A method for determining the depth of the feature point of the portion corresponding to the content in the two frames of images based on the processed feature point, the feature point in the second image corresponding to the feature point, and the acquisition parameter will be described below with reference to fig. 5A and 5B. It is to be noted that fig. 5A and 5B are merely used to explain the image processing method of the present disclosure, but the present disclosure is not limited to the embodiment illustrated in fig. 5A and 5B.
Fig. 5A schematically shows a flowchart of determining the depth of the feature point of the portion corresponding to the content in the two frames of images based on the processed feature point, the feature point in the second image corresponding to the feature point, and the acquisition parameter according to the embodiment of the present disclosure.
As shown in fig. 5A, the method includes operations S510, S520, S530, and S540.
In operation S510, a first feature point is determined, where a position of the first feature point after being subjected to rotation vector processing in a first image is the same as a position of the first feature point in a second image.
In operation S520, a second feature point is determined, where a position of the second feature point after being subjected to the rotation vector processing in the first image is different from a position of the second feature point in the second image.
In operation S530, a change in position of the camera is determined according to the acquisition parameters at the time of two image acquisitions.
In operation S540, a depth of the second feature point in the two frame images is determined according to positions of the first feature point and the second feature point in the two frame images, and a distance and a direction in which the camera moves.
The method described in fig. 5A is described below in conjunction with the embodiment of fig. 5B.
Fig. 5B schematically shows a schematic diagram of a method for determining the depth of the second feature point in two frames of images according to the positions of the first feature point and the second feature point in the two frames of images, and the distance and the direction in which the camera moves, according to an embodiment of the present disclosure.
According to the embodiment of the present disclosure, in operation S510, a first feature point is determined, where the position of the first feature point in the first image after being processed by the method of operation S320 is the same as the position of the first feature point in the second image, and as shown in fig. 5B, the position of the first feature point in the first image in the second image is point D.
According to the embodiment of the present disclosure, as described in operation S510 above, in operation S520, a second feature point is determined. As shown in FIG. 5B, the position of the second feature point in the first image in the second image is point S2。
According to the embodiment of the present disclosure, in operation S530, a position change of the camera in two image acquisitions is determined according to the acquisition parameters, where the acquisition parameters include a focal length, a moving distance S, and a moving direction of the camera, or a position change parameter determined by position coordinates.
According to an embodiment of the present disclosure, in operation S540, C1Position of the camera at a first moment, C2For the position of the camera at time 2, connect C1、C2Two points are extended to intersect with the depth of S at the point D1Intersecting T at depth at point D2. The image point of the S point in the first image is S1The image point in the second image is S2The image point of the T point in the first image is T1The image point in the second image is T2It should be noted that fig. 5B is only a schematic representation, and C is a schematic representation1C2The moving distance and direction of (a) may be arbitrary values. In the case illustrated in fig. 5B, a calculation method for determining the depth of the second feature point in the two frames of images according to the positions of the first feature point and the second feature point in the two frames of images, and the distance and the direction in which the camera moves is as follows.
As shown in fig. 5B:
due to △ S2DC2∽△SD1C2；
△S1DC2∽△SD1C1(ii) a And
S1D＝S1S2+S2D，
therefore, the image depth C of the second feature point in the second image can be obtained2D1：
C2D1＝C1C2*S1D/S1S2。
Similarly, the image depth of the additional second feature point T in the second image, the image depth of the additional second feature point T in the first image, and the image depth of the first feature point in the first image may be found.
The method has the advantages that the process of calculating the image depth is simple, and the calculation amount can be obviously reduced while the calculation precision is ensured.
Fig. 6 schematically shows a flow chart of an image processing method according to another embodiment of the present disclosure.
As shown in fig. 6, the method includes operations S210, S220, and S610.
In operation S610, feature points are clustered based on depths of the feature points in the image, and objects at different depths in the image are identified. According to the embodiment of the disclosure, the depth of each pixel point in the image can be clustered by adopting the existing clustering method, such as K-means, and the pixel points with similar depths are considered to possibly represent an object, so that the object in the image can be identified.
The method can identify objects with different depths in the image so as to be applied to the technical fields of foreground extraction, three-dimensional reproduction and the like.
Fig. 7 schematically shows a block diagram of an image processing system 700 according to an embodiment of the present disclosure.
As shown in fig. 7, the image processing system 700 includes an acquisition module 710 and a processing module 720.
The obtaining module 710, for example, performs the operation S210 described above with reference to fig. 2, and is configured to obtain at least two frames of images at different time instants through a single camera, where the at least two frames of images correspond to at least a part of contents.
The processing module 720, for example, performs operation S220 described above with reference to fig. 2, and is configured to determine a depth of a feature point of a portion corresponding to content in the two frames of images based on an acquisition parameter when the two frames of images are acquired, where the acquisition parameter includes a position change parameter, and the position change parameter is used to determine a position change of the camera when the two images are acquired.
Fig. 8 schematically illustrates a block diagram of a processing module 720 according to an embodiment of the disclosure.
As shown in fig. 8, the processing module 720 includes a first determination submodule 810, a processing submodule 820, and a second determination submodule 830.
The first determining sub-module 810, for example, performs the operation S310 described above with reference to fig. 3, to determine a rotation vector between an acquisition direction of a first image and an acquisition direction of a second image in the two frames of images based on the acquisition parameters when the two frames of images are acquired, wherein the acquisition parameters further include a focal length of the camera.
The processing sub-module 820, for example, performs the operation S320 described above with reference to fig. 3, for processing the feature points of the portion corresponding to the content in the first image based on the rotation vector.
The second determining sub-module 830, for example, performs the operation S330 described above with reference to fig. 3, to determine the depth of the feature point of the portion corresponding to the content in the two frames of images based on the processed feature point, the feature point in the second image corresponding to the feature point, and the acquisition parameter.
According to the embodiment of the present disclosure, the determining, based on the acquisition parameters when the two images are acquired, a rotation vector between the acquisition direction of the first image and the acquisition direction of the second image in the two images includes a plurality of groups of corresponding feature points selected from the two images, and based on a common intersection point of connection lines between feature points of a portion corresponding to content in the first image and corresponding feature points in the second image, which are obtained after the feature points of the portion corresponding to content in the first image are processed by the rotation vector between the acquisition direction of the first image and the acquisition direction of the second image in the two images, an optimal solution of the rotation vector is determined.
Fig. 9 schematically illustrates a block diagram of a second determination submodule 830 according to an embodiment of the present disclosure.
As shown in fig. 9, the second determination submodule 830 includes a first determination unit 910, a second determination unit 920, a third determination unit 930, and a fourth determination submodule 940.
The first determining unit 910, for example, performs operation S510 described above with reference to fig. 5A, for determining a first feature point, where a position of the first feature point after being subjected to rotation vector processing in the first image is the same as a position of the first feature point in the second image.
The second determining unit 920, for example, performs operation S520 described above with reference to fig. 5, for determining a second feature point, where a position of the second feature point after being subjected to the rotation vector processing in the first image is different from a position of the second feature point in the second image.
The third determining unit 930, for example, performs operation S530 described above with reference to fig. 5, and is configured to determine a position change of the camera in two image acquisitions according to the acquisition parameters.
The fourth determining submodule 840, for example, performs operation S540 described above with reference to fig. 5, for determining the depth of the second feature point in the two frames of images according to the positions of the first feature point and the second feature point in the two frames of images, and the distance and the direction in which the camera moves.
Fig. 10 schematically shows a block diagram of an image processing system 1000 according to another embodiment of the present disclosure.
As shown in fig. 10, the image processing system 1000 includes an acquisition module 610, a processing module 620, and a recognition module 1010.
The identification module 1010, for example, performs operation S610 described above with reference to fig. 6, for clustering feature points based on depths of the feature points in the image, identifying objects at different depths in the image.
It is understood that the modules described above may be combined into one module, or any one of the modules may be split into a plurality of modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of the other modules and implemented in one module. According to an embodiment of the present invention, at least one of the above modules may be implemented at least partially as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in hardware or firmware in any other reasonable manner of integrating or packaging a circuit, or in a suitable combination of three implementations of software, hardware, and firmware. Alternatively, at least one of the above modules may be implemented at least partly as a computer program module, which, when executed by a computer, may perform the functions of the respective module.
Fig. 11 schematically shows a block diagram of an electronic device 1100 according to an embodiment of the disclosure.
As shown in fig. 11, electronic device 1100 includes a processor 1110 and a memory 1120. The electronic device 1100 may perform the methods described above with reference to fig. 2, 3, 5A, or 6 to enable the calculation of image depth.
In particular, processor 1110 may include, for example, a general purpose microprocessor, an instruction set processor and/or related chip set and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), and/or the like. The processor 1110 may also include onboard memory for caching purposes. Processor 510 may be a single processing unit or a plurality of processing units for performing different actions of the method flows described with reference to fig. 2, 3, 5A or 6 in accordance with embodiments of the present disclosure.
The memory 1120 may be, for example, any medium that can contain, store, communicate, propagate, or transport the instructions. For example, a readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium. Specific examples of the readable storage medium include: magnetic storage devices, such as magnetic tape or Hard Disk Drives (HDDs); optical storage devices, such as compact disks (CD-ROMs); a memory, such as a Random Access Memory (RAM) or a flash memory; and/or wired/wireless communication links.
The memory 1120 may include a computer program 1121, and the computer program 1121 may include code/computer-executable instructions that, when executed by the processor 1110, cause the processor 1110 to perform a method flow, such as that described above in connection with fig. 2, 3, 5A, or 6, and any variations thereof.
The computer programs 1121 can be configured to have, for example, computer program code including computer program modules. For example, in an example embodiment, code in computer program 1121 may include one or more program modules, including, for example, 1121A, 1121B, … …. It should be noted that the division and number of modules are not fixed, and those skilled in the art may use suitable program modules or program module combinations according to actual situations, which when executed by the processor 1110, enable the processor 1110 to perform the method flows described above in connection with fig. 2, 3, 5A or 6, for example, and any variations thereof.
According to an embodiment of the present invention, at least one of the modules described above may be implemented as a computer program module described with reference to fig. 11, which, when executed by the processor 1110, may implement the corresponding operations described above.
Those skilled in the art will appreciate that various combinations and/or combinations of features recited in the various embodiments and/or claims of the present disclosure can be made, even if such combinations or combinations are not expressly recited in the present disclosure. In particular, various combinations and/or combinations of the features recited in the various embodiments and/or claims of the present disclosure may be made without departing from the spirit or teaching of the present disclosure. All such combinations and/or associations are within the scope of the present disclosure.
While the disclosure has been shown and described with reference to certain exemplary embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the disclosure as defined by the appended claims and their equivalents. Accordingly, the scope of the present disclosure should not be limited to the above-described embodiments, but should be defined not only by the appended claims, but also by equivalents thereof.

Claims (6)

1. An image processing method comprising:
acquiring at least two frames of images at different moments by a single camera, wherein at least part of contents of the at least two frames of images correspond to each other; and
determining the depth of the feature point of the part corresponding to the content in the two frames of images based on the acquisition parameters when the two frames of images are acquired, wherein the acquisition parameters comprise position change parameters which are used for determining the position change of the camera when the two frames of images are acquired,
wherein the determining the depth of the feature point of the portion corresponding to the content in the two frames of images based on the acquisition parameters when the two frames of images are acquired comprises: determining a rotation vector between the acquisition direction of a first image and the acquisition direction of a second image in the two frames of images based on acquisition parameters when the two frames of images are acquired, wherein the acquisition parameters further comprise a focal length of the camera; processing feature points of a portion corresponding to content in the first image based on the rotation vector; and determining the depth of the feature points of the part corresponding to the content in the two frames of images based on the processed feature points, the feature points in the second image corresponding to the feature points and the acquisition parameters,
wherein the determining the depth of the feature point of the portion corresponding to the content in the two frames of images based on the processed feature point, the feature point in the second image corresponding to the feature point, and the acquisition parameter comprises:
determining a first feature point, wherein the position of the first feature point in the first image after being subjected to the rotation vector processing is the same as the position of the first feature point in the second image;
determining a second feature point, wherein the position of the second feature point in the first image after being processed by the rotation vector is different from the position of the second feature point in the second image;
determining the position change of the camera in two image acquisition according to the acquisition parameters; and
and determining the depth of the second characteristic point in the two frames of images according to the positions of the first characteristic point and the second characteristic point in the two frames of images, and the moving distance and the moving direction of the camera.
2. The method of claim 1, wherein the determining a rotation vector between an acquisition direction of a first image and an acquisition direction of a second image in the two frames of images based on the acquisition parameters at the time of acquiring the two frames of images comprises:
and determining an optimal solution of the rotation vector based on a common intersection point of connection lines of the feature points of the part corresponding to the content in the first image and the corresponding feature points in the second image, wherein the feature points are obtained by processing the feature points of the part corresponding to the content in the first image based on the rotation vector between the acquisition direction of the first image and the acquisition direction of the second image in the two images.
3. The method of claim 1, further comprising:
and clustering the feature points based on the depths of the feature points in the image, and identifying objects at different depths in the image.
4. An electronic device, comprising:
a processor; and
a memory having computer-readable instructions stored thereon that, when executed by the processor, cause the processor to:
acquiring at least two frames of images at different moments by a single camera, wherein at least part of contents of the at least two frames of images correspond to each other; and
determining the depth of the feature point of the part corresponding to the content in the two frames of images based on the acquisition parameters when the two frames of images are acquired, wherein the acquisition parameters comprise position change parameters which are used for determining the position change of the camera when the two frames of images are acquired,
wherein the processor determines the depth of the feature point of the portion corresponding to the content in the two frames of images based on the acquisition parameters when the two frames of images are acquired, and comprises: determining a rotation vector between the acquisition direction of a first image and the acquisition direction of a second image in the two frames of images based on acquisition parameters when the two frames of images are acquired, wherein the acquisition parameters further comprise a focal length of the camera; processing feature points of a portion corresponding to content in the first image based on the rotation vector; and determining the depth of the feature points of the part corresponding to the content in the two frames of images based on the processed feature points, the feature points in the second image corresponding to the feature points and the acquisition parameters,
wherein the processor determines the depth of the feature point of the portion corresponding to the content in the two frames of images based on the processed feature point, the feature point in the second image corresponding to the feature point, and the acquisition parameter, and comprises:
determining a first feature point, wherein the position of the first feature point in the first image after being subjected to the rotation vector processing is the same as the position of the first feature point in the second image;
determining a second feature point, wherein the position of the second feature point in the first image after being processed by the rotation vector is different from the position of the second feature point in the second image;
determining the position change of the camera in two image acquisition according to the acquisition parameters; and
and determining the depth of the second characteristic point in the two frames of images according to the positions of the first characteristic point and the second characteristic point in the two frames of images, and the moving distance and the moving direction of the camera.
5. The electronic device of claim 4, wherein the processor determines a rotation vector between an acquisition direction of a first image and an acquisition direction of a second image of the two frames of images based on the acquisition parameters at the time of acquiring the two frames of images comprises:
and determining an optimal solution of the rotation vector based on a common intersection point of connection lines of the feature points of the part corresponding to the content in the first image and the corresponding feature points in the second image, wherein the feature points are obtained by processing the feature points of the part corresponding to the content in the first image based on the rotation vector between the acquisition direction of the first image and the acquisition direction of the second image in the two images.
6. An image processing system comprising:
the system comprises an acquisition module, a processing module and a display module, wherein the acquisition module is used for acquiring at least two frames of images at different moments through a single camera, and at least part of contents of the at least two frames of images correspond to each other; and
the processing module is used for determining the depth of the feature point of the part corresponding to the content in the two frames of images based on the acquisition parameters when the two frames of images are acquired, wherein the acquisition parameters comprise position change parameters, and the position change parameters are used for determining the position change of the camera when the two frames of images are acquired;
wherein the processing module determines the depth of the feature point of the portion corresponding to the content in the two frames of images based on the acquisition parameters when the two frames of images are acquired, and comprises: determining a rotation vector between the acquisition direction of a first image and the acquisition direction of a second image in the two frames of images based on acquisition parameters when the two frames of images are acquired, wherein the acquisition parameters further comprise the focal length of the camera; processing feature points of a portion corresponding to content in the first image based on the rotation vector; and determining the depth of the feature points of the part corresponding to the content in the two frames of images based on the processed feature points, the feature points in the second image corresponding to the feature points and the acquisition parameters,
wherein the processor determines the depth of the feature point of the portion corresponding to the content in the two frames of images based on the processed feature point, the feature point in the second image corresponding to the feature point, and the acquisition parameter, and comprises:
determining a first feature point, wherein the position of the first feature point in the first image after being subjected to the rotation vector processing is the same as the position of the first feature point in the second image;
determining a second feature point, wherein the position of the second feature point in the first image after being processed by the rotation vector is different from the position of the second feature point in the second image;
determining the position change of the camera in two image acquisition according to the acquisition parameters; and
and determining the depth of the second characteristic point in the two frames of images according to the positions of the first characteristic point and the second characteristic point in the two frames of images, and the moving distance and the moving direction of the camera.
CN201710903510.3A 2017-09-28 2017-09-28 Image processing method, electronic device and image processing system Active CN107749069B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201710903510.3A CN107749069B (en) 2017-09-28 2017-09-28 Image processing method, electronic device and image processing system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201710903510.3A CN107749069B (en) 2017-09-28 2017-09-28 Image processing method, electronic device and image processing system

Publications (2)

Publication Number Publication Date
CN107749069A CN107749069A (en) 2018-03-02
CN107749069B true CN107749069B (en) 2020-05-26

Family

ID=61255883

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201710903510.3A Active CN107749069B (en) 2017-09-28 2017-09-28 Image processing method, electronic device and image processing system

Country Status (1)

Country Link
CN (1) CN107749069B (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110800023A (en) * 2018-07-24 2020-02-14 深圳市大疆创新科技有限公司 Image processing method and equipment, camera device and unmanned aerial vehicle
CN112132902B (en) * 2019-06-24 2024-01-16 上海安亭地平线智能交通技术有限公司 Vehicle-mounted camera external parameter adjusting method and device, electronic equipment and medium
CN111652110A (en) * 2020-05-28 2020-09-11 北京市商汤科技开发有限公司 Image processing method and device, electronic equipment and storage medium
CN114422736B (en) * 2022-03-28 2022-08-16 荣耀终端有限公司 Video processing method, electronic equipment and computer storage medium

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016062996A1 (en) * 2014-10-20 2016-04-28 Bae Systems Plc Apparatus and method for multi-camera visual odometry

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102168954B (en) * 2011-01-14 2012-11-21 浙江大学 Monocular-camera-based method for measuring depth, depth field and sizes of objects
US10469828B2 (en) * 2015-07-13 2019-11-05 Texas Instruments Incorporated Three-dimensional dense structure from motion with stereo vision
CN105376484A (en) * 2015-11-04 2016-03-02 深圳市金立通信设备有限公司 Image processing method and terminal
CN107025666A (en) * 2017-03-09 2017-08-08 广东欧珀移动通信有限公司 Depth detection method and device and electronic installation based on single camera

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016062996A1 (en) * 2014-10-20 2016-04-28 Bae Systems Plc Apparatus and method for multi-camera visual odometry

Also Published As

Publication number Publication date
CN107749069A (en) 2018-03-02

Similar Documents

Publication Publication Date Title
EP3028252B1 (en) Rolling sequential bundle adjustment
US9916689B2 (en) Apparatus and method for estimating camera pose
CN107749069B (en) Image processing method, electronic device and image processing system
WO2019042426A1 (en) Augmented reality scene processing method and apparatus, and computer storage medium
US9055216B1 (en) Using sensor data to enhance image data
US20150103183A1 (en) Method and apparatus for device orientation tracking using a visual gyroscope
US20140369557A1 (en) Systems and Methods for Feature-Based Tracking
US11282232B2 (en) Camera calibration using depth data
US10558881B2 (en) Parallax minimization stitching method and apparatus using control points in overlapping region
WO2019104571A1 (en) Image processing method and device
WO2018063608A1 (en) Place recognition algorithm
US20110091131A1 (en) System and method for stabilization of fisheye video imagery
CN110770794A (en) Image depth estimation method and device, readable storage medium and electronic equipment
WO2015108996A1 (en) Object tracking using occluding contours
WO2021184302A1 (en) Image processing method and apparatus, imaging device, movable carrier, and storage medium
CN109479082A (en) Image processing method and device
CN108491818B (en) Detection method, device and the electronic equipment of target object
WO2018063606A1 (en) Robust disparity estimation in the presence of significant intensity variations for camera arrays
US10198842B2 (en) Method of generating a synthetic image
EP3216005B1 (en) Image processing device and method for geometric calibration of images
CN115049548A (en) Method and apparatus for restoring image obtained from array camera
GB2557212A (en) Methods and apparatuses for determining positions of multi-directional image capture apparatuses
CN109374919B (en) Method and device for determining moving speed based on single shooting device
CN113436256A (en) Shooting device state identification method and device, computer equipment and storage medium
US10419666B1 (en) Multiple camera panoramic images

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant