CN111935398B - Image processing method and device, electronic equipment and computer readable medium - Google Patents

Image processing method and device, electronic equipment and computer readable medium Download PDF

Info

Publication number
CN111935398B
CN111935398B CN202010649505.6A CN202010649505A CN111935398B CN 111935398 B CN111935398 B CN 111935398B CN 202010649505 A CN202010649505 A CN 202010649505A CN 111935398 B CN111935398 B CN 111935398B
Authority
CN
China
Prior art keywords
image
offset
magnification
current
roi
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010649505.6A
Other languages
Chinese (zh)
Other versions
CN111935398A (en
Inventor
刘钦
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Megvii Technology Co Ltd
Original Assignee
Beijing Megvii Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Megvii Technology Co Ltd filed Critical Beijing Megvii Technology Co Ltd
Priority to CN202010649505.6A priority Critical patent/CN111935398B/en
Publication of CN111935398A publication Critical patent/CN111935398A/en
Priority to PCT/CN2021/093706 priority patent/WO2022007500A1/en
Application granted granted Critical
Publication of CN111935398B publication Critical patent/CN111935398B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/45—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/63—Control of cameras or camera modules by using electronic viewfinders
    • H04N23/631—Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
    • H04N23/632—Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/667—Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/67—Focus control based on electronic image sensor signals

Abstract

The invention provides an image processing method, an image processing device, electronic equipment and a computer readable medium, which relate to the technical field of image processing and comprise the following steps: acquiring a first image acquired by a first camera device; and determining a target ROI area in the first image; calculating ROI offset of a target ROI area in the first image and the second image; acquiring the current magnification of the first image, and carrying out slicing processing on the ROI offset according to the current magnification to obtain the current superposition offset for carrying out image three-dimensional correction on the first image at the current moment; performing image stereo correction on the first image based on the current superposition offset to obtain a corrected first image; if the current magnification ratio is the preset switching magnification ratio, the mobile terminal is switched to the second camera device by the first camera device to display, and the technical problem that jump is obvious when the camera device is switched in imaging in the prior art is solved.

Description

Image processing method and device, electronic equipment and computer readable medium
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to an image processing method and apparatus, an electronic device, and a computer-readable medium.
Background
For a mobile terminal, an image pickup device of the mobile terminal is generally a fixed focus lens, and only digital zooming can be realized. With the increasing demands of consumers on functions such as shooting and video shooting, in order to solve the defect of digital zooming under high zoom ratio, two or more image capturing devices with different focal lengths are mounted on mobile terminals to match, for example, one lens FOV (field angle) is 80 ° and one lens FOV is 40 °. Digital zoom may now be combined with optical zoom to satisfy the consumer. The combination of digital zooming and optical zooming refers to that an image is magnified by one image pickup device and then is switched to another image pickup device for imaging. In fact, however, as for the focal length, each camera device has a certain error during assembly, and the real focal length is different from a theoretical reference value given by a lens manufacturer; for the optical axis, when two image capturing devices are assembled together, the optical axes thereof will also generate errors rather than being absolutely parallel, which results in the observation directions of the two image capturing devices not being completely consistent, and further results in poor user experience when switching the image capturing devices. For example, the Baseline (i.e. distance) of two cameras may cause the two cameras to generate parallax, and the parallax may cause the two cameras to jump in content when they are switched.
Disclosure of Invention
In view of the above, the present invention provides an image processing method, an image processing apparatus, an electronic device, and a computer readable medium, so as to alleviate the technical problem in the prior art that jump is obvious when switching imaging of an imaging apparatus.
In a first aspect, an embodiment of the present invention provides an image processing method, which is applied to a mobile terminal, where the mobile terminal includes: a first image capture device and a second image capture device, the method comprising: acquiring a first image acquired by the first camera device; and determining a target ROI area in the first image; calculating ROI offset of the target ROI area in the first image and a second image, wherein the second image is an image synchronously shot by the second camera when the first camera collects the first image; acquiring the current magnification of the first image, and carrying out slicing processing on the ROI offset according to the current magnification to obtain the current superposition offset for carrying out image three-dimensional correction on the first image at the current moment; performing image stereo correction on the first image based on the current superposition offset to obtain the corrected first image; if the current magnification ratio is a preset switching magnification ratio, the mobile terminal is switched from the first camera device to the second camera device for displaying; wherein, at the time of switching, a sum total of overlay offset amounts by which the first image is subjected to image stereo correction is the ROI offset amount.
Further, the method further comprises: performing texture detection on an image in the target ROI area in the first image to obtain texture intensity; calculating the ROI offset of the target ROI area in the first image and the second image comprises: and if the texture intensity of the image in the target ROI area in the first image is determined to be greater than the preset texture intensity, calculating the ROI offset of the target ROI area in the first image and the second image.
Further, performing texture detection on an image in the target ROI area in the first image, and obtaining a texture intensity includes: calculating a target value based on a pixel point in the first image, which is located in the target ROI area, and determining the texture intensity based on the target value, wherein the texture intensity comprises a standard deviation calculated by any one of the following values: pixel gray scale value, pixel RGB value.
Further, calculating the ROI offset of the target ROI region in the first and second images comprises: determining image feature points in the first image, wherein the image feature points are located in the target ROI area, and obtaining first image feature points; determining matching feature points of the first image feature points in the second image to obtain second image feature points; determining the ROI offset based on a pixel distance between the first image feature point and the second image feature point.
Further, slicing the ROI offset according to the current magnification, and obtaining a current overlay offset for performing image stereo correction on the first image at the current time includes: acquiring a target magnification, wherein the target magnification comprises: an initial magnification and a preset switching magnification; the initial magnification ratio is the magnification ratio corresponding to the first image when the ROI offset is determined for the first time and the first image pickup device executes digital zooming operation on the first image; the preset switching magnification represents an image magnification when the first image pickup apparatus is switched to the second image pickup apparatus; and slicing the ROI offset according to the size relation between the current magnification and the target magnification to obtain the current superposition offset for performing image stereo correction on the first image at the current moment.
Further, slicing the ROI offset according to the size relationship between the current magnification and the target magnification, and obtaining a current overlay offset for performing image stereo correction on the first image at the current time includes: and if the current magnification is smaller than the initial magnification, determining that the current superposition offset is 0.
Further, the ROI offset is sliced according to the size relation between the current magnification and the target magnification to obtain the current superposition offset for performing image stereo correction on the first image at the current momentThe method comprises the following steps: if the current magnification is larger than the initial magnification and smaller than the preset switching magnification, the calculation formula of the current superposition offset is
Figure BDA0002574106730000031
Wherein s iscurRepresenting the current overlay offset, SROIRepresenting the ROI offset, ulcurRepresents the current magnification, ul0 represents the initial magnification of the first image, and sl represents the preset switching magnification.
Further, slicing the ROI offset according to the size relationship between the current magnification and the target magnification, and obtaining a current overlay offset for performing image stereo correction on the first image at the current time further includes: and if the current magnification is larger than the preset switching magnification, the current superposition offset is the ROI offset.
Further, performing image stereo correction on the first image based on the current overlay offset, and obtaining the corrected first image includes: constructing a target transformation matrix based on the current superposition offset and the image amplification parameter; the image magnification parameter comprises a central point of the first image and a preset switching magnification; and calculating the product between the target transformation matrix and the homogeneous coordinates of the first image at the current moment, and determining the first image after image stereo correction according to the product calculation result.
Further, the method further comprises: after the ROI offset is calculated, detecting whether the ROI offset changes or not according to a preset time interval; if the ROI offset is detected to be changed, determining a target ROI offset based on the changed ROI offset and the current superposition offset of the first image; acquiring a first magnification, wherein the first magnification is the magnification of the first image when the ROI offset is detected to occur; and re-determining the current superposition offset of the first image according to the first magnification and the target ROI offset, and performing image stereo correction on the first image according to the re-determined current superposition offset.
Further, re-determining the current overlay offset for the first image from the first magnification and the target ROI offset comprises: and if the first magnification ratio is the same as the second magnification ratio, the current superposition offset of the first image is the same as the superposition offset at the previous moment, wherein the second magnification ratio is the image magnification ratio corresponding to the image stereo correction operation at the previous moment.
Further, re-determining the current overlay offset for the first image from the first magnification and the target ROI offset comprises: and if the first magnification ratio is different from the second magnification ratio, determining the current superposition offset of the first image according to the first magnification ratio and the target ROI offset.
In a second aspect, an embodiment of the present invention provides an apparatus for processing an image, which is disposed in a mobile terminal, where the mobile terminal includes: a first image capture device and a second image capture device, the device comprising: the first acquisition unit is used for acquiring a first image acquired by the first camera device; and determining a target ROI area in the first image; the calculating unit is used for calculating the ROI offset of the target ROI area in the first image and a second image, wherein the second image is an image synchronously shot by the second camera when the first camera collects the first image; a second obtaining unit, configured to obtain a current magnification of the first image, and perform slicing processing on the ROI offset according to the current magnification to obtain a current overlay offset for performing image stereo correction on the first image at a current time; the image stereo correction unit is used for carrying out image stereo correction on the first image based on the current superposition offset to obtain the corrected first image; the switching display unit is used for switching the mobile terminal from the first camera device to the second camera device for displaying if the current magnification is a preset switching magnification; wherein, at the time of switching, a sum total of overlay offset amounts by which the first image is subjected to image stereo correction is the ROI offset amount.
In a third aspect, an embodiment of the present invention provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the steps of the method in any one of the above first aspects when executing the computer program.
In a fourth aspect, an embodiment of the present invention provides a computer-readable medium having non-volatile program code executable by a processor, where the program code causes the processor to perform the steps of the method described in any one of the above first aspects.
In the embodiment of the invention, first, a first image acquired by a first camera device is acquired; determining a target ROI (region of interest) in the first image, then calculating ROI offset of the target ROI in the first image and the second image, acquiring current magnification of the first image, and slicing the ROI offset according to the current magnification to obtain current superposition offset for performing image stereo correction on the first image at the current moment; then, performing image stereo correction on the first image based on the current superposition offset to obtain the corrected first image; and if the current magnification ratio is a preset switching magnification ratio, the mobile terminal is switched from the first camera device to the second camera device for displaying. As can be seen from the above description, in the present application, by determining the current overlay offset based on the current magnification of the first image and based on the current magnification, and performing image stereo correction on the first image through the current overlay offset, it can be ensured that the content of the ROI concerned by the user does not jump when the two image capturing devices are switched, and further, the technical problem of obvious jump when the image capturing devices are switched in the prior art is solved.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
In order to make the aforementioned and other objects, features and advantages of the present invention comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a schematic diagram of an electronic device according to an embodiment of the invention;
fig. 2 is a schematic diagram of image correction of a bi-camera device according to an embodiment of the invention;
FIG. 3 is a flow chart of a digital zoom operation according to an embodiment of the present invention;
fig. 4 is a schematic diagram of the distance between two cameras according to an embodiment of the present invention;
fig. 5 is a schematic diagram of a variation of a distance between a parallax and a bi-camera device according to an embodiment of the present invention;
FIG. 6 is a flow chart of a method of processing an image according to an embodiment of the invention;
FIG. 7 is a schematic diagram of texture detection of a target ROI area according to an embodiment of the invention;
FIG. 8 is a flow chart of another method of processing an image according to an embodiment of the invention;
FIG. 9 is a flow chart of yet another method of image processing according to an embodiment of the present invention;
fig. 10 is a schematic diagram of an image processing apparatus according to an embodiment of the present invention.
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example 1:
first, an electronic device 100 for implementing an embodiment of the present invention, which can be used to execute a processing method of an image according to embodiments of the present invention, is described with reference to fig. 1.
As shown in FIG. 1, electronic device 100 includes one or more processing devices 102, one or more memory devices 104, an input device 106, an output device 108, and a camera device 110, which are interconnected via a bus system 112 and/or other form of connection mechanism (not shown). It should be noted that the components and structure of the electronic device 100 shown in fig. 1 are exemplary only, and not limiting, and the electronic device may have other components and structures as desired.
The processing device 102 may be a Central Processing Unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in the electronic device 100 to perform desired functions.
The storage 104 may include one or more computer program products that may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, Random Access Memory (RAM), cache memory (cache), and/or the like. The non-volatile memory may include, for example, Read Only Memory (ROM), hard disk, flash memory, etc. On which one or more computer program instructions may be stored that may be executed by processing device 102 to implement client functionality (implemented by a processor) and/or other desired functionality in embodiments of the present invention described below. Various applications and various data, such as various data used and/or generated by the applications, may also be stored in the computer-readable storage medium.
The input device 106 may be a device used by a user to input instructions and may include one or more of a keyboard, a mouse, a microphone, a touch screen, and the like.
The output device 108 may output various information (e.g., images or sounds) to the outside (e.g., a user), and may include one or more of a display, a speaker, and the like.
The camera 110 may take images (e.g., photographs, videos, etc.) desired by the user and store the taken images in the storage device 104 for use by other components.
Exemplarily, an exemplary electronic device for implementing the image processing method according to an embodiment of the present invention may be implemented on a mobile terminal such as a smartphone, a tablet computer, or the like.
Before the image processing method of the present application is introduced, the process of the optical digital joint zoom is first introduced, and is described as follows:
when a user uses a mobile phone to perform or shoot, the digital zooming operation is performed as the user starts to zoom in on the screen of the large FOV shooting device. When the scale of the enlarged image content is the same as that of the image content of the full FOV of the small FOV camera device, the image content is switched to another camera device for imaging, and at the moment, the user continues to enlarge the screen and continues to execute the digital zooming operation. At the moment of switching the image pickup device, an optical zoom operation is performed. Compared with a single camera device pure digital zoom, the operation has the advantage that when the magnification is larger, the operation uses a camera device with a larger focal length to image, and particularly for a medium-long-distance scene, the imaging quality of the operation is higher than that of the single-shot digital zoom.
In addition to satisfying that the screen content is the same before and after the two image capturing devices are switched, the following points are generally required to be satisfied in order to approximate the experience of the real optical zoom as much as possible:
(1) the digital zooming ensures that the amplification process is smooth and natural and is close to the experience of real optical zooming in the process of amplifying the image content;
(2) and when the camera device is switched, the same image content scale is ensured, and no sense of rotation exists between the front frame and the rear frame.
(3) And when the camera device is switched, the same image content scale is ensured, and the jump (image content translation) of the front frame and the back frame is reduced as much as possible.
Therefore, in order to satisfy the above requirements, the following setting requirements need to be satisfied:
(1) reducing the jump when switching the cameras, the base (i.e. distance) of the two cameras is required to be as small as possible: baseline can enable the two camera devices to generate parallax, and the parallax can enable the two camera devices to generate jump in content when being switched, and the principle of the jump is explained later;
(2) when the front and the back images of the camera devices are switched without rotating feeling, the optical axes are required to be parallel, namely the directions of the optical axes shot by the two camera devices are the same;
(3) and ensuring the smooth digital zooming and amplifying process and the same scale of the image content when the camera devices are switched, the accurate determination of the focal length information of the camera devices is required so as to calculate the accurate scale proportion of the camera devices with two different FOVs.
In fact, however, as for the focal length, each camera device has a certain error during assembly, and the real focal length is different from a theoretical reference value given by a lens manufacturer; for the optical axis, when two image capturing devices are assembled together, the optical axes of the two image capturing devices will also generate errors, rather than being absolutely parallel, which results in the direction observed by the two image capturing devices not being completely consistent, and further results in poor user experience when switching the image capturing devices. Therefore, the "optical digital joint zoom" algorithm usually requires a factory to calibrate two camera devices in the production of the mobile phone, and calculate accurate internal and external parameters of the camera devices, wherein the internal parameters include the real focal length of the lens; the external parameters include the included angle of the optical axis between the camera devices, then the calibrated data is stored in the mobile phone chip, when the mobile phone is used on the hand of a user, the calibrated data is read, the related information is obtained, and the conversion operations such as stereo rotation, scale conversion and the like are carried out on the image, namely image stereo correction, so that the image with the parallel optical axes is obtained.
The schematic diagram of the stereo correction is shown in fig. 2, in which the rhombus-shaped plane is a picture photographed by two non-parallel cameras actually, and the rectangular plane is an absolutely parallel plane corrected by an algorithm using calibration data. When the optical axes are parallel, the imaging device switches between the front and the back, and the imaging content does not generate rotation feeling and excessive translation beyond Baseline.
Therefore, to realize optical digital joint zooming, three functions are mainly required to be realized:
function one: digital zooming. Based on any magnification (user level (value range: 1.0-)), the digital zooming of the image is realized. As shown in the following formula:
Figure BDA0002574106730000101
and
Figure BDA0002574106730000102
Hzooman image magnification matrix of 3 x 3, s magnification, (cx, cy) image center; i isinputIs a diagram before digital zooming, IoutputIn the figure after digital zoom, (u, v,1) is the homogeneous coordinate of the input image, and (u ', v',1) is the homogeneous coordinate of the output image.
And a second function: and switching logic. As shown in fig. 3, the image pickup device Wide is an image pickup device having a large FOV, the image pickup device Tele is an image pickup device having a small FOV, and when the image pickup device Wide is enlarged to a FOV equal to the FOV of the image pickup device Tele, the image pickup device Wide is switched to the image pickup device Tele. In implementation, normally, the switching magnification switch level corresponding to switching is calculated without comparing whether FOVs are the same, and as shown in fig. 3, the switch level is 2.
The calculation formula is as follows:
Figure BDA0002574106730000103
wherein, Tele refers to the camera device with small FOV, Wide refers to the camera device with large FOV, f is the focal length, and width is the width of the sensor of the camera device, if these data are calculated by calibration, the accuracy of switch level will be higher. When user level is less than switch level, the camera device with large FOV is used for digital zooming; otherwise, the camera device with small FOV is used for digital zooming.
And function III: the rotation is eliminated. In order to make the/video picture as smooth as possible when switching the image pickup device, it is necessary to perform stereo correction on the image based on the calibration data and to eliminate the sense of rotation and the sense of jump due to the optical axis error. The transformation formula is as follows:
Figure BDA0002574106730000111
Figure BDA0002574106730000112
h is a transformation matrix, K is an internal parameter of the camera device obtained by calibration, and R is a rotation angle obtained by calibration.
The three parts are core parts for realizing optical digital combined smooth zooming based on two image pick-up devices with larger FOV difference. The following focuses on the influence of Baseline on the "optical digital joint zoom" algorithm. As can be seen from the above description, after the image stereo correction operation is performed, it can be achieved that the image content angles of the images captured by the two image capturing devices are the same, and the image content scales are the same. However, due to the distance between the two cameras, although the calibration data can eliminate the scale error and the optical axis parallel error, the parallax between the two cameras cannot be eliminated through calibration. Fig. 4 shows a top view of two cameras after stereo correction: wherein two "bold line segments in the X-axis direction" represent the sensor projection planes of the two image pickup devices, ORAnd OTRepresenting the optical centers of the lens, the distance between the two optical centers being Baseline, in which case the distance between the two cameras isThe optical axes are absolutely parallel and the focus are identical (same dimension).
As shown in FIG. 4, a point P in space is located at a distance Z from the bi-camera system, and is imaged as P and P' on two image planes, respectively, with the left end of the "bold line segment in the X-axis direction" as the coordinate starting point, and the coordinate of P as XRP' has the coordinate xTHence we call D ═ (x)R–xT) Is the Disparity of the spatial midpoint P in the bi-camera system. The parallax D changes along with the change of the distance Z, and when the Z is smaller, namely the distance is closer, the parallax is larger; conversely, the smaller the parallax, the parallax is 0 when the distance is infinity. The relationship can be represented by fig. 5.
Therefore, in the application of the "optical digital joint zoom" algorithm, when the image capturing device is switched, each pixel point in the image will have a different parallax jump due to the different distances from the image capturing device to the spatial point where the pixel point is imaged (as shown in fig. 4, point P, if the image capturing device is switched from the left image capturing device to the right image capturing device, P jumps from point P to point P' in the image). Only objects in the scene that are particularly far (near infinity) will not jump; the closer the object, the more obvious the jump in parallax will be. When a user uses a mobile phone or takes a picture, the user focuses on a scene such as a scene of a distant scene, and most of the contents focused on the scene are in the foreground which is not far away from the camera device. At this time, when the camera device is amplified and switched, the concerned foreground inevitably jumps, which brings a bad experience to the user. This is also the case in most mainstream algorithms, although the calibration data is accurate, still can feel the scene jump in the screen when the joint zoom function is used. Based on this, the application provides an image processing method, which can ensure that the video effect not only ensures that the content of the ROI concerned by a user has no jump when two cameras are switched, but also ensures the smoothness in the whole digital zooming and amplifying process, thereby relieving the technical problem of obvious jump when the cameras are switched for imaging in the prior art.
Example 2:
in accordance with an embodiment of the present invention, there is provided an embodiment of a method for processing an image, it should be noted that the steps illustrated in the flowchart of the figure may be performed in a computer system such as a set of computer executable instructions, and that while a logical order is illustrated in the flowchart, in some cases the steps illustrated or described may be performed in an order different than that presented herein.
Fig. 6 is a flowchart of a method for processing an image according to an embodiment of the present invention, as shown in fig. 6, the method including the steps of:
step S602, acquiring a first image acquired by the first camera device; and determining a target ROI area in the first image.
In the present application, the first image capturing device and the second image capturing device described in the following steps are image capturing devices in a mobile terminal (e.g., a mobile phone), wherein the field angles of the first image capturing device and the second image capturing device are different, for example, the field angle of the first image capturing device is larger than the field angle of the second image capturing device. Specifically, the first camera device and the second camera device may be dual-camera devices in a dual-camera mobile phone.
In the present application, the target ROI region refers to a user region of interest. For a mobile terminal, when a user clicks different positions of a screen to focus, it indicates that the user wants to focus on the region, and therefore the mobile terminal will take the region as a target ROI region. If the user does not focus manually, the mobile terminal automatically takes the central area of the screen as the target ROI area, or automatically detects the position of the face of the person in the screen as the target ROI area, and the like. These are all ways of acquiring a target ROI region that are commonly used by mobile terminals. In the "optical digital joint zoom" algorithm, although two cameras (e.g., a first camera and a second camera) are turned on at the same time, the screen of the mobile terminal displays the screen of only one camera, and thus the target ROI area may be the ROI area of the Wide lens (the first camera) or the ROI area of the Tele lens (the second camera).
Step S604, calculating an ROI shift of the target ROI region in the first image and a second image, where the second image is an image synchronously captured by the second imaging device when the first imaging device captures the first image.
In this application, the ROI shift amount refers to a position shift amount of the target ROI region in the first and second images. For example, it can be expressed as a positional shift amount of a feature point within the target ROI region in the first image and a feature point within the target ROI region in the second image.
Step S606, obtaining the current magnification of the first image, and slicing the ROI offset according to the current magnification to obtain the current superposition offset for performing image stereo correction on the first image at the current moment.
The current magnification ratio is a magnification ratio at which the first image is magnified at the current time during the digital zoom operation on the first image. In the present application, slicing refers to performing segmentation processing on the ROI offset, so as to determine a current overlay offset corresponding to a current time.
Step S608, performing image stereo correction on the first image based on the current overlay offset to obtain the corrected first image.
In the application, after the current overlay offset is determined, the first image can be subjected to image stereo correction by the current overlay offset, so that the corrected first image is obtained.
Step S610, if the current magnification ratio is a preset switching magnification ratio, the mobile terminal is switched from the first camera device to the second camera device for displaying; wherein, at the time of switching, a sum total of overlay offset amounts by which the first image is subjected to image stereo correction is the ROI offset amount.
In this application, the preset switching magnification refers to the switching magnification level described in the above, and represents the magnification corresponding to the first image when the mobile terminal is switched from the first image capturing device to the second image capturing device.
In the embodiment of the invention, first, a first image acquired by a first camera device is acquired; determining a target ROI (region of interest) in the first image, then calculating ROI offset of the target ROI in the first image and the second image, acquiring current magnification of the first image, and slicing the ROI offset according to the current magnification to obtain current superposition offset for performing image stereo correction on the first image at the current moment; then, performing image stereo correction on the first image based on the current superposition offset to obtain the corrected first image; and if the current magnification ratio is a preset switching magnification ratio, the mobile terminal is switched from the first camera device to the second camera device for displaying. As can be seen from the above description, in the present application, by determining the current overlay offset based on the current magnification of the first image and based on the current magnification, and performing image stereo correction on the first image through the current overlay offset, it can be ensured that the content of the ROI concerned by the user does not jump when the two image capturing devices are switched, and further, the technical problem of obvious jump when the image capturing devices are switched in the prior art is solved.
In the application, a first image acquired by the first camera device is acquired; after the target ROI area in the first image is determined, texture detection can be performed on the image, located in the target ROI area, in the first image, so that texture intensity is obtained.
In an alternative embodiment, a target value may be calculated based on pixel points in the first image located in the target ROI region, and the texture strength may be determined based on the target value, wherein the texture strength includes a standard deviation calculated by the following values: pixel gray scale values or pixel RGB values.
As can be seen from the above description, after the target ROI region is determined as described above, the content in the target ROI region can be analyzed to ensure that the content in the target ROI region is textured, so that the data for calculating the offset in the next step is reliable.
In the texture detection method, the standard deviation is calculated mainly based on the pixel gray value and the RGB value of the image in the current target ROI area, and then whether the image in the target ROI area contains the texture is determined according to the standard deviation, wherein the standard deviation can be the target numerical value.
Wherein, the calculation formula of the standard deviation can be:
Figure BDA0002574106730000151
n＝H×W，giand representing the gray value or pixel RGB value of the ith pixel point in the target ROI area, wherein H and W are the length and width of the target ROI area.
In this application, after obtaining the texture intensity of the image in the target ROI region in the manner described above, it may be determined whether the texture intensity of the image in the target ROI region in the first image is greater than a preset texture intensity based on the texture intensity, and if it is determined that the texture intensity of the image in the target ROI region in the first image is greater than the preset texture intensity based on the texture intensity, determining an offset of the target ROI region in the first image and the second image, so as to obtain the ROI offset.
For example, as shown in fig. 7, it is assumed that the target ROI area is composed of H rows and W columns of pixel points, where each pixel point has a corresponding gray value gi, and therefore, the stability of n — H × W pixel values can be obtained by calculating the standard deviation according to the gray value gi. If the texture in the target ROI area is weak, the gray value or RGB value is close to each other, the standard deviation is small, and otherwise, the standard deviation is large. In the application, an empirical value may be set as a control threshold Tolerance (i.e., preset texture intensity), and when the standard deviation σ > Tolerance (i.e., preset texture intensity), the texture in the target ROI region is strong and may be used for subsequently calculating the offset; otherwise, the texture in the target ROI area is weak, which indicates that the currently acquired target ROI area is not suitable for calculating the ROI offset, and then, the method waits for acquiring a new image again and then judges.
In the present application, after obtaining the target ROI area in the first image according to the above-described process, and performing texture detection on the image located in the target ROI area, the offset of the target ROI area in the first image and the second image may be calculated, so as to obtain the ROI offset.
In an alternative embodiment, the step S604 of calculating the ROI shift of the target ROI region in the first image and the second image includes the following processes:
step S6041, determining image feature points in the first image, which are located in the target ROI area, to obtain first image feature points;
step S6042, determining matching feature points of the first image feature points in the second image to obtain second image feature points;
step S6043 of determining the ROI shift amount based on a pixel distance between the first image feature point and the second image feature point.
After the target ROI area is determined, images synchronously shot by the first camera device and the second camera device, namely the first image and the second image, need to be captured. Then, the matching feature points of the two images are searched in the target ROI area. For example, first, image feature points in the first image located in the target ROI region are determined, and first image feature points are obtained, where the image feature points may be feature points of an object included in the target ROI region, for example, feature points of a human face, and the like, which is not limited in this application. And then, determining the matching feature points of the first image feature points in the second image to obtain second image feature points. Then, a pixel distance between the first image feature point and the second image feature point is calculated, and the pixel distance is used as the ROI offset, wherein a coordinate difference between matched pixel points in the first image feature point and the second image feature point can be calculated as the pixel distance.
In the present application, when calculating the ROI shift amount, it is generally assumed that the target ROI region belongs to the same depth in space, that is, the region is at a comparable distance from the imaging device, but not that the region is partially in the foreground region and partially in the background region. At this time, the imaging device corresponding to the target ROI needs to be determined first to determine the image content of the target ROI, and then the image content is matched with another image, and the offset of the region is obtained.
The above process will be exemplified below, assuming that the first image displayed in the current mobile terminal is captured by the first camera device having a larger FOV, and the user manually focuses on a certain region (i.e., the target ROI region).
As shown in fig. 8, a preview screen of a screen of the mobile terminal is an image (first image) of a Wide camera device (first camera device), and a target ROI area is obtained by focusing on a touch screen. The image content in the target ROI area is obtained based on the target ROI area and the first image, then a Tele image (namely, a second camera device) shot at the same time is combined, and a plurality of groups of matching point sets are obtained based on a feature point extraction and matching algorithm (wherein the plurality of groups of matching point sets are the first image feature point and the second image feature point). In the present application, although the image captured by Tele (i.e., the second imaging device) is not displayed on the mobile terminal, Wide (i.e., the first imaging device) and Tele (i.e., the second imaging device) are simultaneously turned on and are ready to be displayed on the screen at all times during the preview/imaging. It should be noted that, in the present application, the feature point extraction and matching algorithm may be any of the following: SIFT, ORB, BRISK, etc., the present invention is not limited to specific matching algorithms.
In the present application, a simple procedure for determining the ROI shift amounts (Δ u, Δ v) is as follows:
firstly, calculating N groups of matching point pairs by using a Lupont characteristic point detection and matching algorithm; namely a first image feature point and a second image feature point;
then, removing the mismatching points for the matched point pairs by using an algorithm RANSAC for removing the mismatching points, and reserving the accurate matching points;
and finally, solving the pixel distances among the multiple groups of matching point pairs, and calculating the average value of the pixel distances to be used as the ROI offset.
In the application, after the ROI offset is obtained, the current magnification of the first image may be obtained, and the ROI offset is sliced according to the current magnification, so as to obtain the current overlay offset for performing image stereo correction on the first image at the current time.
In an optional embodiment, in step S606, the step of performing slice processing on the ROI offset according to the current magnification to obtain a current overlay offset for performing image stereo correction on the first image at the current time includes:
step S6061, acquiring a target magnification, where the target magnification includes: an initial magnification and a preset switching magnification; the initial magnification ratio is the magnification ratio corresponding to the first image when the ROI offset is determined for the first time and the first image pickup device executes digital zooming operation on the first image; the preset switching magnification represents an image magnification when the first image pickup apparatus is switched to the second image pickup apparatus;
step S6062, slice the ROI offset according to the size relationship between the current magnification and the target magnification, and obtain a current overlay offset at which image stereo correction is performed on the first image at the current time.
In the present application, in order to ensure the smoothness of the entire digital zoom process during the digital zoom process performed on the first image, after the ROI offset is obtained, the ROI offset needs to be subjected to a slicing process. The ROI shift amounts (i.e., the superimposition shift amounts) of different proportions are gradually superimposed on the first image in the process of enlarging the first image until the total shift amount superimposition is completed when the enlargement is to the switching magnification.
Specifically, in the present application, the ROI shift amount may be sliced by comparing a magnitude relationship between a current magnification and a target magnification, so as to obtain a current overlay shift for performing image stereo correction on the first image at a current time. By slicing the ROI offset, it can be ensured that there is no jump in the image in the target ROI region during switching, and the specific embodiment is as follows:
because the calculation of the ROI offset and the optical digital combined zooming are carried out synchronously, the initial magnification ratio user level corresponding to the first image when the first image pickup device carries out the digital zooming operation on the first image is ul when the ROI offset is calculated for the first time0Deviation of ROIThe size of the displacement is SROIThe preset switching magnification level is abbreviated as sl. The current magnification user level is ulcurThe current superposition offset which is superposed on each frame image at the current moment is scur，scurThe results can be classified into the following cases:
situation one,
And if the current magnification is smaller than the initial magnification, determining that the current superposition offset is 0.
That is, if ulcur<ul0Then the current overlay may be offset by an amount scurIs set to zero.
The second case,
If the current magnification is larger than the initial magnification and smaller than the preset switching magnification, the calculation formula of the current superposition offset is
Figure BDA0002574106730000191
Wherein s iscurRepresenting the current overlay offset, SROIRepresenting the ROI offset, ulcurRepresents the current magnification, ul0 represents the initial magnification of the first image, and sl represents the preset switching magnification.
That is, if ul0<ulcur<sl, then the current overlay offset scurCan be represented by formula
Figure BDA0002574106730000192
To be determined.
The formula realizes the slice effect of ROI offset, namely the distance ratio between the current magnification and the preset switching magnification is corresponding to the ROI offset SROIAbove, realize ul0With ul0The offset between the next frames of (a) is small enough to prevent the content of the image from making visible translational jumps; at the same time, when ulcurS is close to slcurIs also substantially equal to SROI。
Case three,
And if the current magnification is larger than the preset switching magnification, the current superposition offset is the ROI offset.
In the present application, if ulcur>sl, then scur＝SROI. It should be noted that, at this time, the second image acquired by the second camera device is displayed on the screen, but the current overlay offset is still overlaid on the first image, so that when the user switches the screen from the second camera device to the first camera device, the image in the target ROI area can still be ensured to have no jump.
In the present application, after the current superimposition offset amount of the image stereo correction operation at the present time is determined in the manner described in the above-described cases one to three, the image stereo correction operation may be performed on the first image based on the current superimposition offset amount of the image stereo correction operation at the present time.
In an optional embodiment, performing image stereo correction on the first image based on the current overlay offset, and obtaining the corrected first image includes the following steps:
(1) constructing a target transformation matrix based on the current superposition offset and the image amplification parameter; the image magnification parameter comprises a central point of the first image and a preset switching magnification;
(2) and calculating the product of the target transformation matrix and the homogeneous coordinates of the first image at the current moment, and determining the first image after image stereo correction according to the product calculation result.
As can be seen from the above description, the digital zoom operation is implemented by constructing the image transformation matrix H according to a certain scaling ratiozoomAnd further through an image transformation matrix HzoomAnd realizing the transformation of the image. The specific formulas for constructing the image transformation matrix and realizing the transformation of the image through the image transformation matrix are as follows:
Figure BDA0002574106730000201
thus, it is possible to provideDetermining the current superposition offset s of the image stereo correction operation at the current momentcurAfter ([ delta ] u, [ delta ] v), the transformation matrix H still needs to be constructedshiftAnd is superimposed on the image transformation matrix HzoomWherein the image transformation matrix HzoomIncluding image magnification parameters. By the processing mode, the translation of the image can be completed while the stereo correction is performed.
In the present application, the transformation matrix HshiftThe formula is as follows:
Figure BDA0002574106730000202
the formula of the target transformation matrix constructed based on the current superposition offset and the image amplification parameter of the image stereo correction operation at the current moment is as follows:
Figure BDA0002574106730000203
after obtaining a new Hzoom' (i.e., the target transformation matrix) through the above construction and transformation, it is possible to achieve image magnification and image translation in the digital zoom process, and apply the offset calculated in the above process to the input image of the video frame.
It should be noted that, in the present application, the target ROI area is updatable, and the target ROI area may be updated by the user. It is therefore possible to change the ROI offset, whether it is a scene change or a focus change. Therefore, in the present application, the process of calculating the ROI shift amount also requires that the process be repeated at a certain frequency throughout the "optical digital joint zoom" process. Therefore, it is also necessary to consider how the calculation of the overlay offset should be updated when the ROI offset is changed.
Based on this, the method further comprises the steps of:
step S1, if it is detected that the ROI shift amount has changed, determining a target ROI shift amount based on the changed ROI shift amount and the current superimposition shift amount of the first image;
step S2, acquiring a first magnification, where the first magnification is a magnification of the first image when the ROI shift amount is detected;
step S3, re-determining the current overlay offset of the first image according to the first magnification and the target ROI offset, and performing image stereo correction on the first image according to the re-determined current overlay offset.
Specifically, assume that, before the ROI offset is updated, the ROI offset SROI＝SROI_0(ii) a At the time immediately before the ROI shift update, the magnification user level (i.e., the second magnification in this application) of the first image is ulcur_0. If ul0<ulcur_0<sl, at which time the image is stereoscopically corrected for the corresponding current overlay offset scur_0As shown in
Figure BDA0002574106730000211
Shown, otherwise scur_0Should be equal to 0 or SROI_0。
After the ROI offset is updated, the ROI offset after change is set to SROI_1But since the image frame is already according to S at this timeROI_0For reference, a superimposed offset s is superimposedcur_0The amount of offset of (c). Thus, the amount of target ROI shift S corresponding to the image stereo correction operationROIComprises the following steps: sROI＝SROI_1-scur_0。
After the target ROI shift amount is obtained, the magnification of the first image at the time when the ROI shift amount is detected, that is, the first magnification, may be continuously acquired. Then, the current superposition offset of the first image is redetermined according to the first magnification and the target ROI offset, and the first image is subjected to image stereo correction according to the redetermined current superposition offset.
Therefore, on this basis, re-determining the current overlay shift of the first image based on the first magnification and the target ROI shift may be classified into the following cases:
and if the first magnification ratio is the same as the second magnification ratio, the current superposition offset of the first image is the same as the superposition offset at the previous moment, wherein the second magnification ratio is the image magnification ratio corresponding to the image stereo correction operation at the previous moment.
In this case, the ROI offset is updated, but the magnification user level is not changed, i.e. the first magnification ulcurSecond magnification ulcur_0At this time, the current overlay offset of the image stereo correction operation at the current time is the same as the overlay offset of the image stereo correction operation at the previous time, that is: scur＝scur_0。
In this case, though, the first magnification ulcurMay be greater than the initial magnification ul0Less than the preset switching multiplying power sl, but at the moment, the image frame is superposed with the superposition offset scur_0If the offset s of the superposition of the stereo correction operation of the image at the current moment is calculated according to the slice formulacurIn the case of (1), ROI offset SROIThere is a change, i.e. the magnification user level is not changed, but the content in the screen makes a jump, which jump is perceptible to the naked eye. Therefore, after the ROI offset is updated, it is necessary to determine whether the user-controlled magnification user level (first magnification) has changed, and if not, the original s is still retainedcur＝scur_0。
And if the first magnification ratio is different from the second magnification ratio, determining the current superposition offset of the first image according to the first magnification ratio and the target ROI offset.
In this case, when the first magnification user level is changed, the superimposition offset amount of the image stereo correction operation at the current time is continuously determined in the manner described in the above-described cases one to three, and the description thereof will not be made.
As shown in fig. 9, in the present application, the image processing method mainly includes the following three parts:
firstly, determining a target ROI area
In the present application, the target ROI region refers to a user region of interest. For a mobile terminal, when a user clicks different positions of a screen to focus, it indicates that the user wants to focus on the region, and therefore the mobile terminal will take the region as a target ROI region. If the user does not focus manually, the mobile terminal automatically takes the central area of the screen as the target ROI area, or automatically detects the position of the face of the person in the screen as the target ROI area, and the like. These are all ways of acquiring a target ROI region that are commonly used by mobile terminals. In the "optical digital joint zoom" algorithm, although two cameras (e.g., a first camera and a second camera) are turned on at the same time, the screen of the mobile terminal displays the screen of only one camera, and thus the target ROI area may be the ROI area of the Wide lens (the first camera) or the ROI area of the Tele lens (the second camera).
Secondly, texture detection is carried out
As can be seen from the above description, after the target ROI region in the first image (i.e., the Wide image frame) is determined, the process of obtaining the ROI offset based on the offset of the target ROI region in the first image and the second image (i.e., the Tele image frame) can be summarized as follows:
and (3) ROI textural verification: to ensure that the region of the target ROI is obtained that is matable (i.e., has texture features), the texture of the region is first verified before matching. The verification method is mainly based on the gray value or RGB value of each pixel in the area, the standard deviation is calculated, if the standard deviation is smaller than the preset texture intensity, the area is considered to have no texture, and the current calculation is quitted; otherwise, the next calculation is carried out.
Extracting and matching image feature points: there are many related algorithms, such as SIFT, ORB, BRISK, etc., and the present invention does not explicitly define the specific method for finding the matching point. In the present application, the first image feature point may be determined in the first image through the above algorithm, and the matching feature point of the first image feature point may be determined in the second image, so as to obtain the second image feature point.
Third, calculating ROI offset
After extraction and matching of image feature points are completed, more than one matching point is usually obtained, and at this time, in order to remove an erroneous matching point, the erroneous matching point needs to be removed once based on RANSAC. After the correct matching point pair is obtained, the pixel offset of each point pair is calculated, and then the average value of the pixel offsets is obtained to obtain the final offset (namely, ROI offset).
Fourthly, smoothly eliminating the deviation of ROI
After the ROI shift is obtained, the image during "optical digital joint zoom" can be translated based on this data so that there is no jump at the target ROI region when the image is zoomed in to the switch camera. But on the one hand, the ROI offset is acquired in the process of optical digital combined zooming; on the other hand, the target ROI area is also user-controlled and variable, so the ROI offset calculation needs to be performed frequently and in synchronization with the zooming process to adapt to the scene change and the target ROI area change. Therefore, how to apply the obtained ROI offset to the zooming process can ensure that the target ROI region has no jump during switching, and the whole digital zooming process is smooth, which is the most important problem of the technology.
Firstly, it is assumed that after the current frame acquires the ROI offset, it is impossible to directly apply the ROI offset to the next frame, because in this case, a jump is directly generated between the current frame and the next frame, instead of a smooth zoom; but at the same time it needs to be ensured that when the zoom reaches the camera switching frame, the ROI offset has already acted on the current frame, so that when the next frame is switched to another camera, there is no jump in the image content at the target ROI area. Therefore, the invention provides a method for smoothly eliminating ROI offset, which can ensure smooth amplification in a digital zooming process, avoid jumping caused by the occurrence and update of the ROI offset and ensure that a target ROI area has no jumping when a camera is switched.
It should be noted here that the action of eliminating the ROI shift is mainly a process of translating the image. Since the FOV of the Wide is much larger than Tele, the ROI shift amount based on the calculation mainly acts on the Wide image frame in order to ensure that the Wide is aligned with Tele at the target ROI region when the camera is switched. Therefore, the specific implementation flow is as follows:
the digital zoom process typically center-zooms in or out on an image based on a magnification (user level) before the ROI shift is first obtained. After the ROI offset is obtained for the first time, slicing is carried out on the value of the offset according to the distance between the current user level and the switch level, and in the process of continuous amplification of the user level, the overlapping offsets of different proportions are overlapped, so that the offset is larger when the user level is closer to the switch level, otherwise, the offset is smaller, and the ROI offset completely acts on the image frame until the user level is equal to the switch level. Since the ROI offset has already been sliced, the digital zoom process is smooth and does not jump during zooming. It should be noted that, if the user level is not changed, the current offset is kept to be continuously applied to the image frame; if user level > switch level, the Tele image frame is usually displayed at this time, so that no offset needs to be superimposed on the Tele image frame.
The ROI offset may change due to a scene change or when the user refocuses to a position at a different depth from the camera. Therefore, the change of the ROI shift amount needs to be taken into consideration. At this time, when the ROI offset is updated, the user level2 at this time is also amplified by a certain magnification compared with the user level1 when the ROI offset was obtained last time, that is, the image of the current frame has been translated by a certain pixel value based on the last overlay offset, then the current translated ROI offset needs to be subtracted from the current updated ROI offset to be used as the ROI offset at the current time, and the ROI offset is sliced again, so that even after the ROI offset is updated, the ROI still does not jump when the ROI is switched over when the digital zoom is too smooth.
Example three:
the embodiment of the present invention further provides an image processing apparatus, which is mainly used for executing the image processing method provided by the above-mentioned content of the embodiment of the present invention, and the following describes the image processing apparatus provided by the embodiment of the present invention in detail.
Fig. 10 is a schematic diagram of an image processing apparatus according to an embodiment of the present invention, and as shown in fig. 10, the image processing apparatus mainly includes: a first acquisition unit 10, a calculation unit 20, a second acquisition unit 30, a slice processing unit 40, an image correction unit 50, and a switching display unit 60, wherein:
the first acquisition unit is used for acquiring a first image acquired by the first camera device; and determining a target ROI area in the first image;
the calculating unit is used for calculating the ROI offset of the target ROI area in the first image and a second image, wherein the second image is an image synchronously shot by the second camera when the first camera collects the first image;
a second acquisition unit configured to acquire a current magnification of the first image;
the slice processing unit is used for carrying out slice processing on the ROI offset according to the current magnification to obtain the current superposition offset for carrying out image stereo correction on the first image at the current moment;
the image stereo correction unit is used for carrying out image stereo correction on the first image based on the current superposition offset to obtain the corrected first image;
the switching display unit is used for switching the mobile terminal from the first camera device to the second camera device for displaying if the current magnification is a preset switching magnification; wherein, at the time of switching, a sum total of overlay offset amounts by which the first image is subjected to image stereo correction is the ROI offset amount.
As can be seen from the above description, in the present application, by determining the current overlay offset based on the current magnification of the first image and based on the current magnification, and performing image stereo correction on the first image through the current overlay offset, it can be ensured that the content of the ROI concerned by the user does not jump when the two image capturing devices are switched, and further, the technical problem of obvious jump when the image capturing devices are switched in the prior art is solved.
Optionally, the apparatus is further configured to: the method further comprises the following steps: performing texture detection on an image in the target ROI area in the first image to obtain texture intensity; the computing unit is further to: and if the texture intensity of the image in the target ROI area in the first image is determined to be greater than the preset texture intensity, calculating the ROI offset of the target ROI area in the first image and the second image.
Optionally, the apparatus is further configured to: calculating a target value based on a pixel point in the first image, which is located in the target ROI area, and determining the texture intensity based on the target value, wherein the texture intensity comprises a standard deviation calculated by any one of the following values: pixel gray scale value, pixel RGB value.
Optionally, the computing unit is further configured to: determining image feature points in the first image, wherein the image feature points are located in the target ROI area, and obtaining first image feature points; determining matching feature points of the first image feature points in the second image to obtain second image feature points; determining the ROI offset based on a pixel distance between the first image feature point and the second image feature point.
Optionally, the slice processing unit is configured to: acquiring a target magnification, wherein the target magnification comprises: an initial magnification and a preset switching magnification; the initial magnification ratio is the magnification ratio corresponding to the first image when the ROI offset is determined for the first time and the first image pickup device executes digital zooming operation on the first image; the preset switching magnification represents an image magnification when the first image pickup apparatus is switched to the second image pickup apparatus; and slicing the ROI offset according to the size relation between the current magnification and the target magnification to obtain the current superposition offset for performing image stereo correction on the first image at the current moment.
Optionally, the slice processing unit is further configured to: and if the current magnification is smaller than the initial magnification, determining that the current superposition offset is 0.
Optionally, the slice processing unit is further configured to: if the current magnification is larger than the initial magnification and smaller than the preset switching magnification, the calculation formula of the current superposition offset is
Figure BDA0002574106730000271
Wherein s iscurRepresenting the current overlay offset, SROIRepresenting the ROI offset, ulcurRepresents the current magnification, ul0 represents the initial magnification of the first image, and sl represents the preset switching magnification.
Optionally, the slice processing unit is further configured to: and if the current magnification is larger than the preset switching magnification, the current superposition offset is the ROI offset.
Optionally, the image stereo correction unit is configured to: constructing a target transformation matrix based on the current superposition offset and the image amplification parameter; the image magnification parameter comprises a central point of the first image and a preset switching magnification; and calculating the product between the target transformation matrix and the homogeneous coordinates of the first image at the current moment, and determining the first image after image stereo correction according to the product calculation result.
Optionally, the apparatus is further configured to: after the ROI offset is calculated, detecting whether the ROI offset changes or not according to a preset time interval; if the ROI offset is detected to be changed, determining a target ROI offset based on the changed ROI offset and the current superposition offset of the first image; acquiring a first magnification, wherein the first magnification is the magnification of the first image when the ROI offset is detected to occur; and re-determining the current superposition offset of the first image according to the first magnification and the target ROI offset, and performing image stereo correction on the first image according to the re-determined current superposition offset.
Optionally, the apparatus is further configured to: and if the first magnification ratio is the same as the second magnification ratio, the current superposition offset of the first image is the same as the superposition offset at the previous moment, wherein the second magnification ratio is the image magnification ratio corresponding to the image stereo correction operation at the previous moment.
Optionally, the apparatus is further configured to: and if the first magnification ratio is different from the second magnification ratio, determining the current superposition offset of the first image according to the first magnification ratio and the target ROI offset.
The device provided by the embodiment of the present invention has the same implementation principle and technical effect as the method embodiments, and for the sake of brief description, reference may be made to the corresponding contents in the method embodiments without reference to the device embodiments.
In addition, in the description of the embodiments of the present invention, unless otherwise explicitly specified or limited, the terms "mounted," "connected," and "connected" are to be construed broadly, e.g., as meaning either a fixed connection, a removable connection, or an integral connection; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meanings of the above terms in the present invention can be understood in specific cases to those skilled in the art.
In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings, and are only for convenience of description and simplicity of description, but do not indicate or imply that the device or element being referred to must have a particular orientation, be constructed and operated in a particular orientation, and thus, should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions when actually implemented, and for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some communication interfaces, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
Finally, it should be noted that: the above-mentioned embodiments are only specific embodiments of the present invention, which are used for illustrating the technical solutions of the present invention and not for limiting the same, and the protection scope of the present invention is not limited thereto, although the present invention is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive the technical solutions described in the foregoing embodiments or equivalent substitutes for some technical features within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the embodiments of the present invention, and they should be construed as being included therein. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims (15)

1. An image processing method is applied to a mobile terminal, and the mobile terminal comprises the following steps: a first image capture device and a second image capture device, the method comprising:
acquiring a first image acquired by the first camera device; and determining a target ROI area in the first image;
calculating ROI offset of the target ROI area in the first image and a second image, wherein the second image is an image synchronously shot by the second camera when the first camera collects the first image;
acquiring the current magnification of the first image, and slicing the ROI offset according to the current magnification to obtain the current superposition offset for performing image three-dimensional correction on the first image at the current moment so as to gradually superpose the current superposition offsets in different proportions in the process of amplifying the first image;
performing image stereo correction on the first image based on the current superposition offset to obtain the corrected first image;
if the current magnification ratio is a preset switching magnification ratio, the mobile terminal is switched from the first camera device to the second camera device for displaying; wherein, at the time of switching, a sum total of overlay offset amounts by which the first image is subjected to image stereo correction is the ROI offset amount.
2. The method of claim 1,
the method further comprises the following steps: performing texture detection on an image in the target ROI area in the first image to obtain texture intensity;
calculating the ROI offset of the target ROI area in the first image and the second image comprises: and if the texture intensity of the image in the target ROI area in the first image is determined to be greater than the preset texture intensity, calculating the ROI offset of the target ROI area in the first image and the second image.
3. The method of claim 2, wherein performing texture detection on the image in the target ROI region in the first image, and obtaining the texture strength comprises:
calculating a target value based on a pixel point in the first image, which is located in the target ROI area, and determining the texture intensity based on the target value, wherein the texture intensity comprises a standard deviation calculated by any one of the following values: pixel gray scale value, pixel RGB value.
4. The method of claim 1 or 2, wherein calculating the ROI shift of the target ROI region in the first and second images comprises:
determining image feature points in the first image, wherein the image feature points are located in the target ROI area, and obtaining first image feature points;
determining matching feature points of the first image feature points in the second image to obtain second image feature points;
determining the ROI offset based on a pixel distance between the first image feature point and the second image feature point.
5. The method of claim 1, wherein slicing the ROI offset according to the current magnification to obtain a current overlay offset for performing image stereo correction on the first image at a current time comprises:
acquiring a target magnification, wherein the target magnification comprises: an initial magnification and a preset switching magnification; the initial magnification ratio is the magnification ratio corresponding to the first image when the ROI offset is determined for the first time and the first image pickup device executes digital zooming operation on the first image; the preset switching magnification represents an image magnification when the first image pickup apparatus is switched to the second image pickup apparatus;
and slicing the ROI offset according to the size relation between the current magnification and the target magnification to obtain the current superposition offset for performing image stereo correction on the first image at the current moment.
6. The method of claim 5, wherein slicing the ROI offset according to a magnitude relationship between the current magnification and the target magnification to obtain a current overlay offset for performing image stereo correction on the first image at a current time comprises:
and if the current magnification is smaller than the initial magnification, determining that the current superposition offset is 0.
7. The method of claim 5, wherein slicing the ROI offset according to a magnitude relationship between the current magnification and the target magnification to obtain a current overlay offset for performing image stereo correction on the first image at a current time further comprises:
if the current magnification is larger than the initial magnification and smaller than the preset switching magnification, the calculation formula of the current superposition offset is
Figure FDA0003308607990000031
Wherein s iscurRepresenting the current overlay offset, SROIRepresenting the ROI offset, ulcurRepresents the current magnification, ul0 represents the initial magnification of the first image, and sl represents the preset switching magnification.
8. The method of claim 5, wherein slicing the ROI offset according to a magnitude relationship between the current magnification and the target magnification to obtain a current overlay offset for performing image stereo correction on the first image at a current time further comprises:
and if the current magnification is larger than the preset switching magnification, the current superposition offset is the ROI offset.
9. The method of claim 1, wherein performing image stereo correction on the first image based on the current overlay offset comprises:
constructing a target transformation matrix based on the current superposition offset and the image amplification parameter; the image magnification parameter comprises a central point of the first image and a preset switching magnification;
and calculating the product between the target transformation matrix and the homogeneous coordinates of the first image at the current moment, and determining the first image after image stereo correction according to the product calculation result.
10. The method of claim 1, further comprising:
after the ROI offset is calculated, detecting whether the ROI offset changes or not according to a preset time interval;
if the ROI offset is detected to be changed, determining a target ROI offset based on the changed ROI offset and the current superposition offset of the first image;
acquiring a first magnification, wherein the first magnification is the magnification of the first image when the ROI offset is detected to occur;
and re-determining the current superposition offset of the first image according to the first magnification and the target ROI offset, and performing image stereo correction on the first image according to the re-determined current superposition offset.
11. The method of claim 10, wherein re-determining a current overlay offset for the first image based on the first magnification and the target ROI offset comprises:
and if the first magnification ratio is the same as the second magnification ratio, the current superposition offset of the first image is the same as the superposition offset at the previous moment, wherein the second magnification ratio is the image magnification ratio corresponding to the image stereo correction operation at the previous moment.
12. The method of claim 11, wherein re-determining a current overlay offset for the first image based on the first magnification and the target ROI offset comprises:
and if the first magnification ratio is different from the second magnification ratio, determining the current superposition offset of the first image according to the first magnification ratio and the target ROI offset.
13. An image processing device is arranged in a mobile terminal, and the mobile terminal comprises: first and second imaging devices, the device comprising:
the first acquisition unit is used for acquiring a first image acquired by the first camera device; and determining a target ROI area in the first image;
the calculating unit is used for calculating the ROI offset of the target ROI area in the first image and a second image, wherein the second image is an image synchronously shot by the second camera when the first camera collects the first image;
a second acquisition unit configured to acquire a current magnification of the first image;
the slice processing unit is used for carrying out slice processing on the ROI offset according to the current magnification to obtain a current superposition offset for carrying out image stereo correction on the first image at the current moment so as to gradually superpose the current superposition offsets in different proportions in the process of amplifying the first image;
the image stereo correction unit is used for carrying out image stereo correction on the first image based on the current superposition offset to obtain the corrected first image;
the switching display unit is used for switching the mobile terminal from the first camera device to the second camera device for displaying if the current magnification is a preset switching magnification; wherein, at the time of switching, a sum total of overlay offset amounts by which the first image is subjected to image stereo correction is the ROI offset amount.
14. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the steps of the method of any of the preceding claims 1 to 12 are implemented when the computer program is executed by the processor.
15. A computer-readable medium having non-volatile program code executable by a processor, characterized in that the program code causes the processor to perform the steps of the method of any of the preceding claims 1 to 12.
CN202010649505.6A 2020-07-07 2020-07-07 Image processing method and device, electronic equipment and computer readable medium Active CN111935398B (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202010649505.6A CN111935398B (en) 2020-07-07 2020-07-07 Image processing method and device, electronic equipment and computer readable medium
PCT/CN2021/093706 WO2022007500A1 (en) 2020-07-07 2021-05-13 Image processing method and apparatus, and electronic device, readable medium and mobile terminal

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010649505.6A CN111935398B (en) 2020-07-07 2020-07-07 Image processing method and device, electronic equipment and computer readable medium

Publications (2)

Publication Number Publication Date
CN111935398A CN111935398A (en) 2020-11-13
CN111935398B true CN111935398B (en) 2022-04-22

Family

ID=73313391

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010649505.6A Active CN111935398B (en) 2020-07-07 2020-07-07 Image processing method and device, electronic equipment and computer readable medium

Country Status (2)

Country Link
CN (1) CN111935398B (en)
WO (1) WO2022007500A1 (en)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111935398B (en) * 2020-07-07 2022-04-22 北京迈格威科技有限公司 Image processing method and device, electronic equipment and computer readable medium
CN114531539B (en) * 2020-11-23 2024-03-19 华为技术有限公司 Shooting method and electronic equipment
CN113630549B (en) * 2021-06-18 2023-07-14 北京旷视科技有限公司 Zoom control method, apparatus, electronic device, and computer-readable storage medium
CN115550544B (en) * 2022-08-19 2024-04-12 荣耀终端有限公司 Image processing method and device
CN115200555B (en) * 2022-09-19 2022-11-25 中国科学院长春光学精密机械与物理研究所 Dynamic photogrammetry internal and external orientation element calibration method and device

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103096119B (en) * 2011-10-28 2015-10-07 浙江大华技术股份有限公司 A kind of camera zoom detection method and device
CN105959553B (en) * 2016-05-30 2018-12-04 维沃移动通信有限公司 A kind of switching method and terminal of camera
CN106161941B (en) * 2016-07-29 2022-03-11 南昌黑鲨科技有限公司 Automatic double-camera focus tracking method and device and terminal
CN205883406U (en) * 2016-07-29 2017-01-11 深圳众思科技有限公司 Automatic burnt device and terminal of chasing after of two cameras
CN107770433B (en) * 2016-08-15 2020-08-04 广州立景创新科技有限公司 Image acquisition device and image smooth scaling method thereof
CN106303258A (en) * 2016-09-19 2017-01-04 深圳市金立通信设备有限公司 A kind of image pickup method based on dual camera and terminal
US10429608B1 (en) * 2016-09-23 2019-10-01 Apple Inc. Primary-subordinate camera focus based on lens position sensing
CN107277360B (en) * 2017-07-17 2020-07-14 惠州Tcl移动通信有限公司 Method for zooming through switching of double cameras, mobile terminal and storage device
CN107896303A (en) * 2017-10-23 2018-04-10 努比亚技术有限公司 A kind of image-pickup method, system and equipment and computer-readable recording medium
CN108833768A (en) * 2018-05-10 2018-11-16 信利光电股份有限公司 A kind of image pickup method of multi-cam, camera terminal and readable storage medium storing program for executing
CN109379537A (en) * 2018-12-30 2019-02-22 北京旷视科技有限公司 Slide Zoom effect implementation method, device, electronic equipment and computer readable storage medium
CN110784651B (en) * 2019-11-15 2021-06-29 维沃移动通信有限公司 Anti-shake method and electronic equipment
CN111935398B (en) * 2020-07-07 2022-04-22 北京迈格威科技有限公司 Image processing method and device, electronic equipment and computer readable medium

Also Published As

Publication number Publication date
WO2022007500A1 (en) 2022-01-13
CN111935398A (en) 2020-11-13

Similar Documents

Publication Publication Date Title
CN111935398B (en) Image processing method and device, electronic equipment and computer readable medium
WO2021208371A1 (en) Multi-camera zoom control method and apparatus, and electronic system and storage medium
US9973672B2 (en) Photographing for dual-lens device using photographing environment determined using depth estimation
JP6271990B2 (en) Image processing apparatus and image processing method
CN111935397B (en) Image processing method and device, electronic equipment and computer readable medium
CN112261387B (en) Image fusion method and device for multi-camera module, storage medium and mobile terminal
JP5901447B2 (en) Image processing apparatus, imaging apparatus including the same, image processing method, and image processing program
CN113141448B (en) Image pickup apparatus
JP2015148532A (en) Distance measuring device, imaging apparatus, distance measuring method, and program
JP2018107526A (en) Image processing device, imaging apparatus, image processing method and computer program
CN114640833A (en) Projection picture adjusting method and device, electronic equipment and storage medium
CN111885371A (en) Image occlusion detection method and device, electronic equipment and computer readable medium
JP2023033355A (en) Image processing device and control method therefor
CN114095644B (en) Image correction method and computer equipment
JP6732440B2 (en) Image processing apparatus, image processing method, and program thereof
JP6648916B2 (en) Imaging device
JP7025237B2 (en) Image processing equipment and its control method and program
JP6486453B2 (en) Image processing apparatus, image processing method, and program
JP2020160773A (en) Image processing device, imaging device, image processing method, and program
JP6818734B2 (en) Image processing equipment, imaging equipment, image processing methods and programs
CN114143442B (en) Image blurring method, computer device, and computer-readable storage medium
CN111835968B (en) Image definition restoration method and device and image shooting method and device
JP7458769B2 (en) Image processing device, imaging device, image processing method, program and recording medium
JP2024002631A (en) Image processing device, imaging apparatus, image processing method, computer program, and recording medium
JP2015133532A (en) Imaging apparatus and image processing method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant