WO2023165535A1 - 图像处理方法、装置及设备 - Google Patents

图像处理方法、装置及设备 Download PDF

Info

Publication number
WO2023165535A1
WO2023165535A1 PCT/CN2023/079116 CN2023079116W WO2023165535A1 WO 2023165535 A1 WO2023165535 A1 WO 2023165535A1 CN 2023079116 W CN2023079116 W CN 2023079116W WO 2023165535 A1 WO2023165535 A1 WO 2023165535A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
line segment
processed
corrected
line segments
Prior art date
Application number
PCT/CN2023/079116
Other languages
English (en)
French (fr)
Inventor
杨晨
王小娥
田晶铎
白博
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2023165535A1 publication Critical patent/WO2023165535A1/zh

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/13—Edge detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/63—Control of cameras or camera modules by using electronic viewfinders
    • H04N23/631—Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • H04N23/681—Motion detection
    • H04N23/6812—Motion detection based on additional sensors, e.g. acceleration sensors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30168—Image quality inspection

Definitions

  • the present application relates to the field of image processing, and in particular to an image processing method, device and equipment.
  • the straight lines in the captured scene are shifted and distorted in the image (such as tilting or converging), which affects the beauty of the image composition.
  • the straight lines in the captured scene are shifted and distorted in the image (such as tilting or converging), which affects the beauty of the image composition.
  • the closer to the top of the building the more the edges of the outer walls converge.
  • the present application provides an image processing method, device and equipment, thereby solving the problem of correction errors caused by global correction of an image with axis-shift distortion based on image content, and improving the accuracy of image correction for axis-shift distortion.
  • an image processing method is provided, and the method is executed by a computing device.
  • the computing device acquires the image to be processed
  • the line segment to be corrected is obtained by filtering from the line segments contained in the image to be processed according to the inertial measurement unit (Inertial Measurement Unit, IMU) information contained in the metadata of the image to be processed and at least one of the field of view.
  • IMU Inertial Measurement Unit
  • the image to be processed includes at least one of a vertical line segment and a horizontal line segment in a first scene in which the image to be processed is acquired by the photographing device.
  • the line segment contained in the image to be processed may be a line segment whose line segment is not inclined in the image in the first scene, or a line segment in which the line segment in the first scene is inclined in the image.
  • the line segments to be corrected and the feature regions of the image to be processed are corrected to obtain a processed image.
  • the line segment to be corrected can be accurately located according to the IMU information and the field of view, and the line segment to be corrected can be selected from the line segments contained in the image to be processed. Correcting the line segment and the characteristic area of the image to be processed is corrected, so that the line segment in the processed image is more in line with the distribution characteristics of the horizontal line segment and the vertical line segment in the first scene captured by the shooting device. Compared with the overall correction of the image to be processed, the line segment in the image to be processed is precisely corrected, thereby effectively improving the accuracy of the image's axis-shift distortion correction.
  • the IMU information includes the horizontal inclination angle and the pitch inclination angle of the photographing device that captures the image to be processed.
  • the field of view indicates the maximum range that the camera can observe. Field of view is usually expressed in angles. The larger the field of view, the larger the observation range.
  • the shooting device shoots the first scene upwards or downwards, the vertical line segment in the first scene occurs in the image If tilted, the image to be processed includes the vertical line segment in the first scene that is inclined in the image to be processed.
  • the image to be processed includes the line segment in which the vertical line segment and the horizontal line segment in the first scene are tilted in the image to be processed.
  • the computing device performing image processing in the embodiment of the present application may be a shooting device with a shooting function (for example, a smart phone) or other devices with image processing functions (such as a server, a cloud device, an edge device, etc. ).
  • a shooting device with a shooting function for example, a smart phone
  • other devices with image processing functions such as a server, a cloud device, an edge device, etc.
  • the line segments to be corrected include at least one of vertical line segments and horizontal line segments in the first scene. It can be understood that the line segments to be corrected include some or all of the line segments in which at least one of the vertical line segment and the horizontal line segment in the first scene is inclined in the image to be processed.
  • the image to be processed includes a building image and a portrait with axis-shift distortion
  • the line segment to be corrected includes at least one of the vertical line segment and the horizontal line segment in the building that is inclined in the image, excluding the inclined line segment in the portrait, Only the inclined line segments in the building image are corrected, so as to preserve the effect of long legs in the portrait and make the image composition more beautiful.
  • the shooting device displays the axis-shift distortion control
  • the method further includes: after the shooting device receives the user's operation on the axis-shift distortion control, screening the line segments included in the image to be processed according to the metadata of the image to be processed
  • the line segment to be corrected is obtained, the line segment to be corrected and the feature area of the image to be processed are corrected, and the processed image is obtained.
  • the image is corrected for axis-shift distortion, which reduces the complexity of the operation steps for users to perform axis-shift distortion on images, and improves the flexibility of image correction for axis-shift distortion, thereby improving the user experience of users in image processing .
  • the method further includes: displaying the processed image after correcting the line segment to be corrected and the feature region of the image to be processed to obtain the processed image. Therefore, the user can more intuitively see the image after the axis-shift distortion correction, and the user experience of processing the image is improved.
  • a line segment screening strategy is formulated according to the content contained in the metadata to filter the line segments to be corrected.
  • the line segment filtering strategy is used to indicate the line segment filtering method for the line segments contained in the image to be processed.
  • the line segment to be corrected is obtained by screening the line segment included in the image to be processed according to the metadata of the image to be processed. Therefore, different line segment screening strategies are adapted according to the content contained in the metadata, and the flexibility and accuracy of line segment screening for the line segments contained in the image to be processed are improved.
  • the metadata includes IMU information and field of view; the line segment screening strategy is used to indicate the line segment screening based on the IMU information and field of view; according to the line segment screening strategy determined by the metadata, the line segment to be corrected is filtered from the line segments contained in the image to be processed , including: according to the IMU information and the field of view, the M line segments contained in the image to be processed are adjusted as a whole to obtain the adjusted M line segments. N line segments to be corrected are obtained by screening out the line segments.
  • the line slope threshold range includes a horizontal line slope threshold and a vertical line slope threshold, both M and N are positive integers, and N is less than or equal to M.
  • the shooting device filters the oblique line segments in the image to be processed based on the IMU information and the field of view, which helps to improve the accuracy of line segment screening, thereby improving the image's axis-shifting distortion. Correction accuracy.
  • the metadata does not include the field of view.
  • the line segment to be corrected is obtained from the line segments contained in the image to be processed, including: based on the image content of the image to be processed
  • the line segment to be corrected is obtained by filtering the line segment contained in the image, and the image content includes the angle distribution feature of the line segment.
  • the shooting device accurately screens the line segments based on the inclination angle of the line segments in the image content, thereby improving the accuracy of the axis-shift distortion correction of the image.
  • the metadata includes the field of view and does not include IMU information.
  • the line segment to be corrected is obtained from the line segments contained in the image to be processed, including: based on the field of view and the image content of the image to be processed.
  • the line segment to be corrected is obtained by filtering the line segment included in the image to be processed, and the image content includes the angle distribution feature of the line segment. Due to the different inclination angles of the line segments in different field of view ranges, the larger the field of view, the greater the inclination angle of the edge line segments in the image.
  • the shooting device performs precise screening of line segments based on the inclination angle of the line segment in the image content and the field of view, thereby improving the accuracy of the image's axis-shift distortion correction.
  • the method further includes: prompting the user to select a line segment to be corrected from recommended line segments filtered based on image content of the image to be processed.
  • a line segment to be corrected from recommended line segments filtered based on image content of the image to be processed.
  • users may use upward shooting or downward shooting to capture images for composition effect. At this time, the user may not want to correct the slanted line segment in the image.
  • the shooting device provides a user-interactive straight line recommendation strategy, prompts the user whether to perform line segment correction, and corrects the image for axis-shift distortion according to the user's wishes, so as to improve the accuracy of image axis-shift distortion correction.
  • the metadata also includes a credible identifier, and the credible identifier is used to indicate whether the metadata is credible; according to the line segment screening strategy determined by the metadata, the line segments included in the image to be processed are screened to obtain the Correcting the line segment includes: if the trusted identifier indicates that the metadata is untrustworthy, based on the image content of the image to be processed, the line segment to be corrected is selected from the line segments contained in the image to be processed, and the image content includes the angle distribution characteristics of the line segment. If the credible flag indicates that the metadata is credible, based on at least one of the image content, IMU information, and field of view of the image to be processed, the line segment to be corrected is obtained by screening the line segments contained in the image to be processed.
  • the line segment to be corrected is obtained by screening the line segments contained in the image to be processed according to the metadata of the image to be processed, including: determining the line segment to be corrected according to the line segment screened based on the metadata and the line segment screened based on the image content line segment. In this way, combining the results of two different line segment screening strategies, the line segment to be corrected is determined, and the accuracy of line segment screening based on IMU information and field of view is further improved, thereby improving the accuracy of image axis-shift distortion correction.
  • correcting the line segment to be corrected and the feature area of the image to be processed to obtain the processed image includes: judging whether the number of line segments to be corrected is greater than the number threshold, and whether the length of the line segment to be corrected is greater than the length Threshold; if the number of line segments to be corrected is greater than the number threshold, and the length of the line segment to be corrected is greater than the length threshold, the line segment to be corrected is corrected to obtain the processed image; if the number of line segments to be corrected is less than or equal to the number threshold, or the length of the line segment to be corrected The length is less than or equal to the length threshold, and whether to correct the line segment to be corrected is determined according to the scene features of the first scene, and the scene features include architectural features and character features.
  • the line segment to be corrected and the feature area of the image to be processed are corrected to obtain the processed image, including: constructing a straight line constraint to correct the line segment to be corrected; constructing a content-based homography constraint and shape and Regular constraints correct the background area; build portrait constraints to correct the portrait area; Construct boundary constraints to rectify the edge area to obtain the processed image.
  • the line segment contained in the image to be processed is screened according to the metadata of the image to be processed, but the line segment that needs to be corrected is not screened out, and the image to be processed is corrected according to the characteristic area of the image to be processed, it can effectively compensate for the error of line segment screening, and then improve the accuracy of the image.
  • the accuracy with which the image is corrected for axis-shift distortion is improved.
  • the line segment to be corrected is obtained by filtering the line segments contained in the image to be processed according to the metadata of the image to be processed, including: detecting the image to be processed according to a line detection method to obtain a line segment set.
  • the line segment set contains M inclined line segments, where M is a positive integer.
  • the method includes: updating metadata of the image to be processed including IMU information. For example, delete or modify IMU information. Therefore, the accuracy of line segment screening using IMU information is improved.
  • an image processing device in a second aspect, includes various modules for executing the image processing method in the first aspect or any possible design of the first aspect.
  • a shooting device in a third aspect, includes at least one processor and a memory, and the memory is used to store a set of computer instructions; When the device executes the set of computer instructions, it executes the first aspect or the operation steps of the image processing method in any possible implementation manner of the first aspect.
  • a computing device in a fourth aspect, includes at least one processor and a memory, and the memory is used to store a set of computer instructions; When the device executes the set of computer instructions, it executes the first aspect or the operation steps of the image processing method in any possible implementation manner of the first aspect.
  • a computer-readable storage medium including: computer software instructions; when the computer software instructions are run in the computing device, the computing device is made to execute the computer program described in the first aspect or any one of the possible implementation manners of the first aspect. Operational steps of the method.
  • a computer program product is provided.
  • the computing device executes the operation steps of the method described in the first aspect or any possible implementation manner of the first aspect.
  • FIG. 1 is a schematic diagram of an axis-shift distorted image and a corrected image provided by an embodiment of the present application
  • FIG. 2 is a schematic diagram of an axis-shift distortion correction scene provided by an embodiment of the present application
  • FIG. 3 is a schematic structural diagram of a shooting device provided in an embodiment of the present application.
  • FIG. 4 is a schematic diagram of an image processing method provided in an embodiment of the present application.
  • FIG. 5 is a schematic structural diagram of metadata provided by an embodiment of the present application.
  • FIG. 6 is a schematic diagram of line segment screening based on IMU information and field of view provided by the embodiment of the present application;
  • FIG. 7 is a schematic diagram of a line segment screening visualization provided in the embodiment of the present application.
  • Fig. 8 is a schematic diagram of another image processing method provided by the embodiment of the present application.
  • FIG. 9 is a schematic diagram of line segment screening based on image content provided by an embodiment of the present application.
  • FIG. 10 is a schematic diagram of line segment screening based on the field of view and image content provided by the embodiment of the present application.
  • Fig. 11 is a schematic diagram of another image processing method provided by the embodiment of the present application.
  • Fig. 12 is a schematic diagram of another image processing method provided by the embodiment of the present application.
  • FIG. 13 is a schematic diagram of a line segment selection interface provided by the embodiment of the present application.
  • Fig. 14 is a schematic diagram of another image processing method provided by the embodiment of the present application.
  • FIG. 15 is a schematic diagram of an image processing interface provided by an embodiment of the present application.
  • FIG. 16 is a schematic diagram of another image processing interface provided by the embodiment of the present application.
  • FIG. 17 is a schematic structural diagram of an image processing device provided by an embodiment of the present application.
  • FIG. 18 is a schematic structural diagram of a shooting device provided by an embodiment of the present application.
  • the embodiment of the present application provides an image processing method, in particular, a method for correcting an axis-shift distortion of an image, that is, according to the metadata of the image to be processed, the line segment to be corrected is selected from the line segments contained in the image to be processed, and the line segment to be corrected is obtained from the line segment to be corrected.
  • the corrected line segment can be at least one of the vertical line segment and the horizontal line segment in the actual scene that is inclined in the image to be processed, and the line segment to be corrected and the feature area of the image to be processed are corrected to make the line segment in the processed image more suitable for shooting
  • the distribution characteristics of horizontal line segments and vertical line segments in the actual scene captured by the device Compared with the overall correction of the image to be processed, the precise correction is performed on the oblique line segment in the image to be processed, thereby effectively improving the accuracy of the axis-shift distortion correction of the image.
  • Axis-shift distortion refers to the phenomenon that the imaging plane of the shooting device rotates on at least one of the x-axis, y-axis, and z-axis relative to the target plane of the captured scene, so that the original parallel lines in the target plane are tilted or skewed on the imaging plane .
  • the image obtained by rotating the camera along the z axis that is, the image obtained by shooting horizontally obliquely, relative to the horizontal line segment and vertical line segment in the captured scene
  • the image contains horizontally oblique line segments and vertically slanted line segments.
  • it is the image after the correction of the axis shift distortion, and the horizontal and vertical line segments are corrected, so that the image after the correction of the axis shift distortion is more in line with the horizontal line segment and the vertical line segment in the actual scene captured by the shooting equipment. Distribution characteristics of line segments.
  • the image obtained by rotating the camera along the x-axis that is, the image obtained by the camera shooting upwards, contains vertically inclined line segments relative to the vertical line segments in the captured scene.
  • the device that performs image shift distortion correction has strong image processing and computing capabilities, and the device can be a shooting device, server, cloud device or edge device (for example, a box with a chip with processing capabilities) wait.
  • the shooting device obtains the image to be processed from the local or the cloud, and uses the processor (such as a graphics processing unit (GPU)) of the shooting device to move the image to be processed.
  • the processor such as a graphics processing unit (GPU)
  • Axis distortion correction is shown in (a) in Figure 2.
  • the shooting device uploads the image to be processed, and the server, cloud device or edge device performs axis-shift distortion correction on the image to be processed.
  • the shooting device can be an integrated module.
  • the shooting device includes a camera, an inertial measurement unit (Inertial Measurement Unit, IMU), a communication module and a processor.
  • the shooting device can be a terminal device, such as Mobile terminal, tablet computer, notebook computer, wearable device, virtual reality (virtual reality, VR) device, augmented reality (augmented reality, AR) device, mixed reality (Mixed Reality, MR) device, extended reality (Extended Reality, ER) ) devices and cameras etc.
  • virtual reality virtual reality
  • AR augmented reality
  • MR Mixed reality
  • ER Extended Reality
  • FIG. 3 is a schematic structural diagram of a photographing device provided in an embodiment of the present application.
  • the photographing device 300 includes a processor 310, an external memory interface 320, an internal memory 321, a universal serial bus (universal serial bus, USB) interface 330, a power management module 340, an antenna, a wireless communication module 360, an audio module 370, a loudspeaker 370A, Speaker interface 370B, microphone 370C, sensor module 380, button 390, indicator 391, display screen 392, camera 393, etc.
  • a processor 310 includes a processor 310, an external memory interface 320, an internal memory 321, a universal serial bus (universal serial bus, USB) interface 330, a power management module 340, an antenna, a wireless communication module 360, an audio module 370, a loudspeaker 370A, Speaker interface 370B, microphone 370C, sensor module 380, button 390, indicator 391, display screen 392, camera 393, etc.
  • USB universal serial bus
  • the above-mentioned sensor module 380 may include sensors such as pressure sensor, gyroscope sensor, air pressure sensor, magnetic sensor, acceleration sensor, image sensor, distance sensor, proximity light sensor, fingerprint sensor, temperature sensor, touch sensor and ambient light sensor.
  • the image sensor is used to convert the light image on the photosensitive surface into an electrical signal proportional to the light image by using the photoelectric conversion function of the photoelectric device.
  • the structure shown in this embodiment does not constitute a specific limitation on the photographing device.
  • the photographing device may include more or fewer components than shown in the illustration, or some components may be combined, or some components may be separated, or different component arrangements may be made.
  • the illustrated components can be realized in hardware, software or a combination of software and hardware.
  • the processor 310 may include one or more processing units, for example: the processor 310 may include an application processor (application processor, AP), a modem processor, a GPU, an image signal processor (image signal processor, ISP), a control device, memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.
  • the NPU is a neural-network (NN) computing processor.
  • NPU neural-network
  • the processor 310 is configured to select the line segment to be corrected from the line segments included in the image to be processed according to the line segment screening policy determined by the metadata of the image to be processed.
  • the line segment filtering strategy is used to indicate the line segment filtering method for the line segments contained in the image to be processed.
  • the line segments to be corrected include at least one of the line segments with oblique features in the image to be processed.
  • Metadata also known as intermediary data or relay data, is the data describing data (data about data), mainly describing the information of data attributes (property), used to support such as indicating storage location, historical data, resource search, File recording and other functions.
  • Metadata is some tags embedded in the image file to be processed, which is used to describe the information of image data attributes, such as lens parameters, exposure parameters, sensor information, and so on.
  • cameras automatically add metadata to image files when a photo is taken.
  • metadata include Inertial Measurement Unit (IMU) information and Field of View.
  • IMU information includes but not limited to three-axis attitude angle and acceleration. The field of view indicates the maximum range that the camera can observe, usually expressed in angles. The larger the field of view, the larger the observation range.
  • the inertial measurement component includes a gyroscope sensor and an acceleration sensor.
  • the gyro sensor can be used to determine the motion posture of the photographing device 300 .
  • the angular velocity of the photographing device 300 around three axes may be determined by a gyro sensor.
  • Gyroscope transmission The sensor can be used for image stabilization. Exemplarily, when the shutter is pressed, the gyro sensor detects the shaking angle of the shooting device 300, calculates the distance that the lens module needs to compensate according to the angle, and allows the lens to counteract the shaking of the shooting device 300 through reverse movement to achieve anti-shake. Gyroscope sensors can also be used for navigation and somatosensory game scenes.
  • the acceleration sensor can detect the acceleration of the photographing device 300 in various directions (generally three axes).
  • the magnitude and direction of gravity can be detected when the photographing device 300 is stationary. It can also be used to identify the posture of the shooting device, and can be used in applications such as horizontal and vertical screen switching, pedometers, etc.
  • the controller can be the nerve center and command center of the shooting equipment.
  • the controller can generate an operation control signal according to the instruction opcode and timing signal, and complete the control of fetching and executing the instruction.
  • a memory may also be provided in the processor 310 for storing instructions and data.
  • the memory in processor 310 is a cache memory.
  • the memory may hold instructions or data that the processor 310 has just used or recycled. If the processor 310 needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 310 is reduced, thereby improving the efficiency of the system.
  • processor 310 may include one or more interfaces.
  • the interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous transmitter (universal asynchronous receiver/transmitter, UART) interface, mobile industry processor interface (mobile industry processor interface, MIPI), general-purpose input/output (general-purpose input/output, GPIO) interface, and/or USB interface, etc.
  • I2C integrated circuit
  • I2S integrated circuit built-in audio
  • PCM pulse code modulation
  • PCM pulse code modulation
  • UART universal asynchronous transmitter
  • MIPI mobile industry processor interface
  • GPIO general-purpose input/output
  • USB interface etc.
  • the power management module 340 is used for connecting power.
  • the power management module 340 may also be connected with the processor 310, the internal memory 321, the display screen 392, the camera 393, the wireless communication module 360, and the like.
  • the power management module 340 receives power input and supplies power to the processor 310 , the internal memory 323 , the display screen 392 , the camera 393 and the wireless communication module 360 .
  • the power management module 340 can also be disposed in the processor 310 .
  • the wireless communication function of the photographing device can be realized through the antenna and the wireless communication module 360 and the like.
  • the wireless communication module 360 can provide wireless local area network (wireless local area networks, WLAN) (such as wireless fidelity (wireless fidelity, Wi-Fi) network), bluetooth (bluetooth, BT), global navigation Satellite system (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions.
  • the wireless communication module 360 may be one or more devices integrating at least one communication processing module.
  • the wireless communication module 360 receives electromagnetic waves via the antenna, frequency-modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 310 .
  • the wireless communication module 360 can also receive the signal to be transmitted from the processor 310, frequency-modulate it, amplify it, and convert it into electromagnetic wave and radiate it through the antenna.
  • the antenna of the shooting device is coupled to the wireless communication module 360, so that the shooting device can communicate with the network and other devices through wireless communication technology.
  • the shooting device realizes the display function through the GPU, the display screen 392, and the application processor.
  • the GPU is a microprocessor for image processing, connected to the display screen 392 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering.
  • Processor 310 may include one or more GPUs that execute program instructions to generate or alter display information.
  • the display screen 392 is used to display images, videos and the like. As in the embodiment of this application, the display screen 392 is used to display the Processed image and processed image.
  • the display screen 392 includes a display panel.
  • the display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light emitting diode or an active matrix organic light emitting diode (active-matrix organic light emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diodes (quantum dot light emitting diodes, QLED), etc.
  • LCD liquid crystal display
  • OLED organic light-emitting diode
  • AMOLED active matrix organic light emitting diode
  • FLED flexible light-emitting diode
  • Miniled MicroLed, Micro-oLed
  • the shooting device can realize the shooting function through ISP, camera 393 , video codec, GPU, display screen 392 and application processor.
  • the ISP is used for processing the data fed back by the camera 393 .
  • the ISP may be located in the camera 393 .
  • Camera 393 is used to capture still images or video.
  • the object generates an optical image through the lens and projects it to the photosensitive element.
  • the photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor.
  • CMOS complementary metal-oxide-semiconductor
  • the photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal.
  • the ISP outputs the digital image signal to the DSP for processing.
  • DSP converts digital image signals into standard RGB, YUV and other image signals.
  • the photographing device may include 1 or N cameras 393, where N is a positive integer greater than 1. The embodiment of the present application does not limit the position of the camera 393 on the shooting device.
  • the shooting device may not include a camera, that is, the above-mentioned camera 393 is not set in the shooting device (such as a TV set).
  • the shooting device can be connected with an external camera 393 through an interface (such as a USB interface 330).
  • the external camera 393 can be fixed on the shooting device through an external fixing member (such as a camera bracket with a clip).
  • the external camera 393 may be fixed at the edge of the display screen 392 of the shooting device, such as the upper edge, through an external fixing member.
  • Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the shooting device selects the frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.
  • Video codecs are used to compress or decompress digital video.
  • a capture device can support one or more video codecs. In this way, the shooting device can play or record videos in various encoding formats, such as: moving picture experts group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4, etc.
  • the external memory interface 320 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the shooting device.
  • the external memory card communicates with the processor 310 through the external memory interface 320 to implement a data storage function. Such as saving music, video, image and other files in external memory card.
  • the internal memory 321 may be used to store computer-executable program code, which includes instructions.
  • the processor 310 executes various functional applications and data processing of the photographing device by executing instructions stored in the internal memory 321 .
  • the internal memory 321 may include an area for storing programs and an area for storing data.
  • the stored program area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.) and the like.
  • the storage data area can store data (such as audio data, etc.) created during the use of the shooting device.
  • the internal memory 321 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (universal flash storage, UFS) and the like.
  • the shooting device can pass through the audio module 370, the speaker 370A, the microphone 370C, the speaker interface 370B, and an application processor to implement audio functions. For example, music playback, recording, etc.
  • the microphone 370C may be used to receive voice commands from the user to the shooting device.
  • the speaker 370A can be used to feed back decision-making instructions of the shooting device to the user.
  • the audio module 370 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal.
  • the audio module 370 may also be used to encode and decode audio signals.
  • the audio module 370 can be set in the processor 310 , or some functional modules of the audio module 370 can be set in the processor 310 .
  • Speaker 370A also called “horn” is used to convert audio electrical signals into sound signals.
  • the microphone 370C also called “microphone” or “microphone”, is used to convert sound signals into electrical signals.
  • the speaker interface 370B is used to connect wired speakers.
  • the speaker interface 370B can be a USB interface 330, or a 3.5mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.
  • OMTP open mobile terminal platform
  • CTIA cellular telecommunications industry association of the USA
  • the keys 390 include a power key, a volume key and the like.
  • the key 390 may be a mechanical key. It can also be a touch button.
  • the shooting device can receive key input and generate key signal input related to user settings and function control of the shooting device.
  • the indicator 391 may be an indicator light, and may be used to indicate that the photographing device is in a power-on state, a standby state, or a power-off state. For example, if the indicator light is off, it may indicate that the shooting device is in a shutdown state; if the indicator light is green or blue, it may indicate that the shooting device is in a power-on state; if the indicator light is red, it may indicate that the shooting device is in a standby state.
  • the structure shown in the embodiment of the present application does not constitute a specific limitation on the photographing device. It may have more or fewer components than shown in FIG. 3 , may combine two or more components, or may have a different configuration of components.
  • the photographing device may also include components such as a sound box.
  • the various components shown in Figure 3 may be implemented in hardware, software, or a combination of hardware and software including one or more signal processing or application specific integrated circuits.
  • the methods in the following embodiments can all be implemented in the shooting device with the above hardware structure.
  • the method in the embodiments of the present application will be described by taking the aforementioned photographing device as a smart phone as an example.
  • FIG. 4 is a schematic flowchart of an image processing method provided by an embodiment of the present application.
  • description will be made by taking the camera device 300 as an example to correct the axis-shift distortion of the image.
  • the method includes the following steps.
  • Step 410 the photographing device 300 acquires the image to be processed.
  • the photographing device 300 may perform real-time photographing through a camera, and use the captured images as images to be processed.
  • the photographing device 300 may also acquire images to be processed from a gallery stored in the memory.
  • the shooting device 300 will also acquire images to be processed from the cloud or other devices.
  • the embodiment of the present application does not limit the source of the image to be processed.
  • the image to be processed acquired by the shooting device 300 may be an unedited real-time captured image or an edited image.
  • the image to be processed has been edited by a third-party image application or by an image editing algorithm carried by the shooting device 300 .
  • Image editing includes geometric transformation operations such as cropping, rotating, and straightening.
  • the image to be processed may also be a preprocessed image.
  • Preprocessing includes black level correction, noise reduction, auto exposure, auto white balance and image warping etc.
  • the image to be processed in the embodiment of the present application may be obtained by shooting the first scene when the shooting device 300 is tilted (for example: shooting from above, shooting down, or tilting horizontally).
  • the image to be processed contains vertical and horizontal lines in the first scene at least one of the segments.
  • the image to be processed includes at least one of the vertical line segment and the horizontal line segment in the first scene that is inclined in the image to be processed.
  • the vertical line segment that is inclined in the image to be processed may be referred to as a vertically inclined line segment.
  • the inclined horizontal line segment in the image to be processed in the first scene may be referred to as a horizontal inclined line segment.
  • the vertically inclined line segment may refer to an inclined line segment within the vertical line segment inclination threshold relative to the line segment perpendicular to the horizon.
  • the horizontally inclined line segment may refer to an inclined line segment within the horizontal line segment inclination threshold relative to the line segment parallel to the horizon. Both the vertical line segment inclination threshold and the horizontal line segment inclination threshold may be 5°.
  • the camera is tilted horizontally to capture an image, and the image includes a horizontally inclined line segment and a vertically inclined line segment.
  • the image obtained by the camera looking up the top of the building gradually shrinks, and the image contains vertically inclined line segments.
  • the image obtained by the camera looking down the bottom of the building gradually shrinks, and the image contains vertical sloping line segments.
  • Step 420 the photographing device 300 detects the line segment in the image to be processed according to the line detection method.
  • the line detection method can be Hough line detection or LSD line detection.
  • the photographing device 300 detects the image to be processed according to the line detection method, and obtains M line segments included in the image to be processed, where M is a positive integer.
  • the M line segments include at least one of a horizontal line segment, a vertical line segment, a horizontal inclined line segment, and a vertical inclined line segment.
  • Step 430 the photographing device 300 obtains the line segment to be corrected from the line segments included in the image to be processed according to the IMU information included in the metadata of the image to be processed and the field of view.
  • the shooting device 300 can accurately locate the line segment that needs to be corrected according to the IMU information and the field of view, and select the line segment to be corrected from the line segments included in the image to be processed , correcting the line segment to be corrected and the characteristic area of the image to be processed, so that the line segment in the processed image is more in line with the distribution characteristics of the horizontal line segment and the vertical line segment in the first scene captured by the shooting device.
  • metadata includes original data and newly added data.
  • Raw data includes lens parameters and exposure parameters.
  • the new data contains at least one of IMU information and field of view.
  • the IMU information may refer to an inclination angle of a device that captures an image to be processed, for example, the IMU information includes a horizontal inclination angle and a pitch inclination angle.
  • the image to be processed is captured by the capturing device 300
  • the IMU information may be the IMU information of the capturing device 300 .
  • the method flow described in FIG. 6 is an illustration of the specific operation process included in step 430 in FIG. 4 , as shown in FIG. 6 .
  • Step 431 the shooting device 300 performs overall adjustment on the M line segments contained in the image to be processed according to the IMU information and the field of view to obtain adjusted M line segments.
  • the M line segments detected by the shooting device 300 in the image to be processed according to the straight line detection method are not all vertical or horizontal line segments in the first scene. If the M line segments are screened according to the line segment slope threshold, wrong screening may easily occur. For example, line segments such as portraits and clothes are not vertical or horizontal in actual scenes. Therefore, the shooting device 300 performs an overall adjustment on the M line segments contained in the image to be processed according to the IMU information and the field of view to obtain the adjusted M line segments, so that the adjusted M line segments conform to the line segment angle distribution characteristics of the first scene, that is, the adjusted M line segments The line segments are more in line with the characteristics of vertical line segments and horizontal line segments in the first scene.
  • the shooting device 300 performs perspective transformation on the M line segments to obtain the adjusted M line segments, that is, the M line segments after perspective transformation.
  • the shooting device 300 detects a set of line segments in the image to be processed based on the line detection method M represents the number of line segments.
  • the line segment l i is defined by the starting point coordinates and end point coordinates constitute.
  • the shooting device 300 estimates the perspective transformation matrix H according to the pitch angle, horizontal tilt angle and field of view contained in the metadata, and according to the perspective transformation After the matrix H performs perspective transformation operations on the starting point coordinates and end point coordinates of M line segments, the set of line segments after perspective transformation is obtained
  • the coordinates of the starting point of the line segment l′ i are and the end point coordinates are Exemplarily, the perspective transformation of coordinates (x, y) is as follows formula (1).
  • H represents the perspective transformation matrix.
  • Step 432 the shooting device 300 screens out N line segments to be corrected from the adjusted M line segments according to the line segment inclination threshold.
  • the M line segments include line segments with a large inclination angle relative to the 90° vertical line segment, line segments with a small inclination angle relative to the 90° vertical line segment, line segments with a large inclination angle relative to the 0° horizontal line segment, and line segments with a small inclination angle relative to the 0° horizontal line segment .
  • the line segment with a large inclination angle is not a vertical line segment or a horizontal line segment in the first scene, and may not need to be corrected.
  • the shooting device 300 can select N line segments to be corrected from the adjusted M line segments according to the line segment inclination threshold, and N line segments to be corrected It is more in line with the line segment angle distribution characteristics of the first scene, thereby correcting the N line segments to be corrected, and improving the accuracy of the axis-shift distortion correction of the image to be processed.
  • N is a positive integer, and N is less than or equal to M. It is understandable that N is equal to M, which means that the adjusted M line segments are all line segments to be corrected; N being smaller than M, means that some of the adjusted M line segments are line segments to be corrected.
  • the line segment inclination threshold includes a horizontal line segment inclination threshold and a vertical line segment inclination threshold.
  • the shooting device 300 calculates the angle of the line segment according to the coordinates of the starting point and the coordinates of the end point of each of the adjusted M line segments to obtain a set of angles of the adjusted M line segments For any line segment l' i , judge whether the angle ⁇ ' i belongs to the threshold range of the horizontal line segment inclination or vertical line segment slope threshold range That is to identify whether the angle ⁇ ' i is close to 90° or 0°, and filter to obtain a set of horizontally inclined line segments and a set of vertically inclined line segments.
  • the values of ⁇ h and ⁇ v may both be less than 5°.
  • FIG. 7 it is a schematic diagram of a visualization of line segment screening provided by the embodiment of the present application.
  • it is all line segments obtained by detecting the image to be processed according to the line detection method, including line segments such as portraits, clothes, and chairs.
  • line segments such as portraits, clothes, and chairs.
  • it is a line segment conforming to the real angle distribution in the actual scene after perspective transformation.
  • the line segment to be corrected is screened for the angle of the line segment after the perspective transformation is close to 90° or 0°.
  • the line segments to be corrected include some or all of the line segments in which at least one of the vertical line segment and the horizontal line segment in the first scene is tilted in the image to be processed.
  • the image to be processed includes a building image and a portrait with axis-shift distortion
  • the line segment to be corrected includes at least one of the vertical line segment and the horizontal line segment in the building that is inclined in the image, excluding the inclined line segment in the portrait, Only the inclined line segments in the building image are corrected, so as to preserve the effect of long legs in the portrait and make the image composition more beautiful.
  • Step 440 the photographing device 300 corrects the line segment to be corrected and the feature area of the image to be processed to obtain a processed image.
  • the photographing device 300 corrects the image data of the image to be processed, and the image data includes pixels of the image to be processed.
  • the shooting device 300 can construct constraint items in subregions of the image to be processed, and jointly optimize each constraint item, Obtain the global grid correction displacement, perform image affine transformation on the image to be processed, and obtain the processed image.
  • Regions may include straight line regions, edge regions, background regions, and portrait regions.
  • the straight line constraint constructs horizontal, horizontal and vertical constraints on the vertical line segment and the horizontal line segment in the first scene included in the filtered line segment to be corrected.
  • a content-based homography constraint may be a background region constraint.
  • the homography constraints are as follows formula (2).
  • ⁇ H represents weight, 0 ⁇ H ⁇ 1.0.
  • the coordinate vector after perspective transformation (u,v) (U,V)/W, x i,j and y i,j represent the original coordinates, and i, j indicate the grid point corresponding to row i and column j.
  • the portrait constraint constructs the geometric transformation constraints of the portrait area according to the position of the portrait.
  • the grid point coordinates of the portrait area are constrained to satisfy the spherical transformation relationship, so that perspective distortion such as edge stretching of the portrait can be optimized on the basis of correcting the axis-shift distortion of the background and portrait.
  • the head-to-body ratio of the portrait is out of balance, that is, the head image is larger and the leg image is shorter.
  • the head-to-body ratio of the portrait is out of balance, that is, the head image is smaller and the leg image is longer.
  • the head image or body image of the portrait at the edge of the image is stretched and distorted. Construct a portrait constraint according to the portrait distortion phenomenon, and correct the distortion in the portrait image according to the portrait constraint.
  • Boundary constraints are used to constrain the displacement of pixels corresponding to the edge area of the field of view to the outside and inside. In order to reduce the loss of field of view, the pixels on the upper, lower, left, and right edges of the image are constrained to move outward.
  • the boundary constraints are as follows formula (3).
  • H represents the length of the image
  • W represents the width of the image
  • v i, j (v i, j ⁇ 0 or v i, j >H) represent the x and y coordinates of the pixel point displacement
  • ⁇ B_in represents the weight, 0 ⁇ B_in ⁇ 1.0.
  • the pixels on the upper, lower, left, and right edges of the image are constrained to move inward.
  • the boundary constraints are as follows formula (4).
  • u i, j (0 ⁇ u i, j ⁇ W), v i, j (0 ⁇ v i, j ⁇ H) represent the x and y coordinates of the pixel point displacement, ⁇ B_out represents the weight, 0 ⁇ B_out ⁇ 1.0.
  • Shape Constraints Build shape-preserving constraints to avoid image texture distortion.
  • the transformation of grid points satisfies the constraints of local similarity transformation or shape-preserving transformation.
  • the shape constraints are as follows formula (5).
  • E S ⁇ S ( ⁇ i,j ((v i+1,j -v i,j )+(u i,j+1 -u i,j )) 2 + ⁇ i,j ((u i+ 1,j -u i,j )+(v i,j+1 -v i,j )) 2 )Formula (5)
  • ⁇ S represents the weight, 0 ⁇ S ⁇ 1.0.
  • the second derivative approximation is computed using finite differences and given smoothness constraints using the Frobenius norm.
  • Smoothness constraints may refer to shape constraints.
  • the smoothness constraint is as follows formula (6).
  • ⁇ R represents weight, 0 ⁇ R ⁇ 1.0.
  • Jointly optimize constraints such as straight line constraints, content shape preservation (content-based homography constraints), field of view boundary constraints (boundary constraints), portrait constraints, shape constraints, and smoothness constraints.
  • content shape preservation content-based homography constraints
  • field of view boundary constraints boundary constraints
  • portrait constraints shape constraints
  • shape constraints and smoothness constraints.
  • the corrected image is more reasonable and beautiful.
  • the field of view remains more.
  • the photographing device 300 may also formulate a line segment screening strategy according to the content contained in the metadata to filter the line segments to be corrected.
  • the line segment filtering strategy is used to indicate the line segment filtering method for the line segments contained in the image to be processed. Therefore, different line segment filtering strategies are adapted according to the content contained in the metadata to improve the flexibility and accuracy of line segment filtering.
  • FIG. 8 is a schematic flowchart of an image processing method provided by an embodiment of the present application.
  • description will be made by taking the camera device 300 as an example to correct the axis-shift distortion of the image.
  • the method includes the following steps.
  • Step 810 the photographing device 300 acquires an image to be processed.
  • Step 820 the photographing device 300 detects the line segment in the image to be processed according to the line detection method.
  • step 810 and step 820 For a detailed explanation of step 810 and step 820, reference may be made to the description of step 410 and step 420 above, and details are not repeated here.
  • step 830 the photographing device 300 selects the line segments to be corrected from the line segments included in the image to be processed according to the line segment screening policy determined by the metadata.
  • the line segments to be corrected include some or all of the line segments in which at least one of the vertical line segment and the horizontal line segment in the first scene is tilted in the image to be processed.
  • the first scene refers to a scene when the image to be processed is captured.
  • the line segments to be corrected include vertical line segments in the first scene.
  • the line segment to be corrected includes a horizontal line segment in the first scene.
  • the line segments to be corrected include vertical line segments and horizontal line segments in the first scene.
  • the image to be processed includes a building image and a portrait with axis-shift distortion
  • the line segment to be corrected includes at least one of the vertical line segment and the horizontal line segment in the building that is inclined in the image, excluding the inclined line segment in the portrait, Only the inclined line segments in the building image are corrected, so as to preserve the effect of long legs in the portrait and make the image composition more beautiful.
  • the photographing device 300 may determine different line segment filtering strategies according to different contents included in the metadata.
  • step 830 includes step 831 to step 834 .
  • Step 831 the photographing device 300 determines whether the metadata includes IMU information and field of view.
  • the line segment screening policy indicates to perform line segment screening based on the IMU information and field of view, and the shooting device 300 executes step 832, that is, the shooting device 300 filters the line segments included in the image to be processed based on the IMU information and field of view Get the line segment to be corrected.
  • the shooting device 300 filters the line segments included in the image to be processed based on the IMU information and field of view Get the line segment to be corrected.
  • the line segment screening policy indicates to perform line segment screening based on the image content of the image to be processed, and the shooting device 300 executes step 833, that is, the shooting device 300 screens the line segments contained in the image to be processed based on the image content of the image to be processed Get the line segment to be corrected.
  • the image content contains line segment angle distribution features.
  • the line segment screening strategy indicates to perform line segment screening based on the field of view and the image content of the image to be processed, and the shooting device 300 executes step 834, that is, the shooting device 300 screens the line segment contained in the image to be processed based on the field of view and the image content of the image to be processed to obtain the line segment to be processed. Straighten the line segment.
  • FIG. 9 is a schematic diagram of a line segment screening method based on image content provided by an embodiment of the present application.
  • the photographing device 300 may filter horizontally inclined line segments and vertically inclined line segments among the M line segments according to the line segment screening model. For example, the image to be processed and M line segment identifications are input into the line segment screening model to obtain a set of horizontally inclined line segments and a set of vertically inclined line segments.
  • the photographing device 300 may detect a vertically inclined line segment and a horizontally inclined line segment among the M line segments according to a vanishing point detection method.
  • Step 910 to step 920 are as follows.
  • Step 910 the photographing device 300 performs preliminary screening according to the angles of the M line segments to obtain a set of preliminary screened vertically inclined line segments.
  • the shooting device 300 judges whether the angle of each line segment in the M line segments is within the vertical line segment inclination threshold range (such as [-75°, 75°]), if the angle of a line segment is within the threshold range of the vertical line segment inclination, it means that the angle of the line segment is close to 90°; if the angle of the line segment is not within the range of the vertical line segment inclination threshold, it means the The angle of the line segment is far away from 90°, and the set of possible preliminary screened vertical inclined line segments is obtained
  • the K line segments are shown in the following formula (7).
  • formula (7) ...
  • Step 920 the photographing device 300 obtains a set of vertically inclined line segments from the initially screened set of vertically inclined line segments according to the vanishing point detection method.
  • the line segments in the set of vertically inclined line segments will intersect at a certain point, and the K equations in the above formula (2) are combined, and the singular value decomposition (SVD) decomposition method is used to solve the equations to obtain the vanishing point of the vertical line segment (x vanish , y vanish ).
  • the vanishing point refers to the intersection of parallel lines. When looking up or photographing two vertical parallel lines, they will intersect at a point far away, that is, the vanishing point, also known as the vanishing point. Then, the shooting device 300 calculates the distance from the vanishing point of the vertical line segment to the K preliminary screening line segments Determine whether the distance is less than the distance threshold d thresh .
  • the distance threshold d thresh is set to a smaller value, such as when the image resolution is 4000*3000, the distance threshold Can be set to 5.
  • the distance threshold also increases proportionally.
  • the difference between the line segment screening scheme based on field of view and image content and the line segment screening scheme based on image content is that the vertical tilt threshold range corresponding to different field of view sizes can be adapted according to the field of view. Because, under the same vertical inclination angle and horizontal inclination angle, the inclination angle ranges of the vertical line segments in the image are different under different fields of view. The larger the field of view, The steeper the slope of the vertical line segments at the edge of the image.
  • the first field of view of the dotted frame content is smaller than the second field of view of the real frame content, and the line segment L1, line segment L2, line segment L3 and line segment L4 are parallel to each other and perpendicular to the ground in the actual scene, but due to Upward shooting causes the four vertical line segments to incline, and at the same shooting upward angle, the inclination angles of L3 and L4 are greater than those of L1 and L2. Therefore, adapt the vertical line segment tilt threshold range corresponding to different field of view sizes according to the field of view to improve the accuracy of the initial screening line segment.
  • the photographing device 300 can linearly adapt the tilt threshold range of the vertical line segment corresponding to the field of view. The larger the field of view, the larger the vertical segment tilt threshold range.
  • Step 840 the photographing device 300 corrects the line segment to be corrected and the feature area of the image to be processed to obtain a processed image.
  • step 840 For a detailed explanation of step 840, reference may be made to the description of step 440 above, and details are not repeated here.
  • the photographing device 300 adapts different line segment screening strategies according to the content contained in the metadata, and improves the flexibility and accuracy of line segment screening for the line segments contained in the image to be processed.
  • the image to be processed described in the embodiment of the present application may be an image edited by a third-party image application, and the IMU information and field of view collected when the image is captured cannot accurately describe the image.
  • the metadata may also include at least one of a trusted identification and an editorial identification. Trusted flags are used to indicate whether metadata is trusted. For example, the value of the trusted flag is 1, indicating that the metadata is untrustworthy; the value of the trusted flag is 0, indicating that the metadata is trustworthy. Trusted identification can be provided by the front-end IMU module.
  • the image to be processed has undergone geometric transformation editing operations such as cropping/rotation/correction by a third-party image application, the IMU information and field of view are considered untrustworthy.
  • the editing flag is used to indicate whether the image to be processed has been processed by the image application.
  • step 831 before the photographing device 300 executes step 831 , it may also execute step 850 , that is, determine whether the metadata is credible. Judging whether the metadata is credible can also be replaced by judging whether the image to be processed has been processed by the image application.
  • step 833 is executed, that is, based on the image content of the image to be processed, the line segment to be corrected is selected from the line segments included in the image to be processed.
  • step 832 that is, obtain the line segment to be corrected based on the IMU information and the field of view from the line segments contained in the image to be processed; or, perform step 834, that is, based on The field of view and the image content of the image to be processed are filtered from the line segments contained in the image to be processed to obtain the line segment to be corrected.
  • the embodiment of the present application provides a user-interactive line recommendation strategy.
  • the photographing device 300 may also execute step 860, which prompts the user to select a line segment to be corrected from recommended line segments filtered based on the image content of the image to be processed.
  • step 860 may include the following detailed steps.
  • Step 861 the photographing device 300 judges whether the horizontal direction is recognized stably.
  • step 862 that is, the interface displays the recommended horizontal line segment. If the user does not choose the recommended horizontal line segment, step 863 is executed to prompt the user to manually draw the horizontal inclined line segment to be corrected.
  • step 864 determine whether the vertical direction is recognized stably, and output a horizontally inclined line segment.
  • step 865 that is, the interface displays the recommended vertical line segment. If the user does not select the recommended vertical line segment, step 866 is executed to prompt the user to manually draw the vertical inclined line segment that needs to be corrected.
  • step 840 is executed, that is, the photographing device 300 corrects the corrected line segment obtained by filtering the line segment based on the image content and corrects the feature area of the image to be processed, and obtains the processed image. If the photographing device 300 determines that neither the horizontal direction nor the vertical direction has been stably recognized, it highlights the recommended horizontally inclined line segment or vertically inclined line segment in the interface, and if the line segment selected by the user is recognized, the line segment selected by the user is corrected.
  • the photographing device 300 may determine whether the recognition is stable according to the quantity and distribution of the vertical line segments and horizontal line segments to be screened based on the image content. For example, when the number of line segments obtained through screening is too small, or the distribution of line segments is chaotic, it may be considered that they have not been stably identified.
  • FIG. 13 it is a schematic diagram of a line segment selection interface provided by the embodiment of the present application.
  • the photographing device 300 determines that the horizontal direction or the vertical direction is not recognized stably, and the interface displays a recommended line segment.
  • the camera device 300 determines that the horizontal or vertical direction is not recognized stably, and the user does not select a recommended line segment, and the interface prompts the user to manually draw an oblique line segment that needs to be corrected.
  • the photographing device 300 screens the image to be processed based on two different line segment screening strategies, it is not necessary to perform the process of determining the line segment screening strategy according to the metadata described in FIG.
  • the process of 830 is that the shooting device 300 screens the image to be processed according to two different line segment screening strategies, and takes the intersection or union of the line segment sets screened by the two methods to determine the final line segment to be corrected. For example, the photographing device 300 determines the line segment to be corrected according to the line segment screened based on the IMU information and the field of view and the line segment screened based on the image content.
  • the shooting device 300 screens line segments based on IMU information and field of view, and adds line segments that need to be corrected based on image content identification to improve the accuracy of axis-shift distortion correction.
  • the line segment set LC1 is obtained by line segment screening based on IMU information and field of view
  • the line segment set LC2 is obtained by line segment screening based on image content
  • the final line segment set LC to be corrected is obtained by solving the intersection or union of LC1 and LC2.
  • the accuracy of line segment screening based on IMU information and field of view is further improved, and the accuracy of axis-shift distortion correction for images is improved.
  • the degree of necessity of correcting the oblique line segment in the image is related to the subject and the user's intention.
  • the subject is a scene with strong regular straight lines, such as architectural structures, horizontal lines, horizons, posters, window frames, door frames, cabinets, etc.
  • the necessity of tilt correction is usually high, while when the subject is plants, animals, food, faces, In the case of non-strong regular straight line scenes such as the sky, the necessity of tilt correction is often low.
  • images that contain many inclined line segments are often related to the user's intention. For example, when the subject is a portrait shot from above, or the oblique angle is too large, it is often the user's composition intention.
  • the photographing device 300 executes step 840, that is, before correcting the line segment to be corrected, the photographing device 300 may also analyze user behavior, and adaptively select whether to correct the filtered vertically inclined line segment and horizontal inclined line segment. For example, as shown in FIG. 14 .
  • step 870 the photographing device 300 judges whether the number of line segments to be corrected is greater than the quantity threshold, and whether the length of the line segments to be corrected is greater than the length threshold.
  • step 840 is performed, that is, the line segment to be corrected is corrected to obtain a processed image.
  • step 880 If the number of line segments to be corrected is less than or equal to the number threshold, or the length of the line segments to be corrected is less than or equal to the length threshold, indicating that the first scene is a non-strongly regular line scene, automatic correction is not performed, and step 880 is performed.
  • step 880 the photographing device 300 determines whether to correct the line segment to be corrected according to the scene characteristics of the first scene.
  • Scene features include architectural features and human features.
  • the shooting device 300 may determine different horizontal line segment inclination thresholds and vertical line segment inclination thresholds according to different scene characteristics.
  • the photographing device 300 judges whether the inclination angle of the horizontal line segment is within the threshold range of the horizontal line segment inclination, and if the inclination angle of the horizontal line segment is within the range of the horizontal line segment inclination threshold, corrects the horizontal line segment.
  • the photographing device 300 may also determine whether the inclination angle of the vertically inclined line segment is within the vertical line segment inclination threshold range, and if the inclination angle of the vertically inclined line segment is within the vertical line segment inclination threshold range, correct the vertical oblique line segment.
  • the images to be processed include portrait images and building images shot from above. If correcting the vertically inclined line segment in the portrait may destroy the "big long legs" effect, you can correct the horizontally inclined line segment, as well as the vertically inclined line segment and the horizontally inclined line segment in the building, and do not correct the vertically inclined line segment in the portrait image Make corrections. When the horizontal inclination angle and the pitch inclination angle are too large, it is usually intentional by the user, and the vertical and horizontal inclination line segments in the image will not be corrected. For example, the photographing device 300 corrects the line segments within a range of 25° offset relative to 90° and 0°.
  • the embodiment of the present application combines image metadata, image content analysis results, and user-interactive straight-line recommendation strategy to screen inclined line segments, jointly optimize various constraints, and obtain vertical and horizontal inclinations caused by correcting horizontal inclination and pitch inclination.
  • FIG. 15 is a schematic diagram of an image processing interface provided by an embodiment of the present application. As shown in (a) of FIG. 15 , it is a schematic front view of the smart phone provided by the embodiment of the present application.
  • a photo album application (application, APP) icon 1501 is displayed on the display screen of the smart phone, and the user may click the photo album application icon 1501 .
  • the touch sensor contained in the smart phone receives the touch operation and reports it to the processor, so that the processor responds to the above touch operation to start the photo album application.
  • other methods may be used to enable the smart phone to start the photo album application, and display the user interface of the photo album application on the display screen.
  • a smart phone when a smart phone has a black screen, displays a lock screen interface, or displays a certain user interface after unlocking, it can respond to the user's voice command or shortcut operation, etc., start the photo album application, and display the user interface of the photo album application on the display screen.
  • the smart phone responds to the click operation, and the display screen of the smart phone displays the user interface of the photo album application, the user interface includes photos taken at different times, and a photo search box.
  • the user can select the tilt-shift distorted photo 1502 taken obliquely, and perform tilt-shift distortion correction on the tilt-shift distorted photo 1502 .
  • the smart phone responds to the click operation, and the display screen of the smart phone displays an axis-shift distorted photo 1502, as well as an editing function button of the photo, such as "send" Function button, "Edit” function button, “Favorite” function button, "Delete” function button and “More” function button.
  • the user can click the "edit” function button 1503, and the display screen of the smart phone displays the function options of the "edit” function.
  • the smart phone responds to the click operation, and the display screen of the smart phone displays the function options of the "edit” function.
  • Feature options for the Edit feature include Smart, Trim, Filter, and Adjust, among others.
  • the user can click the "smart” option button 1504, and the display screen of the smart phone displays multiple functional options of the "smart” option.
  • Smart options include auto-optimization, glare removal, brightening, defogging, and architectural correction, among others.
  • the smart phone responds to the click operation, and the display screen of the smart phone can also display the "axis-shift distortion” function option, the "horizontal correction” function option and the "vertical correction” option included in the "smart” option. "Function options, etc.
  • the user can click on the function option 1505 of “Shift Distortion”.
  • the smartphone corrects the tilt-shift distortion on the tilt-shift distortion photo 1502 .
  • the smart phone responds to the click operation, and the display screen displays a photo 1506 after the tilt-shift distortion correction is performed on the tilt-shift distorted photo 1502 .
  • the specific method of correcting the tilt-shift distortion of the tilt-shift distortion photo 1502 by the smart phone reference may be made to the description of the foregoing embodiments, and details are not repeated here.
  • FIG. 16 is a schematic diagram of another image processing interface provided by the embodiment of the present application. As shown in (a) of FIG. 16 , it is a schematic front view of the smart phone provided by the embodiment of the present application.
  • a camera application (application, APP) icon 1601 is displayed on the display screen of the smartphone, and the user may click the camera application icon 1601 .
  • the touch sensor included in the smart phone receives the touch operation and reports it to the processor, so that the processor starts the camera application in response to the touch operation.
  • other methods may be used to enable the smart phone to start the camera application, and display the user interface of the camera application on the display screen.
  • a smart phone has a black screen, a lock screen interface is displayed, or a certain user interface is displayed after unlocking
  • the camera application can be launched in response to the user's voice command or shortcut operation, and the user interface of the camera application can be displayed on the display screen.
  • the smart phone responds to the click operation, and the display screen of the smart phone displays the user interface of the camera application.
  • the user interface includes "small video”, “video recording”, “photographing”, “portrait”, Mode options such as “panorama”, “preview image” function button, "photograph” function button and “front and rear camera conversion” function button.
  • the user can first select the “photographing” mode 1602, and then click the “photographing” function button 1603, and the smart phone automatically turns on the camera in response to the click operation, and the camera acquires an upward shot image of the scene.
  • the user can click on the “preview image” function button 1604 to view the captured image containing the target subject, that is, the tilt-shift distorted photo 1605 .
  • the smart phone responds to the click operation, and the display screen of the smart phone displays an axis-shifting distortion photo 1605, as well as editing function buttons of the photo, such as a "send” function button, an "edit” function button, “Favorite” function button, "Delete” function button and “More” function button.
  • the user may click the "edit” function button 1606, and the display screen of the smartphone displays the function options of the "edit” function.
  • the smart phone responds to the click operation, and the display screen of the smart phone displays the function options of the "edit” function.
  • the functional options of the "editing" function include smart, trimming, filter and adjustment, etc.
  • the user can click the "smart” option button 1607, and the display screen of the smart phone displays multiple functional options of the "smart” option.
  • Smart options include auto-optimization, glare removal, brightening, defogging, and architectural correction, among others.
  • the smart phone responds to the click operation, and the display screen of the smart phone can also display the "axis-shift distortion” function option, the "horizontal correction” function option and the "vertical correction” option included in the "smart” option. "Function options, etc.
  • the user may click on the functional option 1608 of “Shift Distortion”.
  • the smartphone corrects the tilt-shift distortion on the tilt-shift distortion photo 1605 .
  • the smartphone responds to the click operation, and the display screen displays
  • the transformed photo 1605 is a photo 1609 after the axis-shift distortion correction is performed.
  • the specific method of correcting the tilt-shift distortion of the tilt-shift distortion photo 1605 by the smart phone reference may be made to the description of the foregoing embodiments, and details are not repeated here.
  • the shooting device may automatically correct the axis-shift distortion of the image after taking the picture.
  • the smart phone responds to the click operation, the display screen of the smart phone displays the user interface of the camera application, and the smart phone automatically turns on the camera in response to clicking the "photographing" function button 1603, and the camera Obtain the upward shot image of the scene, and correct the vertical shift distortion on the vertical shift distortion photo 1605 to obtain the shaft distortion corrected photo 1609, as shown in (g) in Figure 16, the display screen of the smart phone shows that the shift The axis distortion photo 1605 is a photo 1609 after the axis-shift distortion correction is performed. (c) in Figure 16 is omitted
  • each function option displayed on the display screen of the shooting device may also be referred to as a control or a function control.
  • the photographing device includes hardware structures and/or software modules corresponding to each function.
  • the present application can be implemented in the form of hardware or a combination of hardware and computer software with reference to the units and method steps of the examples described in the embodiments disclosed in the present application. Whether a certain function is executed by hardware or computer software drives the hardware depends on the specific application scenario and design constraints of the technical solution.
  • FIG. 17 is a schematic structural diagram of a possible image processing device provided by this embodiment.
  • These image processing apparatuses can be used to implement the functions of the photographing device in the above method embodiments, and thus can also achieve the beneficial effects of the above method embodiments.
  • the image processing apparatus may be a photographing device 300 as shown in FIG. 3 , or may be a module (such as a chip) applied to a server.
  • the image processing device 1700 includes a communication module 1710 , a line segment filtering module 1720 , a correction module 1730 and a storage module 1740 .
  • the image processing apparatus 1700 is used to implement the functions of the photographing device 300 in the method embodiments shown in FIGS. 4 , 6 , 8 , 11 , 12 and 14 above.
  • the communication module 1710 is configured to acquire an image to be processed, the image to be processed includes at least one of a vertical line segment and a horizontal line segment in a first scene, and the first scene is a scene where the image to be processed is acquired by a shooting device.
  • the communication module 1710 is used to execute step 410 .
  • the communication module 1710 is used to execute step 810 .
  • the line segment screening module 1720 is configured to obtain the line segment to be corrected from the line segments included in the image to be processed according to the metadata of the image to be processed, the metadata including the inertial measurement unit IMU information and field of view of the shooting device at least one of the
  • the line segment filtering module 1720 is used to execute step 420 and step 430 .
  • the line segment screening module 1720 is used to execute step 820 , step 830 , step 850 , step 860 , step 870 and step 880 .
  • the correction module 1730 is configured to correct the line segment to be corrected and the feature area of the image to be processed to obtain a processed image, and the feature area includes a background area, a portrait area and an edge area.
  • the rectification module 1730 is configured to perform step 440 .
  • the rectification module 1730 is configured to execute step 840 .
  • the image processing apparatus 1700 may upload the processed image to a server or an edge device or a cloud device or store it in a local storage.
  • the storage module 1740 is used for storing metadata, and the metadata indicates attribute information of the image to be processed, such as IMU information and field of view, so as to facilitate a line segment screening strategy determined according to the metadata of the image to be processed.
  • the image processing apparatus 1700 may further include a display module 1750, and the display module 1750 is configured to display the image to be processed and the processed image.
  • the image processing device 1700 in the embodiment of the present application may be realized by an application-specific integrated circuit (ASIC), or a programmable logic device (PLD), and the above-mentioned PLD may be a complex program Logic device (complex programmable logical device, CPLD), field-programmable gate array (field-programmable gate array, FPGA), general array logic (generic array logic, GAL) or any combination thereof.
  • ASIC application-specific integrated circuit
  • PLD programmable logic device
  • CPLD complex programmable logical device
  • FPGA field-programmable gate array
  • GAL general array logic
  • the image processing device 1700 and its modules can also be software modules
  • the image processing device 1700 and its modules can also be software modules.
  • the individual modules may also be software modules.
  • the image processing device 1700 may correspond to the implementation of the method described in the embodiment of the present application, and the above-mentioned and other operations and/or functions of the various units in the image processing device 1700 are respectively in order to realize Fig. 4, Fig. 6,
  • Fig. 4 Fig. 6
  • FIG. 8 , FIG. 11 , FIG. 12 and FIG. 14 will not be repeated here.
  • FIG. 18 is a schematic structural diagram of a shooting device 1800 provided in this embodiment.
  • the photographing device 1800 includes a processor 1810, a bus 1820, a memory 1830, a communication interface 1840, a memory unit 1850 (also called a main memory unit) and a camera 1860.
  • the photographing device 1800 may further include a display 1870 for displaying images to be processed and processed images.
  • the processor 1810 , memory 1830 , memory unit 1850 , communication interface 1840 , camera 1860 and display 1870 are connected through a bus 1820 .
  • the processor 1810 may be a CPU, and the processor 1810 may also be other general-purpose processors, digital signal processing (digital signal processing, DSP), ASIC, FPGA or other programmable logic devices, Discrete gate or transistor logic devices, discrete hardware components, etc.
  • a general purpose processor may be a microprocessor or any conventional processor or the like.
  • the processor can also be a graphics processing unit (graphics processing unit, GPU), a neural network processor (neural network processing unit, NPU), a microprocessor, an ASIC, or one or more integrations used to control the execution of the program program of this application circuit.
  • graphics processing unit graphics processing unit, GPU
  • neural network processor neural network processing unit, NPU
  • microprocessor microprocessor
  • ASIC application specific integrated circuit
  • the communication interface 1840 is used to realize communication between the photographing device 1800 and external devices or devices. In this embodiment, when the photographing device 1800 is used to implement the functions of the photographing device 300 shown in FIG. 4 , FIG. 6 , FIG. 8 , FIG. 11 , FIG. 12 and FIG. 14 , the communication interface 1840 is used to receive images to be processed.
  • Bus 1820 may include a path for communicating information between the components described above (eg, processor 1810, memory unit 1850, and storage 1830).
  • the bus 1820 may also include a power bus, a control bus, a status signal bus, and the like.
  • the various buses are labeled as bus 1820 in the figure.
  • the bus 1820 can be a peripheral component interconnection standard (Peripheral Component Interconnect Express, PCIe) bus, or an extended industry standard architecture (extended industry standard architecture, EISA) bus, unified bus (unified bus, Ubus or UB), computer fast link ( compute express link (CXL), cache coherent interconnect for accelerators (CCIX), etc.
  • PCIe peripheral component interconnection standard
  • EISA extended industry standard architecture
  • unified bus unified bus, Ubus or UB
  • CXL compute express link
  • CIX cache coherent interconnect for accelerators
  • the bus 1820 can be divided into address bus, data bus, control bus and so on.
  • the photographing device 1800 may include multiple processors.
  • the processor may be a multi-CPU processor.
  • a processor herein may refer to one or more devices, circuits, and/or computing units for processing data (eg, computer program instructions).
  • the shooting device 1800 when the shooting device 1800 is used to realize the functions of the shooting device 300 shown in FIG. 4 , FIG. 6 , FIG. 8 , FIG. 11 , FIG. 12 and FIG.
  • the line segment to be corrected is obtained from the line segments contained in the image to be processed, and the line segment to be corrected and the feature area of the image to be processed include the background area, portrait area and edge The region is corrected to obtain the processed image.
  • the photographing device 1800 includes one processor 1810 and one memory 1830 as an example.
  • the processor 1810 and the memory 1830 are respectively used to indicate a type of device or device.
  • the quantity of each type of device or equipment can be determined according to business needs.
  • the memory unit 1850 may correspond to a storage medium used to store information such as metadata in the foregoing method embodiments.
  • the memory unit 1850 can be volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory.
  • the non-volatile memory can be read-only memory (read-only memory, ROM), programmable read-only memory (programmable ROM, PROM), erasable programmable read-only memory (erasable PROM, EPROM), electrically programmable Erases programmable read-only memory (electrically EPROM, EEPROM) or flash memory.
  • Volatile memory can be random access memory (RAM), which acts as external cache memory.
  • RAM random access memory
  • SRAM static random access memory
  • DRAM dynamic random access memory
  • SDRAM synchronous dynamic random access memory
  • Double data rate synchronous dynamic random access memory double data date SDRAM, DDR SDRAM
  • enhanced SDRAM enhanced synchronous dynamic random access memory
  • SLDRAM synchronous connection dynamic random access memory
  • direct rambus RAM direct rambus RAM
  • the memory 1830 may correspond to a storage medium for storing information such as computer instructions in the above method embodiments, for example, a magnetic disk, such as a mechanical hard disk or a solid state hard disk.
  • the above-mentioned photographing device 1800 may be a general device or a special device.
  • the photographing device 1800 may be an edge device (eg, a box carrying a chip with processing capabilities), or the like.
  • the photographing device 1800 may also be a server or other devices with computing capabilities.
  • the photographing device 1800 may correspond to the image processing device 1700 in this embodiment, and may correspond to any of The corresponding subject in the method, and the above-mentioned and other operations and/or functions of the various modules in the image processing device 1700 are respectively in order to realize the corresponding For the sake of brevity, the process will not be repeated here.
  • the method steps in this embodiment may be implemented by means of hardware, and may also be implemented by means of a processor executing software instructions.
  • the software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (random access memory, RAM), flash memory, read-only memory (read-only memory, ROM), programmable read-only memory (programmable ROM) , PROM), erasable programmable read-only memory (erasable PROM, EPROM), electrically erasable programmable read-only memory (electrically EPROM, EEPROM), register, hard disk, mobile hard disk, CD-ROM or known in the art any other form of storage medium.
  • An exemplary storage medium is coupled to the processor such that the processor can read from the storage medium Information can be read and information can be written to the storage medium.
  • the storage medium may also be a component of the processor.
  • the processor and storage medium can be located in the ASIC.
  • the ASIC may be located in the camera device.
  • the processor and the storage medium can also exist in the shooting device as discrete components.
  • all or part of them may be implemented by software, hardware, firmware or any combination thereof.
  • software When implemented using software, it may be implemented in whole or in part in the form of a computer program product.
  • the computer program product comprises one or more computer programs or instructions. When the computer program or instructions are loaded and executed on the computer, the processes or functions described in the embodiments of the present application are executed in whole or in part.
  • the computer may be a general purpose computer, a special purpose computer, a computer network, network equipment, user equipment, or other programmable devices.
  • the computer program or instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instructions may be downloaded from a website, computer, A server or data center transmits to another website site, computer, server or data center by wired or wireless means.
  • the computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device such as a server or a data center integrating one or more available media.
  • the available medium may be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; it may also be an optical medium, such as a digital video disc (digital video disc, DVD); it may also be a semiconductor medium, such as a solid state drive (solid state drive). , SSD).

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Quality & Reliability (AREA)
  • Human Computer Interaction (AREA)
  • Studio Devices (AREA)
  • Image Processing (AREA)

Abstract

公开了图像处理方法、装置及设备，涉及图像处理领域。获取到待处理图像后，根据元数据包含的IMU信息和视场中至少一个从待处理图像包含的线段中筛选得到待矫正线段。待处理图像包含拍摄设备拍摄的第一场景中垂直线段和水平线段中至少一种。对待矫正线段及待处理图像的特征区域包括背景区域、人像区域和边缘区域进行矫正，得到处理后图像。由于IMU信息和视场描述了拍摄待处理图像的拍摄设备的属性信息，根据IMU信息和视场准确定位需要矫正的线段，对待矫正线段以及待处理图像的特征区域进行矫正，使处理后图像中的线段更符合拍摄设备所拍摄的第一场景中水平线段和垂直线段的分布特征。有效地提高了对图像的移轴畸变现象进行矫正的精确度。

Description

图像处理方法、装置及设备
本申请要求于2022年3月2日提交国家知识产权局、申请号为202210204484.6、申请名称为“图像处理方法、装置及设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及图像处理领域，尤其涉及一种图像处理方法、装置及设备。
背景技术
通常，用户使用相机拍摄过程中，由于相机倾斜等非固定性原因，导致所拍摄场景中的直线在图像中发生移轴畸变(如：倾斜或汇聚)，影响图像的构图美观。例如仰拍建筑物图像中越向建筑物的顶端，外墙边线越汇聚。
目前，基于图像内容对发生移轴畸变的图像进行全局矫正。但是，对于具有特殊特征的非正方内容采用该矫正方法反而可能产生矫正错误，因此该矫正方法的精确度较低。
发明内容
本申请提供了图像处理方法、装置及设备，由此解决了基于图像内容对发生移轴畸变的图像进行全局矫正而产生的矫正错误的问题，提高了对图像进行移轴畸变矫正的精确度。
第一方面，提供了一种图像处理方法，该方法由计算设备执行。计算设备获取到待处理图像后，根据待处理图像的元数据包含的惯性测量组件(Inertial Measurement Unit，IMU)信息和视场中至少一个从待处理图像包含的线段中筛选得到待矫正线段。待处理图像包含拍摄设备获取该待处理图像的第一场景中垂直线段和水平线段中至少一种。可理解的，待处理图像包含的线段可以是第一场景中线段在图像中未发生倾斜的线段，也可以是第一场景中线段在图像中发生倾斜的线段。进而，对待矫正线段以及待处理图像的特征区域包括背景区域、人像区域和边缘区域进行矫正，得到处理后图像。
如此，由于IMU信息和视场描述了拍摄待处理图像的拍摄设备的属性信息，根据IMU信息和视场可以准确地定位需要矫正的线段，从待处理图像包含的线段中筛选待矫正线段，对待矫正线段以及待处理图像的特征区域进行矫正，使处理后图像中的线段更符合拍摄设备所拍摄的第一场景中水平线段和垂直线段的分布特征。相对于对待处理图像进行整体矫正，针对待处理图像中线段进行精准矫正，从而有效地提高了对图像进行移轴畸变矫正的精确度。
例如，IMU信息包含了拍摄待处理图像的拍摄设备的水平倾角和俯仰倾角。视场表示摄像头能够观察到的最大范围。通常以角度来表示视场。视场越大，观测范围越大。
如果拍摄设备仰拍第一场景或俯拍第一场景，第一场景中垂直线段在图像中发生 倾斜，则待处理图像包含第一场景中垂直线段在待处理图像中发生倾斜的线段。
如果拍摄设备水平倾斜拍摄第一场景，第一场景中垂直线段和水平线段在图像中发生倾斜，则待处理图像包含第一场景中垂直线段和水平线段在待处理图像中发生倾斜的线段。
需要说明的是，本申请实施例中进行图像处理的计算设备可以是携带拍摄功能的拍摄设备(例如，智能手机)也可是其他具备图像处理功能的设备(如：服务器、云端设备、边缘设备等)。
待矫正线段包括第一场景中垂直线段和水平线段中至少一种。可理解的是，待矫正线段包括第一场景中垂直线段和水平线段中至少一种在待处理图像中发生倾斜的线段中部分线段或全部线段。例如，待处理图像包含发生移轴畸变的建筑物图像和人像，待矫正线段包含建筑物中垂直线段和水平线段中至少一种在图像中发生倾斜的线段，不包括人像中发生倾斜的线段，仅对建筑物图像中发生倾斜的线段进行矫正，以此保留人像中大长腿的效果，使图像构图更美观。
在一种可能的实现方式中，拍摄设备显示移轴畸变控件，方法还包括：拍摄设备接收用户对移轴畸变控件的操作后，根据待处理图像的元数据从待处理图像包含的线段中筛选得到待矫正线段，对待矫正线段以及待处理图像的特征区域进行矫正，得到处理后图像。基于人机交互方式对图像进行移轴畸变矫正，降低用户对图像进行移轴畸变的操作步骤的复杂度，提升对图像进行移轴畸变矫正的灵活性，从而提高用户对图像进行处理的用户体验。
在另一种可能的实现方式中，方法还包括：在对待矫正线段以及待处理图像的特征区域进行矫正得到处理后图像后，显示处理后图像。从而，使用户更直观地看到进行移轴畸变矫正后的图像，提高用户对图像进行处理的用户体验。
在另一种可能的实现方式中，根据元数据包含内容制定线段筛选策略来筛选待矫正线段。线段筛选策略用于指示对待处理图像包含的线段进行线段筛选的方法。例如，根据待处理图像的元数据从待处理图像包含的线段中筛选得到待矫正线段，包括：根据元数据确定的线段筛选策略，从待处理图像包含的线段中筛选得到待矫正线段。从而，根据元数据包含的内容适配不同的线段筛选策略，提升对待处理图像包含的线段进行线段筛选的灵活性和精确度。
示例一，元数据包括IMU信息和视场；线段筛选策略用于指示基于IMU信息和视场进行线段筛选；根据元数据确定的线段筛选策略，从待处理图像包含的线段中筛选得到待矫正线段，包括：根据IMU信息和视场对待处理图像包含的M条线段进行整体调整得到调整后M条线段，调整后M条线段符合第一场景的线段角度分布特征，根据线段倾斜阈值从调整后M条线段中筛选得到N条待矫正线段。线段倾斜阈值范围包括水平线段倾斜阈值和垂直线段倾斜阈值，M和N均为正整数，N小于或等于M。如此，由于IMU信息用于指示拍摄设备的倾斜角度，拍摄设备基于IMU信息和视场对待处理图像中的倾斜线段进行筛选，有助于提高线段筛选的精确度，进而提高对图像进行移轴畸变矫正的精确度。
示例二，元数据不包括视场，根据元数据确定的线段筛选策略，从待处理图像包含的线段中筛选得到待矫正线段，包括：基于待处理图像的图像内容从待处理图像包 含的线段中筛选得到待矫正线段，图像内容包含线段角度分布特征。如此，拍摄设备基于图像内容中线段的倾斜角度进行线段精准筛选，进而提高对图像进行移轴畸变矫正的精确度。
示例三，元数据包括视场，且不包括IMU信息，根据元数据确定的线段筛选策略，从待处理图像包含的线段中筛选得到待矫正线段，包括：基于视场和待处理图像的图像内容从待处理图像包含的线段中筛选得到待矫正线段，图像内容包含线段角度分布特征。由于不同视场范围内线段倾斜的角度不同，视场范围越大，图像中边缘线段倾斜角度越大。拍摄设备基于图像内容中线段的倾斜角度和视场进行线段精准筛选，进而提高对图像进行移轴畸变矫正的精确度。
在另一种可能的实现方式中，方法还包括：提示用户从基于待处理图像的图像内容筛选得到的推荐线段中选择待矫正线段。有时用户为了构图效果，可能采用仰拍或俯拍措施拍摄图像。此时，图像中的倾斜线段用户可能并不想矫正。如此，拍摄设备提供了用户交互的直线推荐策略，提示用户是否进行线段矫正，根据用户意愿对图像进行移轴畸变矫正，来提高对图像进行移轴畸变矫正的精确度。
在另一种可能的实现方式中，元数据还包括可信标识，可信标识用于指示元数据是否可信；根据元数据确定的线段筛选策略，从待处理图像包含的线段中筛选得到待矫正线段，包括：若可信标识指示元数据不可信，基于待处理图像的图像内容从待处理图像包含的线段中筛选得到待矫正线段，图像内容包含线段角度分布特征。若可信标识指示元数据可信，基于待处理图像的图像内容、IMU信息和视场中至少一个从待处理图像包含的线段中筛选得到待矫正线段。如此，在根据IMU信息和视场进行线段筛选之前，先预判IMU信息和视场是否可信，根据IMU信息和视场的可信程度，确定进行线段筛选的线段筛选策略，从而提升线段筛选的速率，以及提升线段筛选结果的精确度，进而提升对图像进行移轴畸变矫正的精确度。
在另一种可能的实现方式中，根据待处理图像的元数据从待处理图像包含的线段中筛选得到待矫正线段，包括：根据基于元数据筛选的线段和基于图像内容筛选的线段确定待矫正线段。如此，结合两种不同线段筛选策略的结果，确定待矫正线段，进一步地提高基于IMU信息和视场进行线段筛选的精确度，进而提升对图像进行移轴畸变矫正的精确度。
在另一种可能的实现方式中，对待矫正线段以及待处理图像的特征区域进行矫正，得到处理后图像，包括：判断待矫正线段的数量是否大于数量阈值，且待矫正线段的长度是否大于长度阈值；若待矫正线段的数量大于数量阈值，且待矫正线段的长度大于长度阈值，对待矫正线段进行矫正，得到处理后图像；若待矫正线段的数量小于或等于数量阈值，或待矫正线段的长度小于或等于长度阈值，根据第一场景的场景特征确定是否对待矫正线段进行矫正，场景特征包含建筑特征和人物特征。如此，对图像中线段进行矫正时，充分参考场景特征和用户意图，使得矫正后图像更符合实际场景中线段的分布特征，有效地提高了对图像进行移轴畸变矫正的精确度。
在另一种可能的实现方式中，对待矫正线段以及待处理图像的特征区域进行矫正，得到处理后图像，包括：构建直线约束对待矫正线段进行矫正；构建基于内容的单应性约束和形状和正则约束对背景区域进行矫正；构建人像约束对人像区域进行矫正； 构建边界约束对边缘区域进行矫正，以得到处理后图像。如此，联合优化直线约束、基于内容的单应性约束、形状和正则约束、人像约束和边界约束等各项约束，使得矫正结果更为合理美观，视场保持更多，构图更美观。
若根据待处理图像的元数据对待处理图像包含的线段进行筛选，未筛选到需要矫正的线段，根据待处理图像的特征区域对待处理图像进行矫正，可以有效地弥补线段筛选的误差，进而提高对图像进行移轴畸变矫正的精确度。
在另一种可能的实现方式中，根据待处理图像的元数据从待处理图像包含的线段中筛选得到待矫正线段，包括：根据直线检测方法检测待处理图像，得到线段集。线段集包含M个倾斜线段，M为正整数。
在另一种可能的实现方式中，对待矫正线段进行矫正，得到处理后图像之后，方法包括：更新待处理图像的元数据包括IMU信息。例如，删除或修改IMU信息。从而，提高使用IMU信息进行线段筛选的准确性。
第二方面，提供了一种图像处理装置，所述装置包括用于执行第一方面或第一方面任一种可能设计中的图像处理方法的各个模块。
第三方面，提供一种拍摄设备，该拍摄设备包括至少一个处理器和存储器，存储器用于存储一组计算机指令；当处理器作为第一方面或第一方面任一种可能实现方式中的拍摄设备执行所述一组计算机指令时，执行第一方面或第一方面任一种可能实现方式中的图像处理方法的操作步骤。
第四方面，提供一种计算设备，该计算设备包括至少一个处理器和存储器，存储器用于存储一组计算机指令；当处理器作为第一方面或第一方面任一种可能实现方式中的拍摄设备执行所述一组计算机指令时，执行第一方面或第一方面任一种可能实现方式中的图像处理方法的操作步骤。
第五方面，提供一种计算机可读存储介质，包括：计算机软件指令；当计算机软件指令在计算设备中运行时，使得计算设备执行如第一方面或第一方面任意一种可能的实现方式中所述方法的操作步骤。
第六方面，提供一种计算机程序产品，当计算机程序产品在计算机上运行时，使得计算设备执行如第一方面或第一方面任意一种可能的实现方式中所述方法的操作步骤。
本申请在上述各方面提供的实现方式的基础上，还可以进行进一步组合以提供更多实现方式。
附图说明
图1为本申请实施例提供的一种移轴畸变图像和矫正后图像的示意图；
图2为本申请实施例提供的一种移轴畸变矫正场景示意图；
图3为本申请实施例提供的一种拍摄设备的结构示意图；
图4为本申请实施例提供的一种图像处理方法的示意图；
图5为本申请实施例提供的一种元数据的结构示意图；
图6为本申请实施例提供的一种基于IMU信息和视场进行线段筛选的示意图；
图7为本申请实施例提供的一种线段筛选可视化的示意图；
图8为本申请实施例提供的另一种图像处理方法的示意图；
图9为本申请实施例提供的一种基于图像内容进行线段筛选的示意图；
图10为本申请实施例提供的一种基于视场和图像内容进行线段筛选的示意图；
图11为本申请实施例提供的另一种图像处理方法的示意图；
图12为本申请实施例提供的又一种图像处理方法的示意图；
图13为本申请实施例提供的一种线段选择界面示意图；
图14为本申请实施例提供的又一种图像处理方法的示意图；
图15为本申请实施例提供的一种图像处理的界面示意图；
图16为本申请实施例提供的另一种图像处理的界面示意图；
图17为本申请实施例提供的一种图像处理装置的结构示意图；
图18为本申请实施例提供的一种拍摄设备的结构示意图。
具体实施方式
本申请实施例提供一种图像处理方法，尤其是提供一种对图像进行移轴畸变矫正的方法，即根据待处理图像的元数据从待处理图像包含的线段中筛选得到待矫正线段，该待矫正线段可以是在实际场景中垂直线段和水平线段中至少一种在待处理图像中发生倾斜的线段，对待矫线段以及待处理图像的特征区域进行矫正，使处理后图像中的线段更符合拍摄设备所拍摄的实际场景中水平线段和垂直线段的分布特征。相对于对待处理图像进行整体矫正，针对待处理图像中倾斜线段进行精准矫正，从而有效地提高了对图像进行移轴畸变矫正的精确度。
移轴畸变是指拍摄设备的成像平面相对所拍摄场景的目标平面在x轴、y轴和z轴中至少一个上存在旋转，使得目标平面中原本的平行线在成像平面产生倾斜或者歪斜的现象。
示例地，如图1中的(a)所示，相机沿z轴旋转拍摄得到的图像，即水平倾斜拍摄得到的图像，相对所拍摄场景中的水平线段和垂直线段，图像包含了水平倾斜线段和垂直倾斜线段。如图1中的(b)所示，为移轴畸变矫正后图像，矫正了水平倾斜线段和垂直倾斜线段，使移轴畸变矫正后图像更符合拍摄设备所拍摄的实际场景中水平线段和垂直线段的分布特征。如图1中的(c)所示，相机沿x轴旋转拍摄得到的图像，即相机仰拍得到的图像，相对所拍摄场景中的垂直线段，图像包含了垂直倾斜线段。如图1中的(d)所示，为移轴畸变矫正后图像，矫正了垂直倾斜线段，使移轴畸变矫正后图像更符合拍摄设备所拍摄的实际场景中水平线段和垂直线段的分布特征。
在本申请实施例中，执行图像移轴畸变矫正的设备具备较强的图像处理计算能力，该设备可以是一个拍摄设备、服务器、云端设备或边缘设备(例如，携带具有处理能力芯片的盒子)等。
例如，如图2中的(a)所示，拍摄设备从本地或云端获取到待处理图像，利用拍摄设备的处理器(如：图形处理器(graphics processing unit，GPU))对待处理图像进行移轴畸变矫正。
又如，如图2中的(b)所示，拍摄设备上传待处理图像，由服务器、云端设备或边缘设备对待处理图像进行移轴畸变矫正。
拍摄设备可以是一个集成化模组。拍摄设备包括相机、惯性测量组件(Inertial Measurement Unit，IMU)、通信模块和处理器。例如，拍摄设备可以是终端设备，如 手机终端，平板电脑，笔记本电脑，可穿戴设备，虚拟现实(virtual reality，VR)设备、增强现实(augmented reality，AR)设备、混合现实(Mixed Reality，MR)设备、扩展现实(Extended Reality，ER)设备和摄像头等。
下面以拍摄设备为智能终端为例进行说明。示例地，图3为本申请实施例提供的一种拍摄设备的结构示意图。拍摄设备300包括处理器310，外部存储器接口320，内部存储器321，通用串行总线(universal serial bus，USB)接口330，电源管理模块340，天线，无线通信模块360，音频模块370，扬声器370A，音箱接口370B，麦克风370C，传感器模块380，按键390，指示器391，显示屏392，以及摄像头393等。其中，上述传感器模块380可以包括压力传感器、陀螺仪传感器、气压传感器、磁传感器、加速度传感器、图像传感器、距离传感器、接近光传感器、指纹传感器、温度传感器、触摸传感器和环境光传感器等传感器。图像传感器用于利用光电器件的光电转换功能将感光面上的光像转换为与光像成相应比例关系的电信号。
可以理解的是，本实施例示意的结构并不构成对拍摄设备的具体限定。在另一些实施例中，拍摄设备可以包括比图示更多或更少的部件，或者组合某些部件，或者拆分某些部件，或者不同的部件布置。图示的部件可以以硬件，软件或软件和硬件的组合实现。
处理器310可以包括一个或多个处理单元，例如：处理器310可以包括应用处理器(application processor，AP)，调制解调处理器，GPU，图像信号处理器(image signal processor，ISP)，控制器，存储器，视频编解码器，数字信号处理器(digital signal processor，DSP)，基带处理器，和/或神经网络处理器(neural-network processing unit，NPU)等。其中，不同的处理单元可以是独立的器件，也可以集成在一个或多个处理器中。NPU为神经网络(neural-network，NN)计算处理器，通过借鉴生物神经网络结构，例如借鉴人脑神经元之间传递模式，对输入信息快速处理，还可以不断的自学习。通过NPU可以实现拍摄设备的智能认知等应用，例如：图像识别，人脸识别，语音识别，文本理解等。
在本实施例中，处理器310用于根据待处理图像的元数据确定的线段筛选策略，从待处理图像包含的线段中筛选得到待矫正线段。线段筛选策略用于指示对待处理图像包含的线段进行线段筛选的方法。待矫正线段包括待处理图像中具有倾斜特征的线段中至少一条。元数据(metadata)又称中介数据或中继数据，为描述数据的数据(data about data)，主要是描述数据属性(property)的信息，用来支持如指示存储位置、历史数据、资源查找、文件记录等功能。在本申请实施例中，元数据是嵌入到待处理图像文件中的一些标签，用于描述图像数据属性的信息，比如镜头参数、曝光参数、传感器信息等。通常，相机在拍摄照片时图像文件中自动添加元数据。例如，元数据包括惯性测量组件(Inertial Measurement Unit，IMU)信息和视场。IMU信息包括但不仅限于三轴姿态角和加速度。视场表示摄像头能够观察到的最大范围，通常以角度来表示视场。视场越大，观测范围越大。
在本申请实施例中，惯性测量组件包括陀螺仪传感器和加速度传感器。
陀螺仪传感器可以用于确定拍摄设备300的运动姿态。在一些实施例中，可以通过陀螺仪传感器确定拍摄设备300围绕三个轴(即，x，y和z轴)的角速度。陀螺仪传 感器可以用于拍摄防抖。示例性的，当按下快门，陀螺仪传感器检测拍摄设备300抖动的角度，根据角度计算出镜头模组需要补偿的距离，让镜头通过反向运动抵消拍摄设备300的抖动，实现防抖。陀螺仪传感器还可以用于导航，体感游戏场景。
加速度传感器可检测拍摄设备300在各个方向上(一般为三轴)加速度的大小。当拍摄设备300静止时可检测出重力的大小及方向。还可以用于识别拍摄设备姿态，应用于横竖屏切换，计步器等应用。
控制器可以是拍摄设备的神经中枢和指挥中心。控制器可以根据指令操作码和时序信号，产生操作控制信号，完成取指令和执行指令的控制。
处理器310中还可以设置存储器，用于存储指令和数据。在一些实施例中，处理器310中的存储器为高速缓冲存储器。该存储器可以保存处理器310刚用过或循环使用的指令或数据。如果处理器310需要再次使用该指令或数据，可从所述存储器中直接调用。避免了重复存取，减少了处理器310的等待时间，因而提高了系统的效率。
在一些实施例中，处理器310可以包括一个或多个接口。接口可以包括集成电路(inter-integrated circuit，I2C)接口，集成电路内置音频(inter-integrated circuit sound，I2S)接口，脉冲编码调制(pulse code modulation，PCM)接口，通用异步收发传输器(universal asynchronous receiver/transmitter，UART)接口，移动产业处理器接口(mobile industry processor interface，MIPI)，通用输入输出(general-purpose input/output，GPIO)接口，和/或USB接口等。
电源管理模块340用于连接电源。电源管理模块340还可以与处理器310、内部存储器321、显示屏392、摄像头393和无线通信模块360等连接。电源管理模块340接收电源的输入，为处理器310、内部存储器323、显示屏392、摄像头393和无线通信模块360等供电。在一些实施例中，电源管理模块340也可以设置于处理器310中。
拍摄设备的无线通信功能可以通过天线和无线通信模块360等实现。其中，无线通信模块360可以提供应用在拍摄设备上的包括无线局域网(wireless local area networks，WLAN)(如无线保真(wireless fidelity，Wi-Fi)网络)，蓝牙(bluetooth，BT)，全球导航卫星系统(global navigation satellite system，GNSS)，调频(frequency modulation，FM)，近距离无线通信技术(near field communication，NFC)，红外技术(infrared，IR)等无线通信的解决方案。
无线通信模块360可以是集成至少一个通信处理模块的一个或多个器件。无线通信模块360经由天线接收电磁波，将电磁波信号调频以及滤波处理，将处理后的信号发送到处理器310。无线通信模块360还可以从处理器310接收待发送的信号，对其进行调频，放大，经天线转为电磁波辐射出去。在一些实施例中，拍摄设备的天线和无线通信模块360耦合，使得拍摄设备可以通过无线通信技术与网络以及其他设备通信。
拍摄设备通过GPU，显示屏392，以及应用处理器等实现显示功能。GPU为图像处理的微处理器，连接显示屏392和应用处理器。GPU用于执行数学和几何计算，用于图形渲染。处理器310可包括一个或多个GPU，其执行程序指令以生成或改变显示信息。
显示屏392用于显示图像，视频等。如本申请实施例中，显示屏392用于显示待 处理图像和处理后图像。该显示屏392包括显示面板。显示面板可以采用液晶显示屏(liquid crystal display，LCD)，有机发光二极管(organic light-emitting diode，OLED)，有源矩阵有机发光二极体或主动矩阵有机发光二极体(active-matrix organic light emitting diode，AMOLED)，柔性发光二极管(flex light-emitting diode，FLED)，Miniled，MicroLed，Micro-oLed，量子点发光二极管(quantum dot light emitting diodes，QLED)等。
拍摄设备可以通过ISP，摄像头393，视频编解码器，GPU，显示屏392以及应用处理器等实现拍摄功能。ISP用于处理摄像头393反馈的数据。在一些实施例中，ISP可以设置在摄像头393中。
摄像头393用于捕获静态图像或视频。物体通过镜头生成光学图像投射到感光元件。感光元件可以是电荷耦合器件(charge coupled device，CCD)或互补金属氧化物半导体(complementary metal-oxide-semiconductor，CMOS)光电晶体管。感光元件把光信号转换成电信号，之后将电信号传递给ISP转换成数字图像信号。ISP将数字图像信号输出到DSP加工处理。DSP将数字图像信号转换成标准的RGB，YUV等格式的图像信号。在一些实施例中，拍摄设备可以包括1个或N个摄像头393，N为大于1的正整数。本申请实施例对摄像头393在拍摄设备上的位置不作限定。
或者，拍摄设备可以不包括摄像头，即上述摄像头393并未设置于拍摄设备(如电视机)中。拍摄设备可以通过接口(如USB接口330)外接摄像头393。该外接的摄像头393可以通过外部固定件(如带夹子的摄像头支架)固定在拍摄设备上。例如，外接的摄像头393可以通过外部固定件，固定在拍摄设备的显示屏392的边缘处，如上侧边缘处。
数字信号处理器用于处理数字信号，除了可以处理数字图像信号，还可以处理其他数字信号。例如，当拍摄设备在频点选择时，数字信号处理器用于对频点能量进行傅里叶变换等。视频编解码器用于对数字视频压缩或解压缩。拍摄设备可以支持一种或多种视频编解码器。这样，拍摄设备可以播放或录制多种编码格式的视频，例如：动态图像专家组(moving picture experts group，MPEG)1，MPEG2，MPEG3，MPEG4等。
外部存储器接口320可以用于连接外部存储卡，例如Micro SD卡，实现扩展拍摄设备的存储能力。外部存储卡通过外部存储器接口320与处理器310通信，实现数据存储功能。例如将音乐、视频、图像等文件保存在外部存储卡中。
内部存储器321可以用于存储计算机可执行程序代码，所述可执行程序代码包括指令。处理器310通过运行存储在内部存储器321的指令，从而执行拍摄设备的各种功能应用以及数据处理。内部存储器321可以包括存储程序区和存储数据区。其中，存储程序区可存储操作系统，至少一个功能所需的应用程序(比如声音播放功能，图像播放功能等)等。存储数据区可存储拍摄设备使用过程中所创建的数据(比如音频数据等)等。此外，内部存储器321可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件，闪存器件，通用闪存存储器(universal flash storage，UFS)等。
拍摄设备可以通过音频模块370，扬声器370A，麦克风370C，音箱接口370B， 以及应用处理器等实现音频功能。例如，音乐播放，录音等。在本申请中，麦克风370C可以用于接收用户对拍摄设备发出的语音指令。扬声器370A可以用于向用户反馈拍摄设备的决策指令。
音频模块370用于将数字音频信息转换成模拟音频信号输出，也用于将模拟音频输入转换为数字音频信号。音频模块370还可以用于对音频信号编码和解码。在一些实施例中，音频模块370可以设置于处理器310中，或将音频模块370的部分功能模块设置于处理器310中。扬声器370A，也称“喇叭”，用于将音频电信号转换为声音信号。麦克风370C，也称“话筒”，“传声器”，用于将声音信号转换为电信号。
音箱接口370B用于连接有线音箱。音箱接口370B可以是USB接口330，也可以是3.5mm的开放移动拍摄设备平台(open mobile terminal platform，OMTP)标准接口，美国蜂窝电信工业协会(cellular telecommunications industry association of the USA，CTIA)标准接口。
按键390包括开机键，音量键等。按键390可以是机械按键。也可以是触摸式按键。拍摄设备可以接收按键输入，产生与拍摄设备的用户设置以及功能控制有关的键信号输入。
指示器391可以是指示灯，可以用于指示拍摄设备处于开机状态、待机状态或者关机状态等。例如，指示灯灭灯，可指示拍摄设备处于关机状态；指示灯为绿色或者蓝色，可指示拍摄设备处于开机状态；指示灯为红色，可指示拍摄设备处于待机状态。
可以理解的是，本申请实施例示意的结构并不构成对拍摄设备的具体限定。其可以具有比图3中所示出的更多的或者更少的部件，可以组合两个或更多的部件，或者可以具有不同的部件配置。例如，该拍摄设备还可以包括音箱等部件。图3中所示出的各种部件可以在包括一个或多个信号处理或专用集成电路在内的硬件、软件、或硬件和软件的组合中实现。
以下实施例中的方法均可以在具有上述硬件结构的拍摄设备中实现。以下实施例中以上述拍摄设备是智能手机为例，对本申请实施例的方法进行说明。
接下来，结合图4至图16对本申请实施例提供的图像处理方法进行详细阐述。图4为本申请实施例提供的一种图像处理方法的流程示意图。在这里以拍摄设备300对图像进行移轴畸变矫正为例进行说明。如图4所示，该方法包括以下步骤。
步骤410、拍摄设备300获取待处理图像。
拍摄设备300可以通过摄像头进行实时拍摄，将拍摄获取到的图像作为待处理图像。拍摄设备300还可以从存储器存储的图库中获取待处理图像。拍摄设备300还将从云端或其他设备获取待处理图像。本申请实施例对待处理图像的来源不予限定。
拍摄设备300获取到的待处理图像可以是未经过编辑的实时拍摄图像或已经过编辑的图像。例如，待处理图像已经过第三方图像应用编辑或已经过拍摄设备300携带的图像编辑算法编辑。图像编辑包括裁剪、旋转、矫正等几何变换操作。
可选地，待处理图像也可以是经过预处理后的图像。预处理包括黑电平校正、降噪、自动曝光、自动白平衡和图像变形等。
本申请实施例所述的待处理图像可以是由拍摄设备300发生倾斜(如：仰拍、俯拍或水平倾斜)拍摄第一场景得到的。待处理图像包含第一场景中垂直线段和水平线 段中至少一种。例如，待处理图像包含第一场景中垂直线段和水平线段中至少一种在待处理图像中发生倾斜的线段。第一场景中垂直线段在待处理图像中发生倾斜可以称为垂直倾斜线段。第一场景中水平线段在待处理图像中发生倾斜可以称为水平倾斜线段。可理解的是，垂直倾斜线段可以是指相对与地平线垂直的线段在垂直线段倾斜阈值内的倾斜线段。水平倾斜线段可以是指相对与地平线平行的线段在水平线段倾斜阈值内的倾斜线段。垂直线段倾斜阈值和水平线段倾斜阈值均可以是5°。
示例地，如图1中的(a)所示，相机水平倾斜拍摄得到的图像，图像包含了水平倾斜线段和垂直倾斜线段。如图1中的(c)所示，相机仰拍得到的图像，建筑物顶端逐渐缩小，图像包含了垂直倾斜线段。相反，相机俯拍得到的图像，建筑物底端逐渐缩小，图像包含了垂直倾斜线段。
步骤420、拍摄设备300根据直线检测方法检测待处理图像中的线段。
直线检测方法可以是霍夫直线检测或LSD直线检测。拍摄设备300根据直线检测方法检测待处理图像，得到待处理图像包含的M条线段，M为正整数。M条线段包含水平线段、垂直线段、水平倾斜线段和垂直倾斜线段中至少一种。
步骤430、拍摄设备300根据待处理图像的元数据包括的IMU信息和视场从待处理图像包含的线段中筛选得到待矫正线段。
由于IMU信息和视场描述了拍摄待处理图像的拍摄设备300的属性信息，拍摄设备300根据IMU信息和视场可以准确地定位需要矫正的线段，从待处理图像包含的线段中筛选待矫正线段，对待矫正线段以及待处理图像的特征区域进行矫正，使处理后图像中的线段更符合拍摄设备所拍摄的第一场景中水平线段和垂直线段的分布特征。示例地，如图5所示，元数据包括原始数据和新增数据。原始数据包含了镜头参数和曝光参数。新增数据包含了IMU信息和视场中至少一个。IMU信息可以是指拍摄待处理图像的设备的倾角，比如IMU信息包括水平倾角和俯仰倾角。例如，待处理图像是由拍摄设备300拍摄得到的，IMU信息可以是拍摄设备300的IMU信息。
图6所述的方法流程是对图4中步骤430所包括的具体操作过程的阐述，如图6所示。
步骤431、拍摄设备300根据IMU信息和视场对待处理图像包含的M条线段进行整体调整得到调整后M条线段。
拍摄设备300根据直线检测方法检测待处理图像得到的M条线段并非均是第一场景中的垂直线段或者水平线段，如果根据线段倾斜阈值对M条线段进行筛选，容易发生错误筛选。例如人像、衣服等线段在实际场景中并非垂直线段或者水平线段。因此，拍摄设备300根据IMU信息和视场对待处理图像包含的M条线段进行整体调整得到调整后M条线段，使调整后M条线段符合第一场景的线段角度分布特征，即调整后M条线段更符合第一场景中垂直线段特征和水平线段特征。
在一些实施例中，拍摄设备300对M条线段进行透视变换得到调整后M条线段，即透视变换后M条线段。
拍摄设备300基于直线检测方法检测得到待处理图像中的线段集合M表示线段数目。线段li由起点坐标和终点坐标构成。拍摄设备300根据元数据包含的俯仰倾角、水平倾角和视场，估算透视变换矩阵H，根据透视变换 矩阵H对M条线段的起点坐标和终点坐标均进行透视变换运算后，得到透视变换后线段集合线段l′i的起点坐标为和终点坐标分别为示例地，坐标(x,y)的透视变换如下公式(1)。
其中，H表示透视变换矩阵。坐标(x,y)的透视变换坐标为x’＝U/W,y’＝V/W。
步骤432、拍摄设备300根据线段倾斜阈值从调整后M条线段中筛选得到N条待矫正线段。
调整后M条线段中包含相对90°垂直线段倾斜角度大的线段、相对90°垂直线段倾斜角度小的线段、相对0°水平线段倾斜角度大的线段和相对0°水平线段倾斜角度小的线段。倾斜角度大的线段在第一场景中并非垂直线段或者水平线段，可能无需矫正，因此，拍摄设备300可以根据线段倾斜阈值从调整后M条线段中筛选N条待矫正线段，N条待矫正线段更符合第一场景的线段角度分布特征，从而对N条待矫正线段进行矫正，提高对待处理图像进行移轴畸变矫正的精确度。N为正整数，N小于或等于M。可理解的是，N等于M，表示调整后M条线段均为待矫正线段；N小于M，表示调整后M条线段中的部分线段为待矫正线段。线段倾斜阈值包括水平线段倾斜阈值和垂直线段倾斜阈值。
在一些实施例中，拍摄设备300根据调整后M条线段中每条线段的起点坐标和终点坐标计算线段角度，得到调整后M条线段的角度集合对于任一线段l′i，判断角度θ′i是否属于水平线段倾斜阈值范围或垂直线段倾斜阈值范围 即识别角度θ′i是否接近90°或0°，筛选得到水平倾斜线段集合和垂直倾斜线段集合。其中，Δh和Δv的取值可以均小于5°。
示例地，如图7所示，为本申请实施例提供的一种线段筛选可视化示意图。如图7中的(a)所示，为依据直线检测方法检测待处理图像得到的所有线段，包括的人像、衣服、椅子等线段。如图7中的(b)所示，为经过透视变换后符合实际场景中真实角度分布的线段。如图7中的(c)所示，为透视变换后线段的角度接近90°或0°筛选得到的待矫正线段。
应理解，待矫正线段包括第一场景中垂直线段和水平线段中至少一种在待处理图像中发生倾斜的线段中部分线段或全部线段。例如，待处理图像包含发生移轴畸变的建筑物图像和人像，待矫正线段包含建筑物中垂直线段和水平线段中至少一种在图像中发生倾斜的线段，不包括人像中发生倾斜的线段，仅对建筑物图像中发生倾斜的线段进行矫正，以此保留人像中大长腿的效果，使图像构图更美观。
步骤440、拍摄设备300对待矫正线段以及待处理图像的特征区域进行矫正，得到处理后图像。
拍摄设备300对待处理图像的图像数据进行矫正，图像数据包含了待处理图像的像素。
例如，拍摄设备300可以对待处理图像分区域构建约束项，联合优化各约束项， 获得全局网格矫正位移，对待处理图像进行图像仿射变换，得到处理后图像。区域可以包括直线区域、边缘区域、背景区域和人像区域。
直线约束对筛选得到待矫正线段包含的第一场景中垂直线段和水平线段构建横平竖直约束。
基于内容的单应性约束对图像中具有丰富边缘纹理区域构建单应性约束。由于存在直线漏检情况，对于漏检的直线未做横平竖直约束时，可能导致部分直线未进行矫正，因此约束有丰富纹理区域的网格顶点集，根据单应性映射进行整体变换。基于内容的单应性约束可以是背景区域的约束。单应性约束如下公式(2)。
其中，λH表示权重，0＜λH＜1.0。透视变换后的坐标向量(u,v)＝(U,V)/W，xi，j、yi，j表示原始坐标，i、j指示第i行第j列对应的网格点。
人像约束根据人像所在位置构建人像区域的几何变换约束。比如约束人像区域的网格点坐标满足球形变换关系，使得在矫正背景和人像的移轴畸变的基础上，优化人像的边缘拉伸等透视畸变。例如，俯拍人像时，造成人像头身比例失调，即头部图像较大，腿部图像较短。又如，仰拍人像时，造成人像头身比例失调，即头部图像较小，腿部图像较长。又如，超广角拍摄人像时，造成在图像边缘人像的头部图像或身体图像拉伸畸变。根据人像畸变现象构建人像约束，依据人像约束矫正人像图像中畸变。
边界约束用于约束视场边缘区域对应的像素点向外侧和内侧发生位移。为了减小视场损失，约束图像上下左右边缘的像素点向外移动。边界约束如下公式(3)。
其中，H表示图像的长，W表示图像的宽。ui，j(ui，j<0或者ui，j>W)、vi，j(vi，j<0或者vi，j>H)表示像素点位移后的x、y坐标，λB_in表示权重，0＜λB_in＜1.0。
另外，为了避免图像边缘纹理的拉伸，约束图像上下左右边缘的像素点向内移动。边界约束如下公式(4)。
其中，ui，j(0<ui，j<W)、vi，j(0<vi，j<H)表示像素点位移后的x、y坐标，λB_out表示权重，0＜λB_out＜1.0。
形状约束构建形状保持约束，避免图像纹理扭曲变形。网格点的变换满足局部相似变换或保形变换约束。形状约束如下公式(5)。
ES＝λS(∑i,j((vi+1,j-vi,j)+(ui,j+1-ui,j))2+∑i,j((ui+1,j-ui,j)+(vi,j+1-vi,j))2)公式(5)
其中，λS表示权重，0＜λS＜1.0。
使用有限差分计算二阶导数近似值并使用Frobenius范数给出平滑度约束。平滑度约束可以是指形状约束。平滑度约束如下公式(6)。
其中，λR表示权重，0＜λR＜1.0。
联合优化直线约束、内容形状保持(基于内容的单应性约束)约束、视场边界约束(边界约束)、人像约束、形状约束和平滑度约束等各项约束，矫正后图像更为合理美观，视场保持更多。
从而，相对于对待处理图像进行整体矫正，针对待处理图像中线段进行精准矫正，从而有效地提高了对图像进行移轴畸变矫正的精确度。
在另一些实施例中，拍摄设备300还可以根据元数据包含的内容制定线段筛选策略来筛选待矫正线段。线段筛选策略用于指示对待处理图像包含的线段进行线段筛选的方法。从而，根据元数据包含的内容适配不同的线段筛选策略来提升线段筛选的灵活性和精确度。
图8为本申请实施例提供的一种图像处理方法的流程示意图。在这里以拍摄设备300对图像进行移轴畸变矫正为例进行说明。如图8所示，该方法包括以下步骤。
步骤810、拍摄设备300获取待处理图像。
步骤820、拍摄设备300根据直线检测方法检测待处理图像中的线段。
关于步骤810和步骤820的详细解释可以参考上述步骤410和步骤420的阐述，不予赘述。
步骤830、拍摄设备300根据元数据确定的线段筛选策略，从待处理图像包含的线段中筛选得到待矫正线段。
应理解，待矫正线段包括第一场景中垂直线段和水平线段中至少一种在待处理图像中发生倾斜的线段中部分线段或全部线段。第一场景是指拍摄待处理图像时的场景。例如，待矫正线段包括第一场景中垂直线段。又如，待矫正线段包括第一场景中水平线段。又如，待矫正线段包括第一场景中垂直线段和水平线段。比如，待处理图像包含发生移轴畸变的建筑物图像和人像，待矫正线段包含建筑物中垂直线段和水平线段中至少一种在图像中发生倾斜的线段，不包括人像中发生倾斜的线段，仅对建筑物图像中发生倾斜的线段进行矫正，以此保留人像中大长腿的效果，使图像构图更美观。
拍摄设备300可以根据元数据包含的不同内容确定不同的线段筛选策略。示例地，步骤830包括步骤831至步骤834。
步骤831、拍摄设备300判断元数据是否包含IMU信息和视场。
若元数据包括IMU信息和视场，线段筛选策略指示基于IMU信息和视场进行线段筛选，拍摄设备300执行步骤832，即拍摄设备300基于IMU信息和视场从待处理图像包含的线段中筛选得到待矫正线段。关于拍摄设备300基于IMU信息和视场进行线段筛选的方案参考如图6的详细阐述。
若元数据不包括视场，线段筛选策略指示基于待处理图像的图像内容进行线段筛选，拍摄设备300执行步骤833，即拍摄设备300基于待处理图像的图像内容从待处理图像包含的线段中筛选得到待矫正线段。图像内容包含线段角度分布特征。关于拍 摄设备300基于待处理图像的图像内容进行线段筛选的方案参考如下图9的详细阐述。
若元数据包括视场，且不包括IMU信息。线段筛选策略指示基于视场和待处理图像的图像内容进行线段筛选，拍摄设备300执行步骤834，即拍摄设备300基于视场和待处理图像的图像内容从待处理图像包含的线段中筛选得到待矫正线段。关于拍摄设备300基于视场和待处理图像的图像内容进行线段筛选的方案参考如下图10的详细阐述。
图9为本申请实施例提供的一种基于图像内容的线段筛选方法示意图。
拍摄设备300可以根据线段筛选模型筛选M条线段中水平倾斜线段和垂直倾斜线段。例如，将待处理图像和M条线段标识输入线段筛选模型，得到水平倾斜线段集合和垂直倾斜线段集合。
在另一些实施例中，拍摄设备300可以根据消失点检测方法检测M条线段中垂直倾斜线段和水平倾斜线段。如下步骤910至步骤920。
步骤910、拍摄设备300根据M条线段的角度进行初筛得到初筛垂直倾斜线段集合。
拍摄设备300判断M条线段中每条线段的角度是否在垂直线段倾斜阈值范围(如[-75°,75°])内，若条线段的角度在垂直线段倾斜阈值范围内，表示该线段的角度接近90°，若条线段的角度不在垂直线段倾斜阈值范围内，表示该线段的角度远离90°，得到初步筛选出可能的初筛垂直倾斜线段集合K条线段如下公式(7)所示。
F0(x)＝a0*x+b0*y+c0＝0,
F1(x)＝a1*x+b1*y+c1＝0，公式(7)
……
FK(x)＝aK*x+bK*y+cK＝0
步骤920、拍摄设备300根据消失点检测方法从初筛垂直倾斜线段集合中筛选得到垂直倾斜线段集合。
垂直倾斜线段集合中的线段会相交于某点，联立上述公式(2)中K个方程，利用奇异值分解(singular value decomposition，SVD)分解法求解方程，得到垂直线段的消失点(xvanish，yvanish)。消失点是指平行线条的交点，仰视或者仰拍两条垂直平行线时，它们会相交于很远的某一点，即消失点，也称灭点。然后，拍摄设备300计算垂直线段的消失点到K条初筛线段的距离判断距离是否小于距离阈值dthresh，若距离小于距离阈值，表示线段的角度接近90°，确定该线段为垂直倾斜线段，若距离大于或等于距离阈值，表示线段的角度远离90°，确定该线段不是垂直倾斜线段，从而筛选出最终的垂直倾斜线段。可理解的是，如果第一场景中的垂直线段在图像中发生倾斜，该垂直线段经过消失点，因此，距离阈值dthresh设置为较小值，如图像分辨率为4000*3000时，距离阈值可以设置为5。图像分辨率增大时，距离阈值也等比增大。
基于视场和图像内容的线段筛选方案相对于基于图像内容线段筛选方案的区别在于，可根据视场适配不同视场大小对应的垂直倾斜阈值范围。因为，在同一垂直倾角和水平倾角下，不同的视场下，图像中的垂直线段的倾斜角度范围不同。视场越大， 图像边缘的垂直线段的倾斜度越大。示例地，如图10所示，虚线框内容的第一视场小于实现框内容的第二视场，线段L1、线段L2、线段L3和线段L4在实际场景中相互平行且垂直地面，但由于仰拍导致4条垂直线段发生倾斜，且同样的拍摄仰拍角度下，L3和L4的倾斜角度大于L1和L2的倾斜角度。因此，根据视场适配不同视场大小对应的垂直线段倾斜阈值范围，提升初筛线段的精确度。拍摄设备300可以线性适配视场对应的垂直线段倾斜阈值范围。视场越大，垂直线段倾斜阈值范围越大。
步骤840、拍摄设备300对待矫正线段以及待处理图像的特征区域进行矫正，得到处理后图像。
关于步骤840的详细解释可以参考上述步骤440的阐述，不予赘述。
从而，拍摄设备300根据元数据包含的内容适配不同的线段筛选策略，提升对待处理图像包含的线段进行线段筛选的灵活性和精确度。
本申请实施例所述的待处理图像可以是经过第三方图像应用编辑的图像，则拍摄图像时所采集的IMU信息和视场无法准确描述图像。在一些实施例中，元数据还可以包括可信标识和编辑标识中至少一个。可信标识用于指示元数据是否可信。示例地，可信标识的取值为1，表示元数据不可信；可信标识的取值为0，表示元数据可信。可信标识可以由前端IMU模组提供。可选地，若待处理图像经过了第三方图像应用的裁剪/旋转/矫正等几何变换的编辑操作，则认为IMU信息和视场不可信。编辑标识用于指示待处理图像是否经过图像应用处理。
如图11所示，拍摄设备300执行步骤831之前，还可以执行步骤850，即判断元数据是否可信。判断元数据是否可信也可以替换为判断待处理图像是否经过图像应用处理。
若可信标识指示元数据不可信，或经过图像应用处理，执行步骤833，即基于待处理图像的图像内容从待处理图像包含的线段中筛选得到待矫正线段。
若可信标识指示元数据可信，或未经过图像应用处理，执行步骤832，即基于IMU信息和视场从待处理图像包含的线段中筛选得到待矫正线段；或者，执行步骤834，即基于视场和待处理图像的图像内容从待处理图像包含的线段中筛选得到待矫正线段。
如此，在根据IMU信息和视场进行线段筛选之前，先预判IMU信息和视场是否可信，根据IMU信息和视场的可信程度，确定进行线段筛选的线段筛选策略，从而提升线段筛选的速率，提高对图像中畸变线段进行矫正的精确度。
为了进一步提升基于图像内容的线段筛选结果，本申请实施例提供了用户交互的直线推荐策略。示例地，如图12所示，拍摄设备300执行步骤833或步骤834之后，还可以执行步骤860，即提示用户从基于待处理图像的图像内容筛选得到的推荐线段中选择待矫正线段。一种示例地，关于步骤860可以包含以下详细步骤。
步骤861、拍摄设备300判断水平方向是否被稳定识别。
若水平方向未被稳定识别，执行步骤862，即界面显示推荐水平线段。若用户不对推荐水平线段进行选择，则执行步骤863，提示用户手动画出需要矫正的水平倾斜线段。
若水平方向被稳定识别，执行步骤864，即判断垂直方向是否被稳定识别，输出水平倾斜线段。
若垂直方向未被稳定识别，执行步骤865，即界面显示推荐垂直线段。若用户不对推荐垂直线段进行选择，则执行步骤866，提示用户手动画出需要矫正的垂直倾斜线段。
若垂直方向被稳定识别，输出垂直倾斜线段。
若拍摄设备300确定水平方向和垂直方向均被稳定识别，则执行步骤840，即拍摄设备300对基于图像内容进行线段筛选得到的矫正线段进行矫正以及待处理图像的特征区域进行矫正，得到处理后图像。若拍摄设备300确定水平方向和垂直方向均未被稳定识别，在界面中高亮显示推荐的水平倾斜线段或垂直倾斜线段，识别到用户选择的线段，则对用户选择的线段进行矫正。
可选地，拍摄设备300可以根据基于图像内容待进行线段筛选得到的垂直线段和水平线段的数量和分布判断是否被稳定识别。例如，当筛选得到的线段数目过少，或者线段分布较为混乱时，可认为未被稳定识别。
示例地，如图13所示，为本申请实施例提供的一种线段选择界面示意图。如图13中的(a)所示，为拍摄设备300确定水平方向或垂直方向未被稳定识别，界面显示推荐线段。如图13中的(b)所示，为拍摄设备300确定水平方向或垂直方向未被稳定识别，用户未对推荐线段进行选择，界面提示用户手动画出需要矫正的倾斜线段。
在另一种可能的实现方式中，拍摄设备300基于两个不同的线段筛选策略对待处理图像进行筛选时，无需执行上述图8所述的根据元数据确定线段筛选策略的过程，即无需执行步骤830的过程，即拍摄设备300根据两种不同的线段筛选策略对待处理图像进行筛选，将两种方式筛选的线段集合取交集或并集，确定最终的待矫正线段。示例地，拍摄设备300根据基于IMU信息和视场筛选的线段和基于图像内容筛选的线段确定待矫正线段。该实施例与上述实施例的区别在于，拍摄设备300根据基于IMU信息和视场筛选线段的基础上，增加基于图像内容识别需要矫正的线段来提升移轴畸变矫正精确度。比如，基于IMU信息和视场进行线段筛选得到线段集合LC1，基于图像内容进行线段筛选得到线段集合LC2，通过求解LC1和LC2的交集或者并集来得到最终的待矫正线段集合LC。如此，进一步地提高基于IMU信息和视场进行线段筛选的精确度，以及提高了对图像进行移轴畸变矫正的精确度。
需要说明的是，对图像中倾斜线段的矫正的必要程度与被摄物体和用户意图相关。当拍摄主体为强规则直线场景，比如建筑构造、水平线、地平线、海报、窗框、门框、橱柜等时，倾斜矫正的必要性通常较高，而当拍摄主体为植物、动物、食物、面部、天空等非强规则直线场景时，倾斜矫正的必要性往往较低。另外，图像包含较多的倾斜线段往往与用户意图有关，比如当拍摄主体为仰拍人像，或者拍摄倾斜角度过大时，往往是用户的构图意图。
在另一种可能的实现方式中，拍摄设备300执行步骤840，即对待矫正线段进行矫正之前，拍摄设备300还可以分析用户行为，自适应选择是否矫正筛选得到的垂直倾斜线段和水平倾斜线段。示例地，如图14所示。
步骤870、拍摄设备300判断待矫正线段的数量是否大于数量阈值，且待矫正线段的长度是否大于长度阈值。
若待矫正线段的数量大于数量阈值，且待矫正线段的长度大于长度阈值，表示的 第一场景为强规则直线场景，则执行步骤840，即对待矫正线段进行矫正，得到处理后图像。
若待矫正线段的数量小于或等于数量阈值，或待矫正线段的长度小于或等于长度阈值，表示第一场景为非强规则直线场景，则不进行自动矫正，执行步骤880。
步骤880、拍摄设备300根据第一场景的场景特征确定是否对待矫正线段进行矫正。
场景特征包含建筑特征和人物特征。拍摄设备300可以根据不同的场景特征确定不同的水平线段倾斜阈值和垂直线段倾斜阈值。拍摄设备300判断水平倾斜线段的倾角是否在水平线段倾斜阈值范围内，若水平倾斜线段的倾角在水平线段倾斜阈值范围内，对水平倾斜线段进行矫正。拍摄设备300还可以判断垂直倾斜线段的倾角是否在垂直线段倾斜阈值范围内，若垂直倾斜线段的倾角在垂直线段倾斜阈值范围内，对垂直倾斜线段进行矫正。
例如，待处理图像包含仰拍人像图像和仰拍建筑物图像。如果对人像中垂直倾斜线段进行矫正后可能会破坏“大长腿”效果，可对水平倾斜线段进行矫正，以及建筑物中垂直倾斜线段和水平倾斜线段进行矫正，不对人像图像中的垂直倾斜线段进行矫正。当水平倾角和俯仰倾角过大时，通常是用户有意为之，则不对图像中垂直倾斜线段和水平倾斜线段进行矫正。比如，拍摄设备300对相对90°和0°偏移25°范围内的线段进行矫正。
如此，对图像中倾斜线段进行矫正时，充分参考场景特征和用户意图，使得矫正后图像更符合实际场景中线段的分布特征，有效地提高了对图像进行移轴畸变矫正的精确度。
本申请实施例结合图像的元数据、图像内容分析结果以及用户交互的直线推荐策略，进行倾斜线段筛选，联合优化各项约束，获得矫正水平倾角和俯仰倾角导致的垂直倾斜和水平倾角带来的水平歪斜的全局位移，并结合直线检测结果和用户行为分析，自适应进行不同程度的移轴畸变矫正，有效地提高了对图像进行移轴畸变矫正的精确度。
下面对图像进行移轴畸变矫正的界面操作进行举例说明。
图15为本申请实施例提供的一种图像处理的界面示意图。如图15中的(a)所示，为本申请实施例提供的智能手机的正面示意图。智能手机的显示屏上显示有相册应用(application，APP)图标1501，用户可以点击相册应用图标1501。智能手机包含的触摸传感器接收到触摸操作，上报给处理器，使得处理器响应于上述触摸操作，启动相册应用。此外，本申请实施例中还可以通过其它方式使得智能手机启动相册应用，并在显示屏上显示相册应用的用户界面。例如，智能手机当黑屏、显示锁屏界面或者解锁后显示某一用户界面时，可以响应于用户的语音指令或者快捷操作等，启动相册应用，并在显示屏上显示相册应用的用户界面。如图15中的(b)所示，智能手机响应于点击操作，智能手机的显示屏显示相册应用的用户界面，用户界面包括不同时间拍摄的照片，以及照片搜索框。用户可以选择倾斜拍摄的移轴畸变照片1502，对移轴畸变照片1502进行移轴畸变矫正。如图15中的(c)所示，智能手机响应于点击操作，智能手机的显示屏显示移轴畸变照片1502，以及照片的编辑功能按钮，例如“发送” 功能按钮、“编辑”功能按钮、“收藏”功能按钮、“删除”功能按钮和“更多”功能按钮。用户可以点击“编辑”功能按钮1503，智能手机的显示屏显示“编辑”功能的功能选项。如图15中的(d)所示，智能手机响应于点击操作，智能手机的显示屏显示“编辑”功能的功能选项。“编辑”功能的功能选项包括智能、修剪、滤镜和调节等。用户可以点击“智能”选项按钮1504，智能手机的显示屏显示“智能”选项的多个功能选项。例如，“智能”选项包括自动优化、反光消除、增亮、除雾和建筑矫正等。如图15中的(e)所示，智能手机响应于点击操作，智能手机的显示屏还可以显示“智能”选项包括的“移轴畸变”功能选项、“水平矫正”功能选项和“垂直矫正”功能选项等。用户可以点击“移轴畸变”功能选项1505。智能手机对移轴畸变照片1502进行移轴畸变矫正。如图15中的(f)所示，智能手机响应于点击操作，显示屏显示对移轴畸变照片1502进行移轴畸变矫正后照片1506。智能手机对移轴畸变照片1502进行移轴畸变矫正的具体方法可以参考上述实施例的阐述，不予赘述。
在另一些实施例中，拍摄设备也可以在拍摄照片后根据用户的移轴畸变操作对移轴畸变图像进行矫正。示例地，图16为本申请实施例提供的另一种图像处理的界面示意图。如图16中的(a)所示，为本申请实施例提供的智能手机的正面示意图。智能手机的显示屏上显示有相机应用(application，APP)图标1601，用户可以点击相机应用图标1601。智能手机包含的触摸传感器接收到触摸操作，上报给处理器，使得处理器响应于上述触摸操作，启动相机应用。此外，本申请实施例中还可以通过其它方式使得智能手机启动相机应用，并在显示屏上显示相机应用的用户界面。例如，智能手机当黑屏、显示锁屏界面或者解锁后显示某一用户界面时，可以响应于用户的语音指令或者快捷操作等，启动相机应用，并在显示屏上显示相机应用的用户界面。如图16中的(b)所示，智能手机响应于点击操作，智能手机的显示屏显示相机应用的用户界面，用户界面包括“小视频”、“录像”、“拍照”、“人像”、“全景”等模式选项，“预览图像”功能按钮、“拍照”功能按钮和“前后摄像头转换”功能按钮。用户可以先选择“拍照”模式1602，再点击“拍照”功能按钮1603，智能手机响应于点击操作，自动开启摄像头，摄像头获取场景的仰拍图像。进而，如图16中的(c)所示，用户可以点击“预览图像”功能按钮1604，查看拍摄的包含目标主体的图像，即移轴畸变照片1605。如图16中的(d)所示，智能手机响应于点击操作，智能手机的显示屏显示移轴畸变照片1605，以及照片的编辑功能按钮，例如“发送”功能按钮、“编辑”功能按钮、“收藏”功能按钮、“删除”功能按钮和“更多”功能按钮。用户可以点击“编辑”功能按钮1606，智能手机的显示屏显示“编辑”功能的功能选项。如图16中的(e)所示，智能手机响应于点击操作，智能手机的显示屏显示“编辑”功能的功能选项。“编辑”功能的功能选项包括智能、修剪、滤镜和调节等用户可以点击“智能”选项按钮1607，智能手机的显示屏显示“智能”选项的多个功能选项。例如，“智能”选项包括自动优化、反光消除、增亮、除雾和建筑矫正等。如图16中的(f)所示，智能手机响应于点击操作，智能手机的显示屏还可以显示“智能”选项包括的“移轴畸变”功能选项、“水平矫正”功能选项和“垂直矫正”功能选项等。用户可以点击“移轴畸变”功能选项1608。智能手机对移轴畸变照片1605进行移轴畸变矫正。如图16中的(g)所示，智能手机响应于点击操作，显示屏显示对移轴畸 变照片1605进行移轴畸变矫正后照片1609。智能手机对移轴畸变照片1605进行移轴畸变矫正的具体方法可以参考上述实施例的阐述，不予赘述。
在另一些实施例中，拍摄设备也可以在拍摄照片后，拍摄设备自动对图像进行移轴畸变矫正。例如，如图16中的(b)所示，智能手机响应于点击操作，智能手机的显示屏显示相机应用的用户界面，智能手机响应于点击“拍照”功能按钮1603操作，自动开启摄像头，摄像头获取场景的仰拍图像，并对仰拍的移轴畸变照片1605进行移轴畸变矫正，得到轴畸变矫正后照片1609，如图16中的(g)所示，智能手机的显示屏显示对移轴畸变照片1605进行移轴畸变矫正后照片1609。省去了图16中的(c)
至(f)的步骤，提高了用户使用智能手机的用户体验。
需要说明的是，本申请实施例对“移轴畸变”功能选项、“水平矫正”功能选项和“垂直矫正”功能选项的显示方式不予限定。拍摄设备的显示屏显示的各功能选项也可以称为控件或功能控件。
可以理解的是，为了实现上述实施例中的功能，拍摄设备包括了执行各个功能相应的硬件结构和/或软件模块。本领域技术人员应该很容易意识到，结合本申请中所公开的实施例描述的各示例的单元及方法步骤，本申请能够以硬件或硬件和计算机软件相结合的形式来实现。某个功能究竟以硬件还是计算机软件驱动硬件的方式来执行，取决于技术方案的特定应用场景和设计约束条件。
上文中结合图1至图16，详细描述了根据本实施例所提供的图像处理方法，下面将结合图17，描述根据本实施例所提供的图像处理装置。
图17为本实施例提供的可能的图像处理装置的结构示意图。这些图像处理装置可以用于实现上述方法实施例中拍摄设备的功能，因此也能实现上述方法实施例所具备的有益效果。在本实施例中，该图像处理装置可以是如图3所示的拍摄设备300，还可以是应用于服务器的模块(如芯片)。
如图17所示，图像处理装置1700包括通信模块1710、线段筛选模块1720、矫正模块1730和存储模块1740。图像处理装置1700用于实现上述图4、图6、图8、图11、图12和图14中所示的方法实施例中拍摄设备300的功能。
通信模块1710用于获取待处理图像，所述待处理图像包含第一场景中垂直线段和水平线段中至少一种，所述第一场景为拍摄设备获取所述待处理图像的场景。例如，通信模块1710用于执行步骤410。又如，通信模块1710用于执行步骤810。
线段筛选模块1720，用于根据所述待处理图像的元数据从所述待处理图像包含的线段中筛选得到待矫正线段，所述元数据包括所述拍摄设备的惯性测量组件IMU信息和视场中至少一个。例如，线段筛选模块1720用于执行步骤420和步骤430。又如，线段筛选模块1720用于执行步骤820、步骤830、步骤850、步骤860、步骤870和步骤880。
矫正模块1730，用于对所述待矫正线段以及所述待处理图像的特征区域进行矫正，得到处理后图像，所述特征区域包括背景区域、人像区域和边缘区域。例如，矫正模块1730用于执行步骤440。又如，矫正模块1730用于执行步骤840。
可选地，图像处理装置1700可以将处理后图像上传到服务器或边缘设备或云端设备或存储在本地存储器中。
存储模块1740用于存储元数据，元数据指示了待处理图像的属性信息，例如，IMU信息和视场，以便于根据待处理图像的元数据确定的线段筛选策略。
可选地，图像处理装置1700还可以包括显示模块1750，显示模块1750用于显示待处理图像和处理后图像。
应理解的是，本申请实施例的图像处理装置1700可以通过专用集成电路(application-specific integrated circuit，ASIC)实现，或可编程逻辑器件(programmable logic device，PLD)实现，上述PLD可以是复杂程序逻辑器件(complex programmable logical device，CPLD)，现场可编程门阵列(field-programmable gate array，FPGA)，通用阵列逻辑(generic array logic，GAL)或其任意组合。也可以通过软件实现图4、图6、图8、图11、图12和图14所示的图像处理方法时，图像处理装置1700及其各个模块也可以为软件模块，图像处理装置1700及其各个模块也可以为软件模块。
根据本申请实施例的图像处理装置1700可对应于执行本申请实施例中描述的方法，并且图像处理装置1700中的各个单元的上述和其它操作和/或功能分别为了实现图4、图6、图8、图11、图12和图14中的各个方法的相应流程，为了简洁，在此不再赘述。
图18为本实施例提供的一种拍摄设备1800的结构示意图。如图所示，拍摄设备1800包括处理器1810、总线1820、存储器1830、通信接口1840、内存单元1850(也可以称为主存(main memory)单元)和摄像头1860。可选地，拍摄设备1800还可以包含显示器1870，显示器1870用于显示待处理图像和处理后图像。处理器1810、存储器1830、内存单元1850、通信接口1840、摄像头1860和显示器1870通过总线1820相连。
应理解，在本实施例中，处理器1810可以是CPU，该处理器1810还可以是其他通用处理器、数字信号处理器(digital signal processing，DSP)、ASIC、FPGA或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。通用处理器可以是微处理器或者是任何常规的处理器等。
处理器还可以是图形处理器(graphics processing unit，GPU)、神经网络处理器(neural network processing unit，NPU)、微处理器、ASIC、或一个或多个用于控制本申请方案程序执行的集成电路。
通信接口1840用于实现拍摄设备1800与外部设备或器件的通信。在本实施例中，拍摄设备1800用于实现图4、图6、图8、图11、图12和图14所示的拍摄设备300的功能时，通信接口1840用于接收待处理图像。
总线1820可以包括一通路，用于在上述组件(如处理器1810、内存单元1850和存储器1830)之间传送信息。总线1820除包括数据总线之外，还可以包括电源总线、控制总线和状态信号总线等。但是为了清楚说明起见，在图中将各种总线都标为总线1820。总线1820可以是快捷外围部件互连标准(Peripheral Component Interconnect Express，PCIe)总线，或扩展工业标准结构(extended industry standard architecture，EISA)总线、统一总线(unified bus，Ubus或UB)、计算机快速链接(compute express link，CXL)、缓存一致互联协议(cache coherent interconnect for accelerators，CCIX)等。总线1820可以分为地址总线、数据总线、控制总线等。
作为一个示例，拍摄设备1800可以包括多个处理器。处理器可以是一个多核(multi-CPU)处理器。这里的处理器可以指一个或多个设备、电路、和/或用于处理数据(例如计算机程序指令)的计算单元。在本实施例中，拍摄设备1800用于实现图4、图6、图8、图11、图12和图14所示的拍摄设备300的功能时，处理器1810可以调用存储器1830存储的元数据，根据待处理图像的元数据包含的IMU信息和视场中至少一个从待处理图像包含的线段中筛选得到待矫正线段，对待矫正线段以及待处理图像的特征区域包括背景区域、人像区域和边缘区域进行矫正，得到处理后图像。
值得说明的是，图18中仅以拍摄设备1800包括1个处理器1810和1个存储器1830为例，此处，处理器1810和存储器1830分别用于指示一类器件或设备，具体实施例中，可以根据业务需求确定每种类型的器件或设备的数量。
内存单元1850可以对应上述方法实施例中用于存储元数据等信息的存储介质。内存单元1850可以是易失性存储器或非易失性存储器，或可包括易失性和非易失性存储器两者。其中，非易失性存储器可以是只读存储器(read-only memory，ROM)、可编程只读存储器(programmable ROM，PROM)、可擦除可编程只读存储器(erasable PROM，EPROM)、电可擦除可编程只读存储器(electrically EPROM，EEPROM)或闪存。易失性存储器可以是随机存取存储器(random access memory，RAM)，其用作外部高速缓存。通过示例性但不是限制性说明，许多形式的RAM可用，例如静态随机存取存储器(static RAM，SRAM)、动态随机存取存储器(DRAM)、同步动态随机存取存储器(synchronous DRAM，SDRAM)、双倍数据速率同步动态随机存取存储器(double data date SDRAM，DDR SDRAM)、增强型同步动态随机存取存储器(enhanced SDRAM，ESDRAM)、同步连接动态随机存取存储器(synchlink DRAM，SLDRAM)和直接内存总线随机存取存储器(direct rambus RAM，DR RAM)。
存储器1830可以对应上述方法实施例中用于存储计算机指令等信息的存储介质，例如，磁盘，如机械硬盘或固态硬盘。
上述拍摄设备1800可以是一个通用设备或者是一个专用设备。例如，拍摄设备1800可以是边缘设备(例如，携带具有处理能力芯片的盒子)等。可选地，拍摄设备1800也可以是服务器或其他具有计算能力的设备。
应理解，根据本实施例的拍摄设备1800可对应于本实施例中的图像处理装置1700，并可以对应于执行根据图4、图6、图8、图11、图12和图14中任一方法中的相应主体，并且图像处理装置1700中的各个模块的上述和其它操作和/或功能分别为了实现图4、图6、图8、图11、图12和图14中的各个方法的相应流程，为了简洁，在此不再赘述。
本实施例中的方法步骤可以通过硬件的方式来实现，也可以由处理器执行软件指令的方式来实现。软件指令可以由相应的软件模块组成，软件模块可以被存放于随机存取存储器(random access memory，RAM)、闪存、只读存储器(read-only memory，ROM)、可编程只读存储器(programmable ROM，PROM)、可擦除可编程只读存储器(erasable PROM，EPROM)、电可擦除可编程只读存储器(electrically EPROM，EEPROM)、寄存器、硬盘、移动硬盘、CD-ROM或者本领域熟知的任何其它形式的存储介质中。一种示例性的存储介质耦合至处理器，从而使处理器能够从该存储介质 读取信息，且可向该存储介质写入信息。当然，存储介质也可以是处理器的组成部分。处理器和存储介质可以位于ASIC中。另外，该ASIC可以位于拍摄设备中。当然，处理器和存储介质也可以作为分立组件存在于拍摄设备中。
在上述实施例中，可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。所述计算机程序产品包括一个或多个计算机程序或指令。在计算机上加载和执行所述计算机程序或指令时，全部或部分地执行本申请实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络、网络设备、用户设备或者其它可编程装置。所述计算机程序或指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质传输，例如，所述计算机程序或指令可以从一个网站站点、计算机、服务器或数据中心通过有线或无线方式向另一个网站站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质或者是集成一个或多个可用介质的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质，例如，软盘、硬盘、磁带；也可以是光介质，例如，数字视频光盘(digital video disc，DVD)；还可以是半导体介质，例如，固态硬盘(solid state drive，SSD)。以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到各种等效的修改或替换，这些修改或替换都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以权利要求的保护范围为准。

Claims (28)

  1. 一种图像处理方法，其特征在于，包括：
    获取待处理图像，所述待处理图像包含第一场景中垂直线段和水平线段中至少一种，所述第一场景为拍摄设备获取所述待处理图像的场景；
    根据所述待处理图像的元数据从所述待处理图像包含的线段中筛选得到待矫正线段，所述元数据包括所述拍摄设备的惯性测量组件IMU信息和视场中至少一个；
    对所述待矫正线段以及所述待处理图像的特征区域进行矫正，得到处理后图像，所述特征区域包括背景区域、人像区域和边缘区域。
  2. 根据权利要求1所述的方法，其特征在于，所述待处理图像包含所述第一场景中垂直线段和水平线段中至少一种在所述待处理图像中发生倾斜的线段。
  3. 根据权利要求2所述的方法，其特征在于，所述待矫正线段包括所述第一场景中垂直线段和水平线段中至少一种在所述待处理图像中发生倾斜的线段中部分线段或全部线段。
  4. 根据权利要求1-3中任一项所述的方法，其特征在于，所述拍摄设备显示移轴畸变控件，根据所述待处理图像的元数据从所述待处理图像包含的线段中筛选得到待矫正线段之前，所述方法还包括：
    接收用户对所述移轴畸变控件的操作。
  5. 根据权利要求1-4中任一项所述的方法，其特征在于，根据所述待处理图像的元数据从所述待处理图像包含的线段中筛选得到待矫正线段，包括：
    根据所述元数据确定的线段筛选策略，从所述待处理图像包含的线段中筛选得到待矫正线段，所述线段筛选策略用于指示对所述待处理图像包含的线段进行线段筛选的方法。
  6. 根据权利要求5所述的方法，其特征在于，所述元数据包括所述IMU信息和所述视场；所述线段筛选策略用于指示基于所述IMU信息和所述视场进行线段筛选；
    根据所述元数据确定的线段筛选策略，从所述待处理图像包含的线段中筛选得到待矫正线段，包括：
    根据所述IMU信息和所述视场对所述待处理图像包含的M条线段进行整体调整得到调整后M条线段，所述调整后M条线段符合所述第一场景的线段角度分布特征；
    根据线段倾斜阈值从所述调整后M条线段中筛选得到N条待矫正线段，所述线段倾斜阈值包括水平线段倾斜阈值和垂直线段倾斜阈值，M和N均为正整数，N小于或等于M。
  7. 根据权利要求5所述的方法，其特征在于，所述元数据不包括所述视场，根据所述元数据确定的线段筛选策略，从所述待处理图像包含的线段中筛选得到待矫正线段，包括：
    基于所述待处理图像的图像内容从所述待处理图像包含的线段中筛选得到所述待矫正线段，所述图像内容包含线段角度分布特征。
  8. 根据权利要求5所述的方法，其特征在于，所述元数据包括所述视场，且不包括所述IMU信息，根据所述元数据确定的线段筛选策略，从所述待处理图像包含的线段中筛选得到待矫正线段，包括：
    基于所述视场和所述待处理图像的图像内容从所述待处理图像包含的线段中筛选得到所述待矫正线段，所述图像内容包含线段角度分布特征。
  9. 根据权利要求7或8所述的方法，其特征在于，所述方法还包括：
    提示用户从基于所述待处理图像的图像内容筛选得到的推荐线段中选择待矫正线段。
  10. 根据权利要求5-8中任一项所述的方法，其特征在于，所述元数据还包括可信标识，所述可信标识用于指示所述元数据是否可信；根据所述元数据确定的线段筛选策略，从所述待处理图像包含的线段中筛选得到待矫正线段，包括：
    若所述可信标识指示所述元数据不可信，基于所述待处理图像的图像内容从所述待处理图像包含的线段中筛选得到所述待矫正线段，所述图像内容包含线段角度分布特征；
    若所述可信标识指示所述元数据可信，基于所述待处理图像的图像内容、所述IMU信息和所述视场中至少一个从所述待处理图像包含的线段中筛选得到所述待矫正线段。
  11. 根据权利要求1-4中任一项所述的方法，其特征在于，根据所述待处理图像的元数据从所述待处理图像包含的线段中筛选得到待矫正线段，包括：
    根据基于所述元数据筛选的线段和基于图像内容筛选的线段确定所述待矫正线段。
  12. 根据权利要求1-11中任一项所述的方法，其特征在于，对所述待矫正线段以及所述待处理图像的特征区域进行矫正，得到处理后图像，包括：
    判断所述待矫正线段的数量是否大于数量阈值，且所述待矫正线段的长度是否大于长度阈值；
    若所述待矫正线段的数量大于所述数量阈值，且所述待矫正线段的长度大于所述长度阈值，对所述待矫正线段进行矫正，得到所述处理后图像；
    若所述待矫正线段的数量小于或等于所述数量阈值，或所述待矫正线段的长度小于或等于所述长度阈值，根据第一场景的场景特征确定是否对所述待矫正线段进行矫正，所述场景特征包含建筑特征和人物特征。
  13. 根据权利要求1-12中任一项所述的方法，其特征在于，对所述待矫正线段以及所述待处理图像的特征区域进行矫正，得到处理后图像，包括：
    构建直线约束对所述待矫正线段进行矫正；构建基于内容的单应性约束和形状和正则约束对背景区域进行矫正；构建人像约束对人像区域进行矫正；构建边界约束对边缘区域进行矫正，以得到所述处理后图像。
  14. 一种图像处理装置，其特征在于，包括：
    通信模块，用于获取待处理图像，所述待处理图像包含第一场景中垂直线段和水平线段中至少一种，所述第一场景为拍摄设备获取所述待处理图像的场景；
    线段筛选模块，用于根据所述待处理图像的元数据从所述待处理图像包含的线段中筛选得到待矫正线段，所述元数据包括所述拍摄设备的惯性测量组件IMU信息和视场中至少一个；
    矫正模块，用于对所述待矫正线段以及所述待处理图像的特征区域进行矫正，得到处理后图像，所述特征区域包括背景区域、人像区域和边缘区域。
  15. 根据权利要求14所述的装置，其特征在于，所述待处理图像包含所述第一场 景中垂直线段和水平线段中至少一种在所述待处理图像中发生倾斜的线段。
  16. 根据权利要求15所述的装置，其特征在于，所述待矫正线段包括所述第一场景中垂直线段和水平线段中至少一种在所述待处理图像中发生倾斜的线段中部分线段或全部线段。
  17. 根据权利要求14-16中任一项所述的装置，其特征在于，所述装置还包括显示模块；
    所述显示模块，用于显示移轴畸变控件，以及接收用户对所述移轴畸变控件的操作。
  18. 根据权利要求14-17中任一项所述的装置，其特征在于，所述线段筛选模块根据所述待处理图像的元数据从所述待处理图像包含的线段中筛选得到待矫正线段时，具体用于：
    根据所述元数据确定的线段筛选策略，从所述待处理图像包含的线段中筛选得到待矫正线段，所述线段筛选策略用于指示对所述待处理图像包含的线段进行线段筛选的方法。
  19. 根据权利要求18所述的装置，其特征在于，所述元数据包括所述IMU信息和所述视场；所述线段筛选策略用于指示基于所述IMU信息和所述视场进行线段筛选；
    所述线段筛选模块根据所述元数据确定的线段筛选策略，从所述待处理图像包含的线段中筛选得到待矫正线段时，具体用于：
    根据所述IMU信息和所述视场对所述待处理图像包含的M条线段进行整体调整得到调整后M条线段，所述调整后M条线段符合第一场景的线段角度分布特征；
    根据线段倾斜阈值从所述调整后M条线段中筛选得到N条待矫正线段，所述线段倾斜阈值包括水平线段倾斜阈值和垂直线段倾斜阈值，M和N均为正整数，N小于或等于M。
  20. 根据权利要求18所述的装置，其特征在于，所述元数据不包括所述视场，所述线段筛选模块根据所述元数据确定的线段筛选策略，从所述待处理图像包含的线段中筛选得到待矫正线段时，具体用于：
    基于所述待处理图像的图像内容从所述待处理图像包含的线段中筛选得到所述待矫正线段，所述图像内容包含线段角度分布特征。
  21. 根据权利要求18所述的装置，其特征在于，所述元数据包括所述视场，且不包括所述IMU信息，所述线段筛选模块根据所述元数据确定的线段筛选策略，从所述待处理图像包含的线段中筛选得到待矫正线段时，具体用于：
    基于所述视场和所述待处理图像的图像内容从所述待处理图像包含的线段中筛选得到所述待矫正线段，所述图像内容包含线段角度分布特征。
  22. 根据权利要求20或21所述的装置，其特征在于，所述装置还包括显示模块：
    所述显示模块，用于提示用户从基于所述待处理图像的图像内容筛选得到的推荐线段中选择待矫正线段。
  23. 根据权利要求18-21中任一项所述的装置，其特征在于，所述元数据还包括可信标识，所述可信标识用于指示所述元数据是否可信；所述线段筛选模块根据所述元数据确定的线段筛选策略，从所述待处理图像包含的线段中筛选得到待矫正线段时， 具体用于：
    若所述可信标识指示所述元数据不可信，基于所述待处理图像的图像内容从所述待处理图像包含的线段中筛选得到所述待矫正线段，所述图像内容包含线段角度分布特征；
    若所述可信标识指示所述元数据可信，基于所述待处理图像的图像内容、所述IMU信息和所述视场中至少一个从所述待处理图像包含的线段中筛选得到所述待矫正线段。
  24. 根据权利要求14-17中任一项所述的装置，其特征在于，所述线段筛选模块根据所述待处理图像的元数据从所述待处理图像包含的线段中筛选得到待矫正线段时，具体用于：
    根据基于所述元数据筛选的线段和基于图像内容筛选的线段确定所述待矫正线段。
  25. 根据权利要求14-24中任一项所述的装置，其特征在于，所述矫正模块对所述待矫正线段以及所述待处理图像的特征区域进行矫正，得到处理后图像时，具体用于：
    判断所述待矫正线段的数量是否大于数量阈值，且所述待矫正线段的长度是否大于长度阈值；
    若所述待矫正线段的数量大于所述数量阈值，且所述待矫正线段的长度大于所述长度阈值，对所述待矫正线段进行矫正，得到所述处理后图像；
    若所述待矫正线段的数量小于或等于所述数量阈值，或所述待矫正线段的长度小于或等于所述长度阈值，根据第一场景的场景特征确定是否对所述待矫正线段进行矫正，所述场景特征包含建筑特征和人物特征。
  26. 根据权利要求14-25中任一项所述的装置，其特征在于，所述矫正模块对所述待矫正线段以及所述待处理图像的特征区域进行矫正，得到处理后图像时，具体用于：
    构建直线约束对所述待矫正线段进行矫正；构建基于内容的单应性约束和形状和正则约束对背景区域进行矫正；构建人像约束对人像区域进行矫正；构建边界约束对边缘区域进行矫正，以得到所述处理后图像。
  27. 一种拍摄设备，其特征在于，所述拍摄设备包括：至少一个处理器、存储器、摄像头和惯性测量组件IMU，其中，所述摄像头用于拍摄图像，所述IMU用于获取所述拍摄设备的IMU信息，所述存储器用于存储计算机程序和指令，所述处理器用于调用所述计算机程序和指令，与所述摄像头和所述IMU协助执行如权利要求1-13中任一项所述的方法。
  28. 一种图像处理系统，其特征在于，所述图像处理系统包括至少一个处理器和存储器，所述存储器用于存储计算机程序和指令，所述处理器用于调用所述计算机程序和指令，实现执行如权利要求1-13中任一项所述的方法。
PCT/CN2023/079116 2022-03-02 2023-03-01 图像处理方法、装置及设备 WO2023165535A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210204484.6 2022-03-02
CN202210204484.6A CN116739908A (zh) 2022-03-02 2022-03-02 图像处理方法、装置及设备

Publications (1)

Publication Number Publication Date
WO2023165535A1 true WO2023165535A1 (zh) 2023-09-07

Family

ID=87883073

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/079116 WO2023165535A1 (zh) 2022-03-02 2023-03-01 图像处理方法、装置及设备

Country Status (2)

Country Link
CN (1) CN116739908A (zh)
WO (1) WO2023165535A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117853382B (zh) * 2024-03-04 2024-05-28 武汉人工智能研究院 基于稀疏标记物的图像矫正方法、装置及存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150103172A1 (en) * 2013-10-11 2015-04-16 Fujitsu Limited Image processing apparatus and method
CN110084236A (zh) * 2019-04-29 2019-08-02 北京朗镜科技有限责任公司 一种图像的矫正方法及装置
CN110460769A (zh) * 2019-07-05 2019-11-15 浙江大华技术股份有限公司 图像矫正方法、装置、计算机设备和存储介质
CN113850709A (zh) * 2020-06-28 2021-12-28 华为技术有限公司 图像变换方法和装置
CN113850726A (zh) * 2020-06-28 2021-12-28 华为技术有限公司 图像变换方法和装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150103172A1 (en) * 2013-10-11 2015-04-16 Fujitsu Limited Image processing apparatus and method
CN110084236A (zh) * 2019-04-29 2019-08-02 北京朗镜科技有限责任公司 一种图像的矫正方法及装置
CN110460769A (zh) * 2019-07-05 2019-11-15 浙江大华技术股份有限公司 图像矫正方法、装置、计算机设备和存储介质
CN113850709A (zh) * 2020-06-28 2021-12-28 华为技术有限公司 图像变换方法和装置
CN113850726A (zh) * 2020-06-28 2021-12-28 华为技术有限公司 图像变换方法和装置

Also Published As

Publication number Publication date
CN116739908A (zh) 2023-09-12

Similar Documents

Publication Publication Date Title
US9860448B2 (en) Method and electronic device for stabilizing video
US9819870B2 (en) Photographing method and electronic device
US10250800B2 (en) Computing device having an interactive method for sharing events
US9036943B1 (en) Cloud-based image improvement
WO2020019873A1 (zh) 图像处理方法、装置、终端及计算机可读存储介质
US11134191B2 (en) Image display method and electronic device
US11870951B2 (en) Photographing method and terminal
US9742995B2 (en) Receiver-controlled panoramic view video share
US20150103193A1 (en) Method and apparatus for long term image exposure with image stabilization on a mobile device
US10148875B1 (en) Method and system for interfacing multiple channels of panoramic videos with a high-definition port of a processor
US9398217B2 (en) Video stabilization using padded margin pixels
KR20150005270A (ko) 전자 장치에 의해 촬영된 이미지들을 프리뷰하는 방법 및 이를 위한 전자 장치
CN111064895B (zh) 一种虚化拍摄方法和电子设备
WO2024001506A1 (zh) 显示方法与电子设备
WO2022022633A1 (zh) 图像显示方法、装置和电子设备
WO2023165535A1 (zh) 图像处理方法、装置及设备
US10911677B1 (en) Multi-camera video stabilization techniques
WO2022227752A1 (zh) 拍照方法及装置
EP4274224A1 (en) Multi-scene video recording method and apparatus, and electronic device
US20230353864A1 (en) Photographing method and apparatus for intelligent framing recommendation
WO2021103919A1 (zh) 构图推荐方法和电子设备
WO2023005355A1 (zh) 图像防抖方法与电子设备
US10051192B1 (en) System and apparatus for adjusting luminance levels of multiple channels of panoramic video signals
US10148874B1 (en) Method and system for generating panoramic photographs and videos
WO2021238317A1 (zh) 一种全景拍摄方法及设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23762933

Country of ref document: EP

Kind code of ref document: A1