WO2017206656A1

WO2017206656A1 - Image processing method, terminal, and computer storage medium

Info

Publication number: WO2017206656A1
Application number: PCT/CN2017/082941
Authority: WO
Inventors: 戴向东
Original assignee: 努比亚技术有限公司
Priority date: 2016-05-31
Filing date: 2017-05-03
Publication date: 2017-12-07

Abstract

Disclosed are an image processing method and a terminal, comprising: obtaining an image data stream using an electronic aperture, the image data stream comprising multiple frames of image data; registering the respective frames of image data in the image data stream; and fusing the registered respective frames of image data to obtain a target image. By means of adding a handheld mode of the electronic aperture, embodiments of the present invention increase user convenience in photographing, guarantee the photographing effect, and improve user photographing experience.

Description

Image processing method and terminal, computer storage medium

Cross-reference to related applications

This application is based on a Chinese patent application with the application number 201610375523.3, the application date is May 31, 2016, and the Chinese patent application with the application number 201610377764.1, the application date is May 31, 2016, and requires the Chinese patent application. Priority is hereby incorporated by reference in its entirety in its entirety in its entirety in its entirety in its entirety in its entirety in

Technical field

The present invention relates to image processing technology in the field of photographing, and in particular to an image processing method and terminal, and a computer storage medium.

Background technique

The camera function is one of the commonly used functions of the mobile terminal, and the camera function of some mobile terminals has an electronic aperture mode. In the electronic aperture mode, after the user adjusts the aperture value, the mobile terminal performs continuous and uninterrupted shooting during the exposure time, and then transparently processes the captured multiple images before superimposing the overall effect and the "slow shutter" The actual effect is very consistent, mainly highlighting the long exposure time and even reaching the B-level exposure time. The algorithm used in the electronic aperture mode does not cause overexposure, and the overall effect of the picture is relatively natural.

When taking pictures in the electronic aperture mode, since multiple images need to be continuously captured for image fusion, each image needs to be aligned, and the mobile terminal cannot be shaken, otherwise the image pixels will be blurred. Based on this, it is usually necessary to cooperate with a tripod. Shooting in the electronic iris mode, but this limits the ease of use and user experience of the electronic iris mode. In addition, black spots are formed at the boundary of the image due to image alignment, and the pixels at these black edges are merged in a weighted average manner to form a difference in brightness, which affects the overall vision of the image.

Summary of the invention

To solve the above technical problem, an embodiment of the present invention provides an image processing method, a terminal, and a computer storage medium.

The terminal provided by the embodiment of the present invention includes:

An acquiring unit configured to acquire an image data stream by using an electronic aperture, the image data stream comprising a plurality of frames of image data;

a registration unit configured to register image data of each frame in the image data stream;

The fusion unit is configured to perform fusion processing on the image data of each frame after registration to obtain a target image.

In the embodiment of the present invention, the acquiring unit includes:

a shooting subunit configured to capture an original image data stream by using an electronic aperture in a handheld mode, the original image data stream comprising a plurality of frames of raw image data;

a pre-processing unit configured to pre-process each frame of the original image data stream to obtain the image data stream; wherein the pre-processing comprises at least one of: image filtering, contrast stretching .

In the embodiment of the present invention, the terminal further includes:

The prompting unit is configured to display a prompt box on the display interface, and the prompting box is configured to prompt the orientation of the handheld shaking when the electronic aperture is used in the handheld mode.

In the embodiment of the present invention, the registration unit is further configured to: use the first frame image data in the image data stream as a reference frame, and divide each of the image data streams except the first frame image data. The frame image data is aligned with the reference frame; wherein the alignment refers to aligning pixel points of the same spatial location.

An image processing method provided by an embodiment of the present invention includes:

Acquiring an image data stream using an electronic aperture, the image data stream comprising a plurality of frames of image data;

Registering each frame of image data in the image data stream;

The image data of each frame after registration is subjected to fusion processing to obtain a target image.

In the embodiment of the present invention, the acquiring an image data stream by using an electronic aperture includes:

In the handheld mode, an original image data stream is obtained by taking an electronic aperture, the original image data stream comprising a plurality of frames of original image data;

Performing pre-processing on each frame of the original image data in the original image data stream to obtain the image data stream;

Wherein, the preprocessing comprises at least one of the following: image filtering, contrast stretching.

In the embodiment of the present invention, the method further includes:

A prompt box is displayed on the display interface; the prompt box is used to indicate the orientation of the handheld shake when the electronic aperture is used in the handheld mode.

In the embodiment of the present invention, the registering each frame image data in the image data stream includes:

Aligning, by using the first frame image data in the image data stream as a reference frame, each frame image data of the image data stream except the first frame image data, and the reference frame;

Wherein, the alignment refers to aligning pixel points of the same spatial position.

In the embodiment of the present invention, the image processing of each frame image after the registration is performed, including:

Each pixel point of the image data of each frame after registration is superimposed according to the spatial position correspondence.

A terminal provided by another embodiment of the present invention includes:

a registration unit, configured to determine a reference frame from the image data stream, and register, in the image data stream, image data of each frame other than the reference frame with the reference frame;

The merging unit is configured to perform merging processing on the image data of each frame after the registration, and replace the pixels in the black area at the image boundary with the pixels corresponding to the reference frame, and perform fusion processing to obtain a target image.

In the embodiment of the present invention, the fusion unit includes:

The analysis subunit is configured to analyze the image data of each frame after registration to determine a black area at a boundary of each frame image;

The replacement and fusion subunits are configured to perform fusion processing on the image data of each frame after the registration, and replace the pixels in the black area at the image boundary with the pixels corresponding to the reference frame, and perform fusion processing to obtain a target image.

In the embodiment of the present invention, the replacing and merging subunit is further configured to determine, according to a black area at the image boundary, a reference area corresponding to the black area in the reference frame; The pixel of the region is replaced with the pixel in the reference region in the reference frame, and then the fusion processing is performed to obtain the target image.

An image processing method according to another embodiment of the present invention includes:

Determining a reference frame from the image data stream, and registering each frame image data of the image data stream except the reference frame with the reference frame;

When the image data of each frame after the registration is subjected to the fusion processing, the pixels of the black area at the image boundary are replaced with the pixels corresponding to the reference frame, and then the fusion processing is performed to obtain the target image.

In the embodiment of the present invention, when performing the merging process on the image data of each frame after the registration, the pixel of the black area at the image boundary is replaced with the pixel corresponding to the reference frame, and then the fusion process is performed to obtain the target image, including:

Performing analysis on the image data of each frame after registration to determine a black area at the boundary of each frame image;

In the embodiment of the present invention, after the pixel of the black area at the image boundary is replaced with the pixel corresponding to the reference frame, the fusion process is performed to obtain the target image, including:

Determining a reference area corresponding to the black area in the reference frame according to a black area at the boundary of the image;

The pixel in the black area at the image boundary is replaced with the pixel in the reference area in the reference frame, and then the fusion processing is performed to obtain the target image.

In the embodiment of the present invention, the determining a reference frame from the image data stream, and registering each frame image data of the image data stream except the reference frame with the reference frame, includes:

The terminal provided by the embodiment of the present invention includes: a camera and a processor;

The camera is configured to acquire an image data stream using an electronic aperture, the image data stream comprising a plurality of frames of image data;

The processor is configured to register image data of each frame in the image data stream, and perform fusion processing on the image data of each frame after registration to obtain a target image.

The processor, configured to determine a reference frame from the image data stream, the image And storing, in the data stream, image data of each frame other than the reference frame and the reference frame; and performing fusion processing on the image data of each frame after registration, replacing pixels of the black area at the boundary of the image with the reference The pixel corresponding to the frame is subjected to fusion processing to obtain a target image.

The computer storage medium provided by the embodiment of the present invention stores computer executable instructions configured to perform any of the image processing methods described.

In the technical solution of the embodiment of the present invention, in the handheld mode, an image data stream is acquired by using an electronic aperture, the image data stream includes multi-frame image data; and each frame image data in the image data stream is registered; The image data of each frame after registration is subjected to fusion processing to obtain a target image. It can be seen that the embodiment of the present invention increases the hand-held mode of the electronic aperture, improves the convenience of the user's shooting, and introduces the image registration in the embodiment of the present invention to ensure the shooting effect and improve the user's shooting experience. . In addition, the embodiment of the present invention further adds a prompt box on the display interface to prevent the handheld terminal from being over-ranged when the user shoots.

In the technical solution of the embodiment of the present invention, in the handheld mode, an image data stream is acquired by using an electronic aperture, the image data stream includes multi-frame image data; a reference frame is determined from the image data stream, and the image data is determined. And storing, in the stream, image data of each frame other than the reference frame and the reference frame; and performing fusion processing on the image data of each frame after the registration, replacing pixels of the black area at the boundary of the image with the reference frame The corresponding pixel is subjected to fusion processing to obtain a target image. Through the implementation of the technical solution of the embodiment of the present invention, the user can perform the shooting of the electronic aperture by the hand-held terminal, thereby improving the convenience of the user's shooting, avoiding the problem of unclear image caused by the hand-held, ensuring the effect of shooting, and improving the effect of the shooting. User shooting experience. When image fusion is performed, the black side of the pixel caused by image registration is processed, which ensures that the pixel point transition of each position of the entire image is natural.

DRAWINGS

1 is a schematic structural diagram of hardware of an optional mobile terminal embodying various embodiments of the present invention;

2 is a schematic diagram of a wireless communication system of the mobile terminal shown in FIG. 1;

3 is a schematic flowchart of an image processing method according to Embodiment 1 of the present invention;

4 is a schematic diagram of pixel point matching of an optical flow image;

Figure 5 is a schematic diagram of a simple mobile phone motion model;

FIG. 6 is a flowchart of aligning a multi-frame image by using an optical flow according to an embodiment of the present invention; FIG.

FIG. 7 is a schematic flowchart diagram of an image processing method according to Embodiment 2 of the present invention; FIG.

FIG. 8 is a schematic diagram of an interface of a prompt box according to an embodiment of the present invention; FIG.

9 is a schematic structural diagram of a terminal according to Embodiment 1 of the present invention;

FIG. 10 is a schematic structural diagram of a terminal according to an embodiment of the present invention;

11 is a schematic flowchart of an image processing method according to Embodiment 3 of the present invention;

FIG. 12 is a schematic flowchart diagram of an image processing method according to Embodiment 4 of the present invention; FIG.

FIG. 13 is a schematic diagram of image fusion according to an embodiment of the present invention; FIG.

14 is a schematic structural diagram of a terminal according to Embodiment 2 of the present invention;

Figure 15 is a block diagram of the electrical structure of the camera.

detailed description

A mobile terminal embodying various embodiments of the present invention will now be described with reference to the accompanying drawings. In the following description, the suffixes such as "module," "component," or "unit" used to denote an element are merely illustrative of the embodiments of the present invention, and do not have a specific meaning per se. Therefore, "module" and "component" can be used in combination.

The mobile terminal can be implemented in various forms. For example, the terminal described in the embodiments of the present invention may include, for example, a mobile phone, a smart phone, a notebook computer, a digital broadcast receiver, a personal digital assistant (PDA, Personal Digital Assistant), a tablet (PAD), a portable multimedia player (PMP). , Portable Media Player), mobile devices of navigation devices and the like, and fixed terminals such as digital TVs, desktop computers, and the like. In the following, it is assumed that the terminal is a mobile terminal. However, those skilled in the art will appreciate that configurations in accordance with embodiments of the present invention can be applied to fixed type terminals in addition to components that are specifically for mobile purposes.

FIG. 1 is a schematic diagram showing the hardware structure of a mobile terminal embodying various embodiments of the present invention.

The mobile terminal 100 may include a wireless communication unit 110, an audio/video (A/V) input unit 120, a user input unit 130, a sensing unit 140, an output unit 150, a memory 160, an interface unit 170, a controller 180, and a power supply unit 190. and many more. Figure 1 illustrates a mobile terminal having various components, but it should be understood that not all illustrated components are required to be implemented. More or fewer components can be implemented instead. The elements of the mobile terminal will be described in detail below.

Wireless communication unit 110 typically includes one or more components that permit radio communication between mobile terminal 100 and a wireless communication system or network. For example, the wireless communication unit may include at least one of a broadcast receiving module 111, a mobile communication module 112, a wireless internet module 113, a short-range communication module 114, and a location information module 115.

The broadcast receiving module 111 receives a broadcast signal and/or broadcast associated information from an external broadcast management server via a broadcast channel. The broadcast channel can include a satellite channel and/or a terrestrial channel. The broadcast management server may be a server that generates and transmits a broadcast signal and/or broadcast associated information or a server that receives a previously generated broadcast signal and/or broadcast associated information and transmits it to the terminal. The broadcast signal may include a TV broadcast signal, a radio broadcast signal, a data broadcast signal, and the like. Moreover, the broadcast signal may further include a broadcast signal combined with a TV or radio broadcast signal. The broadcast associated information may also be provided via a mobile communication network, and in this case, the broadcast associated information may be received by the mobile communication module 112. The broadcast signal may exist in various forms, for example, it may be an Electronic Program Guide (EPG) of Digital Multimedia Broadcasting (DMB), a digital video broadcast handheld (DVB-H, Digital Video Broadcasting-Handheld). ) exists in the form of an ESG (Electronic Service Guide) and the like. The broadcast receiving module 111 can receive a signal broadcast by using various types of broadcast systems. In particular, the broadcast receiving module 111 can use, for example, DMB-T (Digital Multimedia Broadcasting-Terrestrial), Digital Multimedia Broadcasting-Satellite (DMB-S, Digital Multimedia Broadcasting-Satellite), Digital Video Broadcasting Handheld (DVB-H), Forward Link Media (MediaFLO, Media Forward Link Only) data broadcasting system, Digital Broadcasting Integrated Services (ISDB-T, Integrated Services Digital Broadcasting-Terrestrial), etc. The system receives digital broadcasts. The broadcast receiving module 111 can be constructed as various broadcast systems suitable for providing broadcast signals as well as the above-described digital broadcast system. The broadcast signal and/or broadcast associated information received via the broadcast receiving module 111 may be stored in the memory 160 (or other type of storage medium).

The mobile communication module 112 transmits the radio signals to and/or receives radio signals from at least one of a base station (e.g., an access point, a Node B, etc.), an external terminal, and a server. Such radio signals may include voice call signals, video call signals, or various types of data transmitted and/or received in accordance with text and/or multimedia messages.

The wireless internet module 113 supports wireless internet access of the mobile terminal. The module can be internally or externally coupled to the terminal. The wireless Internet access technologies involved in the module may include Wi-Fi (WLAN, Wireless Local Area Networks), Wireless Broadband (Wibro), Worldwide Interoperability for Microwave Access (Wimax), and High Speed Downlink Packet Access. (HSDPA, High Speed Downlink Packet Access) and so on.

The short range communication module 114 is a module for supporting short range communication. Some examples of short-range communication technologies include Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), Zigbee, and the like.

The location information module 115 is a module for checking or acquiring location information of the mobile terminal. A typical example of a location information module is the Global Positioning System (GPS). According to the current technology, the position information module 115 as a GPS calculates distance information and accurate time information from three or more satellites and applies triangulation to the calculated information to accurately calculate the three-dimensional current based on longitude, latitude, and altitude. location information. Currently, the method used to calculate position and time information uses three satellites and is corrected by using another satellite. The error of the position and time information. Further, the position information module 115 as a GPS can calculate the speed information by continuously calculating the current position information in real time.

The A/V input unit 120 is for receiving an audio or video signal. The A/V input unit 120 may include a camera 121 and a microphone 122 that processes image data of still pictures or video obtained by the image capturing device in a video capturing mode or an image capturing mode. The processed image frame can be displayed on the display unit 151. The image frames processed by the camera 121 may be stored in the memory 160 (or other storage medium) or transmitted via the wireless communication unit 110, and two or more cameras 121 may be provided according to the configuration of the mobile terminal. The microphone 122 can receive sound (audio data) via a microphone in an operation mode of a telephone call mode, a recording mode, a voice recognition mode, and the like, and can process such sound as audio data. The processed audio (voice) data can be converted to a format output that can be transmitted to the mobile communication base station via the mobile communication module 112 in the case of a telephone call mode. The microphone 122 can implement various types of noise cancellation (or suppression) algorithms to cancel (or suppress) noise or interference generated during the process of receiving and transmitting audio signals.

The user input unit 130 may generate key input data according to a command input by the user to control various operations of the mobile terminal. The user input unit 130 allows the user to input various types of information, and may include a keyboard, a pot, a touch pad (eg, a touch sensitive component that detects changes in resistance, pressure, capacitance, etc. due to contact), a scroll wheel , rocker, etc. In particular, when the touch panel is superimposed on the display unit 151 in the form of a layer, a touch screen can be formed.

The sensing unit 140 detects the current state of the mobile terminal 100 (eg, the open or closed state of the mobile terminal 100), the location of the mobile terminal 100, the presence or absence of contact (ie, touch input) by the user with the mobile terminal 100, and the mobile terminal. The orientation of 100, the acceleration or deceleration movement and direction of the mobile terminal 100, and the like, and generates a command or signal for controlling the operation of the mobile terminal 100. For example, when the mobile terminal 100 is implemented as a slide type mobile phone, the sensing unit 140 can sense whether the slide type phone is turned on or off. In addition, the sensing unit 140 can detect the power supply unit 190 Whether power is supplied or whether the interface unit 170 is coupled to an external device. The sensing unit 140 may include a proximity sensor 141.

The interface unit 170 serves as an interface through which at least one external device can connect with the mobile terminal 100. For example, the external device may include a wired or wireless headset port, an external power (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device having an identification module, and an audio input/output. (I/O) port, video I/O port, headphone port, and more. The identification module may be stored to verify various information used by the user using the mobile terminal 100 and may include a User Identification Module (UIM), a Subscriber Identity Module (SIM), and a Universal Customer Identification Module (USIM, Universal). Subscriber Identity Module) and more. In addition, the device having the identification module (hereinafter referred to as "identification device") may take the form of a smart card, and thus the identification device may be connected to the mobile terminal 100 via a port or other connection device. The interface unit 170 can be configured to receive input from an external device (eg, data information, power, etc.) and transmit the received input to one or more components within the mobile terminal 100 or can be used at the mobile terminal and external device Transfer data between.

In addition, when the mobile terminal 100 is connected to the external base, the interface unit 170 may function as a path through which power is supplied from the base to the mobile terminal 100 or may be used as a transmission of various command signals allowing input from the base to the mobile terminal 100 The path to the terminal. Various command signals or power input from the base can be used as signals for identifying whether the mobile terminal is accurately mounted on the base. Output unit 150 is configured to provide an output signal (eg, an audio signal, a video signal, an alarm signal, a vibration signal, etc.) in a visual, audio, and/or tactile manner. The output unit 150 may include a display unit 151, an audio output module 152, an alarm unit 153, and the like.

The display unit 151 can display information processed in the mobile terminal 100. For example, when the mobile terminal 100 is in the phone call mode, the display unit 151 can display a user interface related to a call or other communication (eg, text messaging, multimedia file download, etc.) (UI, User) Interface) or Graphical User Interface (GUI). When the mobile terminal 100 is in a video call mode or an image capturing mode, the display unit 151 may display a captured image and/or a received image, a UI or GUI showing a video or image and related functions, and the like.

Meanwhile, when the display unit 151 and the touch panel are superposed on each other in the form of a layer to form a touch screen, the display unit 151 can function as an input device and an output device. The display unit 151 may include a Liquid Crystal Display (LCD), a Thin Film Transistor (LCD), an Organic Light-Emitting Diode (OLED) display, a flexible display, and a three-dimensional (3D) At least one of a display or the like. Some of these displays may be configured to be transparent to allow a user to view from the outside, which may be referred to as a transparent display, and a typical transparent display may be, for example, a transparent organic light emitting diode (TOLED) display or the like. According to a particular desired embodiment, the mobile terminal 100 may include two or more display units (or other display devices), for example, the mobile terminal may include an external display unit (not shown) and an internal display unit (not shown) . The touch screen can be used to detect touch input pressure as well as touch input position and touch input area.

The audio output module 152 may convert audio data received by the wireless communication unit 110 or stored in the memory 160 when the mobile terminal is in a call signal receiving mode, a call mode, a recording mode, a voice recognition mode, a broadcast receiving mode, and the like. The audio signal is output as sound. Moreover, the audio output module 152 can provide audio output (eg, call signal reception sound, message reception sound, etc.) associated with a particular function performed by the mobile terminal 100. The audio output module 152 can include a speaker, a buzzer, and the like.

The alarm unit 153 can provide an output to notify the mobile terminal 100 of the occurrence of an event. Typical events may include call reception, message reception, key signal input, touch input, and the like. In addition to audio or video output, the alert unit 153 can provide an output in a different manner to notify of the occurrence of an event. For example, the alarm unit 153 can provide an output in the form of vibrations, and when a call, message, or some other incoming communication is received, the alarm unit 153 A tactile output (ie, vibration) can be provided to notify the user of it. By providing such a tactile output, the user is able to recognize the occurrence of various events even when the user's mobile phone is in the user's pocket. The alarm unit 153 can also provide an output of the notification event occurrence via the display unit 151 or the audio output module 152.

The memory 160 may store a software program or the like for processing and control operations performed by the controller 180, or may temporarily store data (for example, a phone book, a message, a still image, a video, etc.) that has been output or is to be output. Moreover, the memory 160 can store data regarding vibrations and audio signals of various manners that are output when a touch is applied to the touch screen.

The memory 160 may include at least one type of storage medium including a flash memory, a hard disk, a multimedia card, a card type memory (for example, SD or DX memory, etc.), a random access memory (RAM), and a static memory. Random Access Memory (SRAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Programmable Read Only Memory (PROM) , Programmable Read Only Memory), magnetic memory, disk, optical disk, etc. Moreover, the mobile terminal 100 can cooperate with a network storage device that performs a storage function of the memory 160 through a network connection.

The controller 180 typically controls the overall operation of the mobile terminal. For example, the controller 180 performs the control and processing associated with voice calls, data communications, video calls, and the like. In addition, the controller 180 may include a multimedia module 181 for reproducing (or playing back) multimedia data, which may be constructed within the controller 180 or may be configured to be separate from the controller 180. The controller 180 may perform a pattern recognition process to recognize a handwriting input or a picture drawing input performed on the touch screen as a character or an image.

The power supply unit 190 receives external power or internal power under the control of the controller 180 and provides appropriate power required to operate the various components and components.

The various embodiments described herein can be used, for example, in computer software, hardware, or any of them. The combined computer readable medium is implemented. For hardware implementation, the embodiments described herein may use an Application Specific Integrated Circuit (ASIC), a Digital Signal Processing (DSP), a Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), processor, controller, microcontroller, microprocessor, electronics designed to perform the functions described herein At least one of the units is implemented, and in some cases, such an implementation may be implemented in controller 180. For software implementations, implementations such as procedures or functions may be implemented with separate software modules that permit the execution of at least one function or operation. The software code can be implemented by a software application (or program) written in any suitable programming language, which can be stored in memory 160 and executed by controller 180.

So far, the mobile terminal has been described in terms of its function. Hereinafter, for the sake of brevity, a slide type mobile terminal among various types of mobile terminals such as a folding type, a bar type, a swing type, a slide type mobile terminal, and the like will be described as an example. Therefore, the present invention can be applied to any type of mobile terminal, and is not limited to a slide type mobile terminal.

The mobile terminal 100 as shown in FIG. 1 may be configured to operate using a communication system such as a wired and wireless communication system and a satellite-based communication system that transmits data via frames or packets.

A communication system in which a mobile terminal is operable according to an embodiment of the present invention will now be described with reference to FIG.

Such communication systems may use different air interfaces and/or physical layers. For example, the air interface used by the communication system includes, for example, Frequency Division Multiple Access (FDMA), Time Division Multiple Access (TDMA), Code Division Multiple Access (CDMA), and General Purpose Code Division Multiple Access (CDMA). UMTS (Universal Mobile Telecommunications System) (particularly, Long Term Evolution (LTE), Global System for Mobile Communications (GSM), and the like. As By way of non-limiting example, the following description relates to CDMA communication systems, but such teachings are equally applicable to other types of systems.

Referring to FIG. 2, the CDMA wireless communication system may include a plurality of mobile terminals 100, a plurality of base stations (BS) 270, a base station controller (BSC) 275, and a mobile switching center (MSC) 280. . The MSC 280 is configured to interface with a Public Switched Telephone Network (PSTN) 290. The MSC 280 is also configured to interface with a BSC 275 that can be coupled to the base station 270 via a backhaul line. The backhaul line can be constructed in accordance with any of a number of well known interfaces including, for example, E1/T1, ATM, IP, PPP, Frame Relay, HDSL, ADSL, or xDSL. It will be appreciated that the system as shown in FIG. 2 can include multiple BSCs 275.

Each BS 270 can serve one or more partitions (or regions), with each partition covered by a multi-directional antenna or an antenna pointing in a particular direction radially away from the BS 270. Alternatively, each partition may be covered by two or more antennas for diversity reception. Each BS 270 can be configured to support multiple frequency allocations, and each frequency allocation has a particular frequency spectrum (eg, 1.25 MHz, 5 MHz, etc.).

The intersection of partitioning and frequency allocation can be referred to as a CDMA channel. BS 270 may also be referred to as a Base Transceiver Subsystem (BTS) or other equivalent terminology. In such a case, the term "base station" can be used to generally refer to a single BSC 275 and at least one BS 270. A base station can also be referred to as a "cell station." Alternatively, each partition of a particular BS 270 may be referred to as multiple cellular stations.

As shown in FIG. 2, a broadcast transmitter (BT, Broadcast Transmitter) 295 transmits a broadcast signal to the mobile terminal 100 operating within the system. A broadcast receiving module 111 as shown in FIG. 1 is provided at the mobile terminal 100 to receive a broadcast signal transmitted by the BT 295. In Figure 2, several satellites 300 are shown, for example GPS satellites 300 may be employed. The satellite 300 helps locate at least one of the plurality of mobile terminals 100.

In Figure 2, a plurality of satellites 300 are depicted, but it is understood that useful positioning information can be obtained using any number of satellites. The position information module 115 as a GPS as shown in FIG. 1 is generally configured to cooperate with the satellite 300 to obtain desired positioning information. Instead of GPS tracking technology or in addition to GPS tracking technology, other techniques that can track the location of the mobile terminal can be used. Additionally, at least one GPS satellite 300 can selectively or additionally process satellite DMB transmissions.

As a typical operation of a wireless communication system, BS 270 receives reverse link signals from various mobile terminals 100. Mobile terminal 100 typically participates in calls, messaging, and other types of communications. Each reverse link signal received by a particular base station 270 is processed within a particular BS 270. The obtained data is forwarded to the relevant BSC 275. The BSC provides call resource allocation and coordinated mobility management functions including a soft handoff procedure between the BSs 270. The BSC 275 also routes the received data to the MSC 280, which provides additional routing services for interfacing with the PSTN 290. Similarly, PSTN 290 interfaces with MSC 280, which forms an interface with BSC 275, and BSC 275 controls BS 270 accordingly to transmit forward link signals to mobile terminal 100.

The mobile communication module 112 of the wireless communication unit 110 in the mobile terminal accesses the mobile based on necessary data (including user identification information and authentication information) of the mobile communication network (such as 2G/3G/4G mobile communication network) built in the mobile terminal. The communication network transmits mobile communication data (including uplink mobile communication data and downlink mobile communication data) for services such as web browsing and network multimedia playback of the mobile terminal user.

The wireless internet module 113 of the wireless communication unit 110 implements a function of a wireless hotspot by operating a related protocol function of a wireless hotspot, and the wireless hotspot supports access of a plurality of mobile terminals (any mobile terminal other than the mobile terminal) by multiplexing the mobile communication module. The mobile communication connection between the mobile communication network and the mobile communication network transmits mobile communication data (including uplink mobile communication data and downlink mobile communication data) for the mobile terminal user's web browsing, network multimedia playback, etc., since the mobile terminal is substantially complex Transmitting a mobile communication connection between a mobile terminal and a communication network The communication data is transmitted, so that the traffic of the mobile communication data consumed by the mobile terminal is included in the communication fee of the mobile terminal by the charging entity on the communication network side, thereby consuming the data traffic of the mobile communication data included in the communication tariff used by the mobile terminal.

Based on the above-described hardware structure of the mobile terminal 100 and the communication system, various embodiments of the method of the present invention are proposed.

3 is a schematic flowchart of an image processing method according to Embodiment 1 of the present invention. The image processing method in this example is applied to a terminal. As shown in FIG. 3, the image processing method includes the following steps:

Step 301: Acquire an image data stream by using an electronic aperture, the image data stream comprising a plurality of frames of image data.

In the embodiment of the present invention, the terminal may be an electronic device such as a mobile phone or a tablet computer. The terminal has a photographing function, and the photographing function of the terminal has an electronic aperture mode; when photographing by using an electronic aperture, the user needs to set the photographing function to the electronic aperture mode.

In the electronic aperture mode, after the user adjusts the aperture value, the terminal performs continuous and uninterrupted shooting during the exposure time, and then fuses the captured multiple images. When taking pictures with an electronic aperture, since it is necessary to continuously take multiple images for fusion processing, each image needs to be aligned. In order to ensure the ease of use and user experience of the electronic aperture, the embodiment of the present invention adds a handheld mode for the electronic aperture shooting. In the gesture mode, the user can conveniently use the electronic aperture to shoot.

At the time of shooting, an image data stream is first acquired, the image data stream comprising a plurality of frames of image data. Specifically, the captured original image data stream is first acquired synchronously; then the original image data stream is subjected to data reading and image preprocessing to obtain the image data stream. Here, due to the image signal processing (ISP, Image Signal Processing) process of the camera and the unpredictable transformation of the external environment, the raw image data of each frame in the original image data stream acquired synchronously is in illumination, noise, sharpness, and focus point. There is a difference. Before performing the fusion processing, it is necessary to preprocess the original image data of each frame in the original image data stream through a necessary preprocessing process. Here, the preprocessing The process includes image filtering to eliminate noise, contrast stretching to improve image sharpness and image illumination differences. After the pre-processing is performed, the difference of the image data of each frame in the image data stream will be reduced, which is helpful for the improvement of the effect of the subsequent image registration algorithm.

Step 302: Register each frame image data in the image data stream.

In the embodiment of the present invention, after the image data of each frame in the image data stream is registered, the pixels in the same spatial position in each frame image data are aligned, thereby avoiding blurring when the image is merged.

In the embodiment of the present invention, registration is also called alignment, and there are many algorithms for image alignment, which are mainly divided into a local feature based method and a global feature based method. Among them, the typical method based on local features is to extract the key feature points of the image, and then use these key feature points to calculate the mapping matrix of the image space alignment model, and finally use the mapping matrix to perform image alignment. The registration effect of this type of method can generally meet the requirements of many scenes, such as changes in illumination (combination of different exposure images), large-scale image shift (panoramic image stitching), dark light image (noise increase) and other complexities. Scene. The other type is a search alignment method based on global mutual information matching, which can reduce the matching error caused by random feature points.

The optical flow field is also a point-based matching algorithm, which is the instantaneous velocity of the pixel motion of the space moving object on the observation imaging plane, which is the change of the pixel in the time domain and the correlation between adjacent frames. A method for finding the correspondence between the previous frame and the current frame, thereby calculating the motion information of the object between adjacent frames. The purpose of studying the optical flow field is to approximate the motion field that cannot be directly obtained from the sequence of pictures. Here, the motion field is actually the motion of the object in the three-dimensional real world; the optical flow field is the projection of the motion field on the two-dimensional image plane (human eye or camera).

Through a sequence of pictures, the motion velocity and direction of motion of each pixel in each image are found to be the optical flow field. As shown in Figure 4, the position of point A in the T frame is (x1, y1), then we find point A in the T+1 frame, if its position is (x2, y2), then we Just The motion vector of point A can be determined: V = (x2, y2) - (x1, y1).

How to find the position of point A when the t+1 frame is obtained by Lucas-Kanade optical flow method, the basic process is as follows:

Suppose that the color of an object does not have a large and significant change in the two frames before and after. Based on this idea, an image constraint equation can be obtained. Different optical flow algorithms solve the problem of optical flow that assumes different additional conditions. Using partial derivatives for spatial and temporal coordinates, the image constraint equation can be written as:

I(x,y,t)=I(x+dx,y+dy,t+dt) (1a)

Where I(x, y, t) is the pixel value of the image at the (x, y) position.

among them,

Assuming that the movement is small enough, then using the Taylor formula for the image constraint equation, you can get:

Among them, HOT refers to higher order, and HOT can be ignored when the movement is small enough. From equation (2a) you can get:

Vx=dx, Vx=dx (4a)

Where Vx and Vy are the compositions of x, y in the optical flow vector of I(x, y, t), respectively. Ix and Iy are the differences of the image in the corresponding direction at (x, y, t). So there are:

Ix*Vx+Iy*Vy=-It (6a)

There are 2 unknowns in the above equation, and at least two non-correlated equations are needed to solve. The Lucas-Kanade optical flow method assumes that the spatial pixel points move in unison, and adjacent points on one scene are projected onto the image as neighboring points, and the neighboring points are at the same speed. This is unique to the Lucas-Kanade optical flow method. Assumptions, because the optical flow method has only one basic equation constraint, and requires speed in the x, y direction, there are two unknown variables. Assuming similar motions in the neighborhood of the feature points, it is possible to find the velocity in the x, y direction by n multiple equations (n is the total number of points in the neighborhood of the feature points, including the feature points). You can get the following equation:

In order to solve this overdetermined problem, the least squares method is used:

Then the optical flow can be obtained adjacent to V:

The small motion hypothesis mentioned in the above scheme assumes that when the target speed is fast, the assumption will not be established, and multi-scale can solve this problem. First, create a Gaussian pyramid for each frame, with the largest scale image at the top and the original image at the bottom. Then, from the top level, estimate the position of the next frame as the initial position of the next layer, search down the pyramid, and repeat the estimation action until it reaches the bottom of the pyramid. This search can quickly locate the direction and position of the pixel.

Figure 6 shows the alignment process for multi-frame images using optical flow. Using the calculation method of the optical flow field, the sparse matching points between the two images can be obtained, and then the coordinates of the points are calculated. Like a mapping model. In the image alignment step, it is important to choose the correct image alignment transformation model. Common spatial transformation models include affine transformation and perspective transformation models.

The affine transformation can be visually represented in the following form. Any parallelogram in one plane can be mapped to another parallelogram by affine transformation. The mapping operation of the image is performed in the same spatial plane, and different transformation parameters are used to deform to obtain different types of parallelograms.

Transmission transformation is a more general transformation model. Compared with affine transformation, transmission transformation is more flexible. A transmission transformation can transform a rectangle into a trapezoid. It describes the projection of one plane in space into another spatial plane. Shot transformation is a special case of perspective transformation.

The meaning of each element of the above matrix is as follows:

a ₀₂ and a ₁₂ are displacement parameters;

a ₀₀ a ₀₁ and a ₁₀ a ₁₁ are scaling and rotation parameters;

a ₂₀ a ₂₁ is the amount of deformation in the horizontal and vertical directions.

Here we need to choose the perspective transformation model, mainly considering the handheld terminal. For example, when the mobile phone continuously captures multiple images, the jitter motion of the mobile phone is basically not in the same plane. The simple motion model is shown in Figure 5.

Step 303: Perform fusion processing on the image data of each frame after registration to obtain a target image.

In the embodiment of the present invention, after the image data of each frame is registered (that is, aligned), the image needs to be fused, and here, a fusion method in which image pixels are sequentially superimposed is adopted. Fusion processing according to formula (14a):

Where I is for each image, m is the mth image, k is the number of images that have been synthesized, and N is the total number of composite images.

According to the technical solution of the embodiment of the present invention, an image processing method is proposed, which uses image registration principle to perform image alignment on image data to be synthesized in multiple frames, and the alignment method allows image jitter error within a certain range, and the final composite effect of the image The pixel deviation that occurs is small.

FIG. 7 is a schematic flowchart of an image processing method according to Embodiment 2 of the present invention. The image processing method in this example is applied to a terminal. As shown in FIG. 7, the image processing method includes the following steps:

Step 701: In the handheld mode, the original image data stream is obtained by using an electronic aperture, the original image data stream includes a plurality of frames of original image data, and the original image data of each frame in the original image data stream is preprocessed to obtain The image data stream.

At the time of shooting, an image data stream is first acquired, the image data stream comprising a plurality of frames of image data. Specifically, the captured original image data stream is first acquired synchronously; then the original image data stream is subjected to data reading and image preprocessing to obtain the image data stream. Here, due to the ISP processing flow of the camera shooting and the unpredictable transformation of the external environment, the raw image data of each frame in the synchronously acquired original image data stream is different in illumination, noise, sharpness, and focus point. In progress Before the fusion processing, the original image data of each frame in the original image data stream needs to be preprocessed through a necessary preprocessing process. Here, the preprocessing process includes: image filtering to eliminate noise and contrast stretching to improve image clarity and The difference in illumination of the image. After the pre-processing is performed, the difference of the image data of each frame in the image data stream will be reduced, which is helpful for the improvement of the effect of the subsequent image registration algorithm.

Step 702: Align each frame image data of the image data stream except the first frame image data with the reference frame by using the first frame image data in the image data stream as a reference frame.

The optical flow field is also a point-based matching algorithm, which is the instantaneous velocity of the pixel motion of the space moving object on the observation imaging plane, which is the change of the pixel in the time domain and the correlation between adjacent frames. A method for finding the correspondence between the previous frame and the current frame, thereby calculating the motion information of the object between adjacent frames.

Step 703: Display a prompt box on the display interface; the prompt box is used to prompt the orientation of the handheld shake when the electronic aperture is used in the handheld mode.

Referring to FIG. 8, on the application interaction interface, a user prompt box is displayed, which prompts the user to manually hold the direction of the jitter, which can facilitate the user to correct in time.

Step 704: Superimpose each pixel point of each frame image data after registration according to spatial position correspondence to obtain a target image.

In the embodiment of the present invention, after the image data of each frame is registered (that is, aligned), the image needs to be fused, and here, a fusion method in which image pixels are sequentially superimposed is adopted.

According to the technical solution of the embodiment of the present invention, an image processing method is proposed, which uses image registration principle to perform image alignment on image data to be synthesized in multiple frames, and the alignment method allows image jitter error within a certain range, and the final composite effect of the image The pixel deviation that occurs is small. In addition, during the shooting of the electronic aperture, a prompt box is added to prevent the terminal from being over-ranged when shooting.

FIG. 9 is a schematic structural diagram of a terminal according to Embodiment 1 of the present invention. As shown in FIG. 9, the terminal includes:

The obtaining unit 91 is configured to acquire an image data stream by using an electronic aperture, the image data stream comprising multi-frame image data;

The registration unit 92 is configured to register image data of each frame in the image data stream;

The merging unit 93 is configured to perform merging processing on the image data of each frame after registration to obtain a target image.

The obtaining unit 91 includes:

The photographing subunit 911 is configured to obtain an original image data stream by using an electronic aperture in a handheld mode, the original image data stream comprising a plurality of frames of original image data;

The pre-processing sub-unit 912 is configured to pre-process the original image data of each frame in the original image data stream to obtain the image data stream; wherein the pre-processing includes at least one of the following: image filtering, contrast pull Stretch.

The terminal further includes:

The prompting unit 94 is configured to display a prompt box on the display interface; the prompt box is used for prompting In the hand-held mode, the orientation of the hand-held shake is taken when shooting with an electronic aperture.

The registration unit 92 is further configured to use the first frame image data in the image data stream as a reference frame, and image data of each frame except the first frame image data in the image data stream. The reference frames are aligned; wherein the alignment refers to aligning pixel points of the same spatial location.

The merging unit 93 is further configured to superimpose each pixel point of each frame image data after registration according to spatial position correspondence.

It should be noted here that the description of the above terminal embodiment is similar to the description of the above method embodiment, and has similar advantages as the method embodiment. For technical details not disclosed in the terminal embodiment of the present invention, please refer to the description of the method embodiment of the present invention.

10 is a schematic structural diagram of a terminal according to Embodiment 1 of the present invention. The terminal includes: a processor 1001, a camera 1002, and a display screen 1003. The processor 1001, the camera 1002, and the display screen 1003 are all connected by a bus 1004.

The camera 1002 is configured to acquire an image data stream by using an electronic aperture, the image data stream comprising a plurality of frames of image data;

The processor 1001 is configured to register image data of each frame in the image data stream, and perform fusion processing on the image data of each frame after registration to obtain a target image.

Specifically, the camera 1002 is configured to obtain an original image data stream by using an electronic aperture in a handheld mode, where the original image data stream includes multiple frames of original image data;

The processor 1001 is configured to perform pre-processing on each frame of the original image data stream to obtain the image data stream, where the pre-processing includes at least one of the following: image filtering, contrast pull And mapping the image data of each frame in the image data stream; and performing the fusion processing on the image data of each frame after registration to obtain a target image.

The display screen 1003 is configured to display a prompt box on the display interface; the prompt box is used to prompt the orientation of the handheld shake when the electronic aperture is used in the handheld mode.

11 is a schematic flowchart of an image processing method according to Embodiment 3 of the present invention. The image processing method in this example is applied to a terminal. As shown in FIG. 11, the image processing method includes the following steps:

Step 1101: Acquire an image data stream using an electronic aperture, the image data stream comprising a plurality of frames of image data.

At the time of shooting, an image data stream is first acquired, the image data stream comprising a plurality of frames of image data. Specifically, the captured original image data stream is first acquired synchronously; then the original image data stream is subjected to data reading and image preprocessing to obtain the image data stream. Here, due to the ISP process captured by the camera and the unpredictable transformation of the external environment, the raw image data of each frame in the synchronously acquired original image data stream is different in illumination, noise, sharpness, and focus point. Before the fusion processing, the original image data of each frame in the original image data stream needs to be preprocessed through a necessary preprocessing process. Here, the preprocessing process includes: image filtering to eliminate noise and contrast stretching to improve image clarity. Degree and the difference in illumination of the image. After the pre-processing is performed, the difference of the image data of each frame in the image data stream will be reduced, which is helpful for the improvement of the effect of the subsequent image registration algorithm.

Step 1102: Determine a reference frame from the image data stream, and register each frame image data of the image data stream except the reference frame with the reference frame.

Through a sequence of pictures, the motion velocity and direction of motion of each pixel in each image are found to be the optical flow field. As shown in Figure 4, the position of point A in the T frame is (x1, y1), then we find point A in the T+1 frame, if its position is (x2, y2), then we It is possible to determine the motion vector of point A: V = (x2, y2) - (x1, y1).

Suppose that the color of an object does not have a large and significant change in the two frames before and after. Based on this thinking Road, you can get the image constraint equation. Different optical flow algorithms solve the problem of optical flow that assumes different additional conditions. Using partial derivatives for spatial and temporal coordinates, the image constraint equation can be written as:

I(x,y,t)=I(x+dx,y+dy,t+dt) (1b)

Where I(x, y, t) is the pixel value of the image at the (x, y) position.

among them,

Among them, HOT refers to higher order, and HOT can be ignored when the movement is small enough. From equation (2b) you can get:

Vx=dx, Vx=dx (4b)

Ix*Vx+Iy*Vy=-It (6b)

There are 2 unknowns in the above equation, and at least two non-correlated equations are needed to solve. The Lucas-Kanade optical flow method assumes that the spatial pixel points move in unison, and adjacent points on one scene are projected onto the image as neighboring points, and the neighboring points are at the same speed. This is a unique assumption of the Lucas-Kanade optical flow method because the basic equation of the optical flow method has only one constraint, and the speed in the x, y direction requires two unknown variables. Assuming similar motions in the neighborhood of the feature points, it is possible to find the velocity in the x, y direction by n multiple equations (n is the total number of points in the neighborhood of the feature points, including the feature points). You can get the following equation:

Then the optical flow can be obtained adjacent to V:

Figure 6 shows the alignment process for multi-frame images using optical flow. Using the calculation method of the optical flow field, the sparse matching points between the two images can be obtained, and then the image mapping model is calculated by using the coordinates of these points. In the image alignment step, it is important to choose the correct image alignment transformation model. Common spatial transformation models include affine transformation and perspective transformation models.

The affine transformation can be visually represented in the following form. Any parallelogram in one plane can be mapped to another parallelogram by affine transformation, and the mapping operation of the image is in the same space. In-plane, different deformation parameters are used to deform different types of parallelograms.

The meaning of each element of the above matrix is as follows:

a ₀₂ and a ₁₂ are displacement parameters;

a ₀₀ a ₀₁ and a ₁₀ a ₁₁ are scaling and rotation parameters;

Step 1103: When performing the fusion processing on the image data of each frame after the registration, the pixels in the black area at the image boundary are replaced with the pixels corresponding to the reference frame, and then the fusion processing is performed to obtain the target image.

In the embodiment of the present invention, after the image data of each frame is registered (that is, aligned), the image needs to be fused, and here, a fusion method in which image pixels are sequentially superimposed is adopted. Fusion processing according to formula (14b):

Where I is for each image, m is the mth image, and k is the number of images that have been synthesized, N The total number of sheets synthesized for the image.

Referring to Fig. 13, at the boundary of the image, black edges are formed after image map conversion, and if the pixels at these black edges are fused in a weighted average manner, a difference in luminance is formed, which affects the overall vision of the image. Taking the reference frame as the first frame image data in the image data stream as an example, the other frame image data are all registered and fused under the first frame image data, and the reference frame does not have black-rim pixels, and the image with black edges In terms of data, the pixel at the black edge is replaced with the pixel at the corresponding position of the reference frame to participate in the weighted average, thereby effectively solving the problem of black edge of the image.

According to the technical solution of the embodiment of the present invention, an image processing method is proposed, which uses image registration principle to perform image alignment on image data to be synthesized in multiple frames, and the alignment method allows image jitter error within a certain range, and the final composite effect of the image The pixel deviation that occurs is small. More importantly, in the image fusion, the black edges of the pixels caused by image registration are processed to ensure that the pixel points of the entire image are naturally transitioned.

FIG. 12 is a schematic flowchart of an image processing method according to Embodiment 4 of the present invention. The image processing method in this example is applied to a terminal. As shown in FIG. 12, the image processing method includes the following steps:

Step 1201: In the handheld mode, the original image data stream is obtained by using an electronic aperture, the original image data stream includes a plurality of frames of original image data; and the original image data of each frame in the original image data stream is preprocessed to obtain The image data stream.

In the electronic aperture mode, after the user adjusts the aperture value, the terminal performs continuous and uninterrupted shooting during the exposure time, and then fuses the captured multiple images. When taking pictures with an electronic aperture, since it is necessary to continuously take multiple images for fusion processing, each image needs to be aligned. In order to ensure the ease of use and user experience of the electronic aperture, the embodiment of the present invention adds a handheld mode for the electronic aperture shooting. In the gesture mode, the user can conveniently use the hand. The terminal uses the electronic aperture to shoot.

At the time of shooting, an image data stream is first acquired, the image data stream comprising a plurality of frames of image data. Specifically, the captured original image data stream is first acquired synchronously; then the original image data stream is subjected to data reading and image preprocessing to obtain the image data stream. Here, due to the ISP processing flow of the camera shooting and the unpredictable transformation of the external environment, the raw image data of each frame in the synchronously acquired original image data stream is different in illumination, noise, sharpness, and focus point. Before the fusion processing, the original image data of each frame in the original image data stream needs to be preprocessed through a necessary preprocessing process. Here, the preprocessing process includes: image filtering to eliminate noise and contrast stretching to improve image clarity. Degree and the difference in illumination of the image. After the pre-processing is performed, the difference of the image data of each frame in the image data stream will be reduced, which is helpful for the improvement of the effect of the subsequent image registration algorithm.

Step 1202: Align each frame image data of the image data stream except the first frame image data with the reference frame by using the first frame image data in the image data stream as a reference frame.

Step 1203: Display a prompt box on the display interface; the prompt box is used to prompt the orientation of the handheld shake when shooting in the handheld mode with the electronic aperture.

Step 1204: Analyze the image data of each frame after registration to determine a black area at the boundary of each frame image; when performing fusion processing on the image data of each frame after registration, replace the pixel of the black area at the boundary of the image with The pixels corresponding to the reference frame are subjected to fusion processing to obtain a target image.

Here, the black area at the boundary of the image can be divided by a clustering algorithm, such as the K-means algorithm.

Referring to Fig. 13, at the boundary of the image, black edges are formed after image map conversion, and if the pixels at these black edges are fused in a weighted average manner, a difference in luminance is formed, which affects the overall vision of the image. Taking the reference frame as the first frame image data in the image data stream as an example, The image data of each other frame is registered and fused under the image data of the first frame, and the black pixel is not present in the reference frame. For the image data with black border, the pixel at the black edge is replaced with the corresponding position of the reference frame. The pixels are involved in the weighted average, which effectively solves the problem of black edges of the image.

According to the technical solution of the embodiment of the present invention, an image processing method is proposed, which uses image registration principle to perform image alignment on image data to be synthesized in multiple frames, and the alignment method allows image jitter error within a certain range, and the final composite effect of the image The pixel deviation that occurs is small. More importantly, in the image fusion, the black edges of the pixels caused by image registration are processed to ensure that the pixel points of the entire image are naturally transitioned. In addition, during the shooting of the electronic aperture, a prompt box is added to prevent the terminal from being over-ranged when shooting.

FIG. 14 is a schematic structural diagram of a terminal according to Embodiment 2 of the present invention. As shown in FIG. 14, the terminal includes:

An acquiring unit 41, configured to acquire an image data stream by using an electronic aperture, where the image data stream includes multi-frame image data;

The registration unit 42 is configured to determine a reference frame from the image data stream, and register, in the image data stream, image data of each frame other than the reference frame with the reference frame;

The merging unit 43 is configured to perform merging processing on the image data of each frame after the registration, and replace the pixels in the black area at the image boundary with the pixels corresponding to the reference frame, and perform fusion processing to obtain a target image.

The fusion unit 43 includes:

The analyzing sub-unit 431 is configured to analyze the image data of each frame after registration, and determine a black area at a boundary of each frame image;

The replacement and fusion sub-unit 432 is configured to perform merging processing on the image data of each frame after the registration, and replace the pixels in the black area at the image boundary with the pixels corresponding to the reference frame, and perform fusion processing to obtain a target image.

The replacement and fusion subunit 432 is further configured to: according to a black area at the boundary of the image, Determining a reference area corresponding to the black area in the reference frame; replacing a pixel of the black area at the image boundary with a pixel in the reference area in the reference frame, and performing a blending process to obtain a target image.

The registration unit 42 is further configured to use, by using the first frame image data in the image data stream as a reference frame, image data of each frame except the first frame image data in the image data stream. The reference frames are aligned; wherein the alignment refers to aligning pixel points of the same spatial location.

The merging unit 43 is further configured to superimpose each pixel point of each frame image data after registration according to spatial position correspondence.

10 is a schematic structural diagram of a terminal according to an embodiment of the present invention. The terminal includes: a processor 1001, a camera 1002, and a display screen 1003. The processor 1001, the camera 1002, and the display screen 1003 are all connected by a bus 1004.

The camera 1102 is configured to acquire an image data stream using an electronic aperture, the image data stream comprising a plurality of frames of image data;

The processor 1001 is configured to determine a reference frame from the image data stream, and register image data of each frame except the reference frame in the image data stream with the reference frame; When the subsequent image data of each frame is subjected to the fusion processing, the pixels of the black area at the image boundary are replaced with the pixels corresponding to the reference frame, and then the fusion processing is performed to obtain the target image.

Specifically, the camera 1102 is configured to obtain an original image data stream by using an electronic aperture in a handheld mode, where the original image data stream includes multiple frames of original image data;

The processor 1101 is configured to perform pre-processing on each frame of the original image data stream to obtain the image data stream; wherein the pre-processing includes at least the following a: image filtering, contrast stretching; determining a reference frame from the image data stream, and registering each frame image data of the image data stream except the reference frame with the reference frame; When the image data of each frame is subjected to the fusion processing, the pixels of the black area at the image boundary are replaced with the pixels corresponding to the reference frame, and then the fusion processing is performed to obtain the target image.

The display screen 1103 is configured to display a prompt box on the display interface; the prompt box is used to prompt the orientation of the handheld shake when photographing with the electronic aperture in the handheld mode.

Figure 15 is a block diagram of the electrical structure of the camera.

The lens 1211 is composed of a plurality of optical lenses for forming a subject image, and is a single focus lens or a zoom lens. The lens 1211 is movable in the optical axis direction under the control of the lens driver 1221, and the lens driver 1221 controls the focus position of the lens 1211 in accordance with a control signal from the lens driving control circuit 1222, and in the case of the zoom lens, the focus distance can also be controlled. . The lens drive control circuit 1222 performs drive control of the lens driver 1221 in accordance with a control command from the microprocessor 1217.

An imaging element 1212 is disposed in the vicinity of the position of the subject image formed by the lens 1211 on the optical axis of the lens 1211. The imaging element 1212 is for capturing an image of a subject and acquiring captured image data. Photodiodes constituting each pixel are arranged two-dimensionally and in a matrix on the imaging element 1212. Each photodiode generates a photoelectric conversion current corresponding to the amount of received light, and the photoelectric conversion current is charged by a capacitor connected to each photodiode. A red-green-blue (RGB) color filter of a Bayer arrangement is disposed on the front surface of each pixel.

The imaging element 1212 is connected to the imaging circuit 1213. The imaging circuit 1213 performs charge accumulation control and image signal readout control in the imaging element 1212, and performs waveform shaping after reducing the reset noise of the read image signal (analog image signal). Further, gain improvement or the like is performed to obtain an appropriate level signal.

The imaging circuit 1213 is connected to an A/D converter 1214 that performs analog-to-digital conversion on the analog image signal and outputs a digital image signal to the bus 1227 (hereinafter referred to as an image). data).

The bus 1227 is a transmission path for transmitting various data read or generated inside the camera. The A/D converter 1214 is connected to the bus 1227, and an image processor 1215, a JPEG processor 1216, a microprocessor 1217, a Synchronous Dynamic Random Access Memory (SDRAM) 1218, and a memory are also connected. An interface (hereinafter referred to as a memory I/F) 1219 and a liquid crystal display (LCD) driver 1220.

The image processor 1215 performs optical black (OB, Optical Black) subtraction processing, white balance adjustment, color matrix calculation, gamma conversion, color difference signal processing, noise removal processing, and simultaneous processing on the image data based on the output of the imaging element 1212. Various image processing such as edge processing. The JPEG processor 1216 compresses the image data read out from the SDRAM 1218 in accordance with the JPEG compression method when the image data is recorded on the storage medium 1225. Further, the JPEG processor 1216 performs decompression of JPEG image data for image reproduction display. At the time of decompression, the file recorded in the storage medium 1225 is read, and after the compression processing is performed in the JPEG processor 1216, the decompressed image data is temporarily stored in the SDRAM 1218 and displayed on the LCD 1226. Further, in the present embodiment, the JPEG method is adopted as the image compression/decompression method. However, the compression/decompression method is not limited thereto, and other compression/decompression methods such as MPEG, TIFF, and H.264 may be used.

The microprocessor 1217 functions as a control unit of the entire camera, and collectively controls various processing sequences of the camera. The microprocessor 1217 is connected to the operating unit 1223 and the flash memory 1224.

The operating unit 1223 includes, but is not limited to, a physical button or a virtual button, and the entity or virtual button may be a power button, a camera button, an edit button, a dynamic image button, a reproduction button, a menu button, a cross button, an OK button, a delete button, an enlarge button The operation controls such as various input buttons and various input keys detect the operational state of these operation controls.

The detection result is output to the microprocessor 1217. Also, in the LCD1226 as a display The front surface is provided with a touch panel to detect the touch position of the user, and the touch position is output to the microprocessor 1217. The microprocessor 1217 executes various processing sequences corresponding to the user's operation in accordance with the detection result from the operation position of the operation unit 1223.

Flash memory 1224 stores programs for executing various processing sequences of microprocessor 1217. The microprocessor 1217 performs overall control of the camera in accordance with the program. Further, the flash memory 1224 stores various adjustment values of the camera, and the microprocessor 1217 reads out the adjustment value, and performs control of the camera in accordance with the adjustment value.

The SDRAM 1218 is an electrically replaceable volatile memory for temporarily storing image data or the like. The SDRAM 1218 temporarily stores image data output from the analog/digital (A/D) converter 1214 and image data processed in the image processor 1215, the JPEG processor 1216, and the like.

The memory interface 1219 is connected to the storage medium 1225, and performs control for writing image data and a file header attached to the image data to the storage medium 1225 and reading out from the storage medium 1225. The storage medium 1225 can be implemented as a storage medium such as a memory card that can be detachably attached to the camera body. However, the storage medium 1225 is not limited thereto, and may be a hard disk or the like built in the camera body.

The LCD driver 1210 is connected to the LCD 1226, and stores the image data processed by the image processor 1215 in the SDRAM 1218. When display is required, the image data stored in the SDRAM 1218 is read and displayed on the LCD 1226, or the JPEG processor 1216 is compressed. The image data is stored in the SDRAM 1218. When display is required, the JPEG processor 1216 reads the compressed image data of the SDRAM 1218, decompresses it, and displays the decompressed image data through the LCD 1226.

The LCD 1226 is disposed on the back side of the camera body for image display. However, the present invention is not limited thereto, and image display may be performed using various display panels based on organic EL, that is, OLED.

In the embodiment of the present invention, if the terminal is implemented in the form of a software function module and sold or used as a stand-alone product, it may also be stored in a computer readable storage medium. Based on such understanding, the technical solution of the embodiments of the present invention contributes essentially or to the prior art. Portions may be embodied in the form of a software product stored in a storage medium, including instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) to perform various embodiments of the present invention All or part of the method. The foregoing storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a magnetic disk, or an optical disk. Thus, embodiments of the invention are not limited to any specific combination of hardware and software.

Accordingly, an embodiment of the present invention further provides a computer storage medium in which a computer program is stored, the computer program being configured to perform an image processing method according to an embodiment of the present invention.

The technical solutions described in the embodiments of the present invention can be arbitrarily combined without conflict.

In the several embodiments provided by the present invention, it should be understood that the disclosed method and smart device may be implemented in other manners. The device embodiments described above are merely illustrative. For example, the division of the unit is only a logical function division. In actual implementation, there may be another division manner, such as: multiple units or components may be combined, or Can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, or direct coupling, or communication connection of the components shown or discussed may be indirect coupling or communication connection through some interfaces, devices or units, and may be electrical, mechanical or other forms. of.

The units described above as separate components may or may not be physically separated, and the components displayed as the unit may or may not be physical units, that is, may be located in one place or distributed to multiple network units; Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one second processing unit, or each unit may be separately used as one unit, or two or more units may be integrated into one unit; The above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional units.

The above is only a specific embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily think of changes or substitutions within the technical scope of the present invention. It should be covered by the scope of the present invention.

Industrial applicability

The technical solution of the embodiment of the invention increases the hand-held mode of the electronic aperture, improves the convenience of the user's shooting, and introduces the image registration in the embodiment of the invention, which ensures the effect of the shooting and improves the user. Shooting experience. In addition, the embodiment of the present invention further adds a prompt box on the display interface to prevent the handheld terminal from being over-ranged when the user shoots. In addition, the user can carry out the shooting of the electronic aperture by the hand-held terminal, which improves the convenience of the user's shooting, avoids the problem that the image is unclear due to the hand-held, ensures the shooting effect, and improves the user's shooting experience. When image fusion is performed, the black side of the pixel caused by image registration is processed, which ensures that the pixel point transition of each position of the entire image is natural.

Claims

A terminal, the terminal comprising:

An acquiring unit configured to acquire an image data stream by using an electronic aperture, the image data stream comprising a plurality of frames of image data;

a registration unit configured to register image data of each frame in the image data stream;

The fusion unit is configured to perform fusion processing on the image data of each frame after registration to obtain a target image.
The terminal according to claim 1, wherein the obtaining unit comprises:

a shooting subunit configured to capture an original image data stream by using an electronic aperture in a handheld mode, the original image data stream comprising a plurality of frames of raw image data;

a pre-processing unit configured to pre-process each frame of the original image data stream to obtain the image data stream; wherein the pre-processing comprises at least one of: image filtering, contrast stretching .
The terminal of claim 2, wherein the terminal further comprises:

The prompting unit is configured to display a prompt box on the display interface, and the prompting box is configured to prompt the orientation of the handheld shaking when the electronic aperture is used in the handheld mode.
The terminal according to any one of claims 1 to 3, wherein the registration unit is further configured to divide the image data stream by using first frame image data in the image data stream as a reference frame. Each frame image data other than the first frame image data is aligned with the reference frame; wherein the alignment refers to aligning pixel points of the same spatial position.
An image processing method, the method comprising:

Acquiring an image data stream using an electronic aperture, the image data stream comprising a plurality of frames of image data;

Registering each frame of image data in the image data stream;

The image data of each frame after registration is subjected to fusion processing to obtain a target image.
The image processing method according to claim 5, wherein said utilizing an electronic aperture Get the image data stream, including:

In the handheld mode, an original image data stream is obtained by taking an electronic aperture, the original image data stream comprising a plurality of frames of original image data;

Performing pre-processing on each frame of the original image data in the original image data stream to obtain the image data stream;

Wherein, the preprocessing comprises at least one of the following: image filtering, contrast stretching.
The image processing method according to claim 6, wherein the method further comprises:

A prompt box is displayed on the display interface; the prompt box is used to indicate the orientation of the handheld shake when the electronic aperture is used in the handheld mode.
The image processing method according to any one of claims 5 to 7, wherein the registering each frame image data in the image data stream comprises:

Aligning, by using the first frame image data in the image data stream as a reference frame, each frame image data of the image data stream except the first frame image data, and the reference frame;

Wherein, the alignment refers to aligning pixel points of the same spatial position.
The image processing method according to any one of claims 5 to 7, wherein the performing the fusion processing on the image data of each frame after registration comprises:

Each pixel point of the image data of each frame after registration is superimposed according to the spatial position correspondence.
A terminal, the terminal comprising:

An acquiring unit configured to acquire an image data stream by using an electronic aperture, the image data stream comprising a plurality of frames of image data;

a registration unit, configured to determine a reference frame from the image data stream, and register, in the image data stream, image data of each frame other than the reference frame with the reference frame;

The fusion unit is configured to perform fusion processing on the image data of each frame after registration The pixel corresponding to the black area at the boundary is replaced with the pixel corresponding to the reference frame, and then the fusion processing is performed to obtain the target image.
The terminal according to claim 10, wherein the fusion unit comprises:

The analysis subunit is configured to analyze the image data of each frame after registration to determine a black area at a boundary of each frame image;

The replacement and fusion subunits are configured to perform fusion processing on the image data of each frame after the registration, and replace the pixels in the black area at the image boundary with the pixels corresponding to the reference frame, and perform fusion processing to obtain a target image.
The terminal according to claim 11, wherein the replacement and fusion subunit is further configured to determine a reference area corresponding to the black area in the reference frame according to a black area at the image boundary; The pixel in the black area at the image boundary is replaced with the pixel in the reference area in the reference frame, and then the fusion processing is performed to obtain the target image.
The terminal according to any one of claims 10 to 12, wherein the registration unit is further configured to divide the image data stream by using first frame image data in the image data stream as a reference frame. Each frame image data other than the first frame image data is aligned with the reference frame; wherein the alignment refers to aligning pixel points of the same spatial position.
An image processing method, the method comprising:

Acquiring an image data stream using an electronic aperture, the image data stream comprising a plurality of frames of image data;

Determining a reference frame from the image data stream, and registering each frame image data of the image data stream except the reference frame with the reference frame;

When the image data of each frame after the registration is subjected to the fusion processing, the pixels of the black area at the image boundary are replaced with the pixels corresponding to the reference frame, and then the fusion processing is performed to obtain the target image.
The image processing method according to claim 14, wherein when the image data of each frame after the registration is subjected to the fusion processing, the pixels of the black area at the image boundary are replaced with the pixels corresponding to the reference frame, and then the fusion processing is performed. , get the target image, including:

Performing analysis on the image data of each frame after registration to determine a black area at the boundary of each frame image;

When the image data of each frame after the registration is subjected to the fusion processing, the pixels of the black area at the image boundary are replaced with the pixels corresponding to the reference frame, and then the fusion processing is performed to obtain the target image.
The image processing method according to claim 15, wherein the replacing the pixels of the black area at the image boundary with the pixels corresponding to the reference frame and performing the merging process to obtain the target image comprises:

Determining a reference area corresponding to the black area in the reference frame according to a black area at the boundary of the image;

The pixel in the black area at the image boundary is replaced with the pixel in the reference area in the reference frame, and then the fusion processing is performed to obtain the target image.
The image processing method according to any one of claims 14 to 16, wherein said determining a reference frame from said image data stream, and extracting image data of each frame other than said reference frame in said image data stream Registration with the reference frame, including:

Aligning, by using the first frame image data in the image data stream as a reference frame, each frame image data of the image data stream except the first frame image data, and the reference frame;

Wherein, the alignment refers to aligning pixel points of the same spatial position.
The image processing method according to any one of claims 14 to 16, wherein the performing the fusion processing on the image data of each frame after registration comprises:

Each pixel point of the image data of each frame after registration is superimposed according to the spatial position correspondence.
A terminal, the terminal comprising: a camera and a processor;

The camera is configured to acquire an image data stream using an electronic aperture, the image data stream comprising a plurality of frames of image data;

The processor is configured to register image data of each frame in the image data stream, and perform fusion processing on the image data of each frame after registration to obtain a target image.
A terminal, the terminal comprising: a camera and a processor;

The camera is configured to acquire an image data stream using an electronic aperture, the image data stream comprising a plurality of frames of image data;

The processor is configured to determine a reference frame from the image data stream, and register image data of each frame other than the reference frame in the image data stream with the reference frame; When the image data of each frame is subjected to the fusion processing, the pixels of the black area at the image boundary are replaced with the pixels corresponding to the reference frame, and then the fusion processing is performed to obtain the target image.
A computer storage medium having stored therein computer executable instructions configured to perform the image processing method of any of claims 5-9.
A computer storage medium having stored therein computer executable instructions configured to perform the image processing method of any of claims 14-18.