WO2022056729A1 - 电子装置和电子装置的图像处理方法 - Google Patents

电子装置和电子装置的图像处理方法 Download PDF

Info

Publication number
WO2022056729A1
WO2022056729A1 PCT/CN2020/115590 CN2020115590W WO2022056729A1 WO 2022056729 A1 WO2022056729 A1 WO 2022056729A1 CN 2020115590 W CN2020115590 W CN 2020115590W WO 2022056729 A1 WO2022056729 A1 WO 2022056729A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
image signal
processor
electronic device
isp
Prior art date
Application number
PCT/CN2020/115590
Other languages
English (en)
French (fr)
Inventor
黄飞
胡翠
张伟成
储洁宇
秦秋石
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to CN202080011168.6A priority Critical patent/CN114514552A/zh
Priority to CN202211583539.5A priority patent/CN115984083B/zh
Priority to PCT/CN2020/115590 priority patent/WO2022056729A1/zh
Priority to EP20953587.1A priority patent/EP4198869A4/en
Publication of WO2022056729A1 publication Critical patent/WO2022056729A1/zh
Priority to US18/184,659 priority patent/US20230214955A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/20—Processor architectures; Processor configuration, e.g. pipelining
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Definitions

  • the embodiments of the present application relate to the field of electronic technologies, and in particular, to an electronic device and an image processing method for the electronic device.
  • smart terminals have integrated more and more functions. Thanks to the development of image processing technology, more and more users like to use smart terminal devices to take photos, record videos, and make video calls.
  • the AI processor is set at the back end of the ISP as a supplementary correction to the ISP image processing results, that is, AI post-processing.
  • the ISP stores the processed image to the off-chip memory, and the AI processor reads the image stored by the ISP from the off-chip memory, and further corrects the image on the basis of the ISP image processing to generate the final image.
  • the electronic device and the image processing method for the electronic device provided by the present application can improve the image processing effect.
  • the present application adopts the following technical solutions.
  • an embodiment of the present application provides an electronic device, the electronic device includes: an artificial intelligence AI processor, configured to perform first image signal processing on a first image signal to obtain a second image signal, the first image signal The image signal is obtained based on image data output by the image sensor; the image signal processor ISP is configured to perform second image signal processing on the second image signal to obtain an image processing result.
  • an artificial intelligence AI processor configured to perform first image signal processing on a first image signal to obtain a second image signal, the first image signal The image signal is obtained based on image data output by the image sensor; the image signal processor ISP is configured to perform second image signal processing on the second image signal to obtain an image processing result.
  • the processing capability of the AI processor can be fully utilized in the entire image signal processing process, so that the AI processor replaces the traditional ISP to perform part of the processing process, which can improve the The quality of the image processing results output by the ISP.
  • the ISP is further configured to: receive the image data from the image sensor, and perform third image signal processing on the image data to obtain the first image data image signal.
  • the first image processing process performed by the AI processor can be set between multiple image processing processes performed by the ISP, and the AI processor can replace the ISP to perform a part of the image processing process in the middle of the image processing process to achieve the preset effect, improve the flexibility of the combination of AI processor and ISP, so as to improve the image processing effect.
  • the third image signal processing includes multiple processing procedures, and in two adjacent processing procedures among the multiple processing procedures, the previous processing procedure is used for A third image signal is generated, and the latter processing process is used for processing the fourth image signal; the AI processor is further configured to perform fourth image signal processing on the third image signal to obtain the fourth image signal.
  • This scheme further improves the flexibility of performing image processing processes with AI processors.
  • the first image signal processing includes at least one of the following processing procedures: noise removal, black level correction, shadow correction, white balance correction, demosaicing, chromatic aberration correction, or gamma correction Horse correction.
  • the second image signal processing includes at least one of the following processing procedures: noise removal, black level correction, shadow correction, white balance correction, demosaicing, chromatic aberration correction, gamma correction Horse correction, chromatic aberration correction or RGB to YUV domain.
  • the third image signal processing includes at least one of the following processing procedures: noise removal, black level correction, shadow correction, white balance correction, or demosaicing.
  • the fourth image signal processing includes at least one of the following processing procedures: black level correction, shadow correction, white balance correction, demosaicing, or chromatic aberration correction.
  • the electronic device further includes: a memory, coupled to the AI processor and the ISP, for transferring between the AI processor and the ISP The first image element in any image signal.
  • the memory, the AI processor and the ISP are located in a system-on-chip in the electronic device; the memory includes an on-chip random access memory RAM.
  • the first image unit includes any one of the following: a single-frame image or an image block in a single-frame image.
  • the electronic device further includes: an off-chip memory, located outside the system-on-chip, and used for transferring any memory between the AI processor and the ISP A second image unit in the image signal, the second image unit includes multiple frames of images.
  • the electronic device further includes: a controller, configured to trigger the AI processor to perform the first image signal processing, and control the ISP to perform the first image signal processing 2. Image signal processing.
  • an interruption signal is transmitted between the AI processor and the ISP through an electronic circuit connection.
  • the electronic circuit connection includes connection through an interrupt controller.
  • the interrupted signal transmission is realized by setting up an electronic circuit connection between the AI processor and the ISP, which does not need to be forwarded by other processors such as CPU, which can improve the signal transmission speed.
  • the image output delay can be reduced. , which is beneficial to improve the user experience.
  • the electronic device further includes the image sensor.
  • an embodiment of the present application provides an image processing method for an electronic device, the image processing method comprising: controlling an artificial intelligence AI processor to perform first image signal processing on a first image signal to obtain a second image signal, the The first image signal is obtained based on image data output by the image sensor; the image signal processor ISP is controlled to perform second image signal processing on the second image signal to obtain an image processing result.
  • the method before controlling the artificial intelligence AI processor to perform the first image signal processing on the first image signal, the method further includes: controlling the ISP to receive the image data from the image sensor, and performing third image signal processing on the image data to obtain the first image signal.
  • the third image signal processing includes multiple processing procedures, and in two adjacent processing procedures among the multiple processing procedures, the previous processing procedure is used for generating a third image signal, and the latter processing process is used to process the fourth image signal; the method further includes: controlling the AI processor to perform fourth image signal processing on the third image signal to obtain the fourth image Signal.
  • the first image signal processing includes at least one of the following processing procedures: noise removal, black level correction, shadow correction, white balance correction, demosaicing, chromatic aberration correction, or gamma correction Horse correction.
  • the second image signal processing includes at least one of the following processing procedures: noise removal, black level correction, shadow correction, white balance correction, demosaicing, chromatic aberration correction, gamma correction Horse correction, chromatic aberration correction or RGB to YUV domain.
  • the third image signal processing includes at least one of the following processing procedures: noise removal, black level correction, shadow correction, white balance correction, or demosaicing.
  • the fourth image signal processing includes at least one of the following processing procedures: black level correction, shadow correction, white balance correction, demosaicing, or chromatic aberration correction.
  • an embodiment of the present application provides an image processing apparatus, the image processing apparatus includes: an AI processing module configured to perform first image signal processing on a first image signal to obtain a second image signal, the first image signal The signal is obtained based on the image data output by the image sensor; the image signal processing module is configured to perform second image signal processing on the second image signal to obtain the image processing result.
  • the image signal processing module is further configured to: receive the image data from the image sensor, and perform third image signal processing on the image data to obtain the first image data. an image signal.
  • the third image signal processing includes multiple processing procedures, and in two adjacent processing procedures among the multiple processing procedures, the previous processing procedure is used for A third image signal is generated, and the latter processing process is used for processing the fourth image signal; the AI processing module is further configured to perform fourth image signal processing on the third image signal to obtain the fourth image signal.
  • the first image signal processing includes at least one of the following processing procedures: noise removal, black level correction, shadow correction, white balance correction, demosaicing, chromatic aberration correction, or gamma correction Horse correction.
  • the second image signal processing includes at least one of the following processing procedures: noise removal, black level correction, shadow correction, white balance correction, demosaicing, chromatic aberration correction, gamma correction Horse correction, chromatic aberration correction or RGB to YUV domain.
  • the third image signal processing includes at least one of the following processing procedures: noise removal, black level correction, shadow correction, white balance correction, or demosaicing.
  • the fourth image signal processing includes at least one of the following processing procedures: black level correction, shadow correction, white balance correction, demosaicing, or chromatic aberration correction.
  • an embodiment of the present application provides an electronic device, the electronic device includes a memory and at least one processor, where the memory is used to store a computer program, and the at least one processor is configured to call the memory to store All or part of the computer program of the above-mentioned second aspect executes the method.
  • the at least one processor includes the AI processor and an ISP.
  • the electronic device further includes the image sensor.
  • an embodiment of the present application provides a system-on-chip, the system-on-chip includes at least one processor and an interface circuit, and the interface circuit is used to obtain a computer program from outside the system-on-chip; the computer program is described by the When executed by at least one processor, it is used to implement the method described in the second aspect.
  • the at least one processor includes the AI processor and an ISP.
  • an embodiment of the present application provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by at least one processor, is used to implement the second aspect.
  • the at least one processor includes the AI processor and an ISP.
  • an embodiment of the present application provides a computer program product, which is used to implement the method described in the second aspect above when the computer program product is executed by at least one processor.
  • the at least one processor includes the AI processor and an ISP.
  • FIG. 1 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application.
  • FIG. 2 is a schematic flowchart of an image processing method provided by an embodiment of the present application.
  • FIG. 3 is another schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application.
  • FIG. 4 is another schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application.
  • FIG. 5 is another schematic flowchart of the image processing method provided by the embodiment of the present application.
  • FIG. 6 is another schematic flowchart of the image processing method provided by the embodiment of the present application.
  • FIG. 7 is another schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application.
  • FIG. 8 is another schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application.
  • FIG. 9 is a schematic diagram of a software structure of an electronic device provided by an embodiment of the present application.
  • references herein to "first,” or “second,” and similar terms do not denote any order, quantity, or importance, but are merely used to distinguish the different parts. Likewise, words such as “a” or “an” do not denote a quantitative limitation, but rather denote the presence of at least one. Words like “coupled” are not limited to physical or mechanical direct connections, but may include electrical connections, whether direct or indirect, equivalent to communication in a broad sense.
  • words such as “exemplary” or “for example” are used to represent examples, illustrations or illustrations. Any embodiments or designs described in the embodiments of the present application as “exemplary” or “such as” should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as “exemplary” or “such as” is intended to present the related concepts in a specific manner.
  • the meaning of "plurality” refers to two or more. For example, a plurality of processors refers to two or more processors.
  • the electronic device provided by the embodiments of the present application may be an electronic device or a module, chip, chip set, circuit board or component integrated in the electronic device.
  • the electronic device may be a user equipment (User Equipment, UE), such as various types of devices such as a mobile phone, a tablet computer, a smart screen, or an image capturing device.
  • UE User Equipment
  • the electronic device may be provided with a camera, which may also be referred to as an image sensor, for capturing image data.
  • the electronic device may also be installed with various software applications, such as camera applications, video calling applications, or online video shooting applications, which are used to drive the camera device to capture images.
  • the user can use the camera device to take pictures by starting the above applications. or video.
  • users can also personalize various image beautification settings through this type of application.
  • the user can select the picture presented to the screen during a video call (such as the presented facial avatar, or the presented picture). background image) for automatic adjustment (such as "one-click beautification").
  • the image processing service supported by the above-mentioned various applications in the electronic device can trigger the electronic device to process the image data collected by the camera, thereby The processed image is presented on the screen of the electronic device to achieve the effect of image beautification.
  • the above-mentioned image beautification may include, but is not limited to: improving the brightness of part of the image or the entire frame, changing the display color of the image, skinning the facial objects presented in the image, adjusting the saturation of the image, adjusting the exposure of the image, adjusting the sharpness of the image, Adjust screen highlights, adjust screen contrast, adjust screen sharpness, or adjust screen clarity, etc.
  • the image processing described in this embodiment of the present application may include, but is not limited to, noise removal, black level correction, shadow correction, white balance correction, demosaicing, chromatic aberration correction, gamma correction, or red-green-blue (RGB) conversion to YUV ( YCrCb) domain, so as to achieve the above image beautification effect.
  • the screen of the electronic device used by user A is displayed on the screen of the electronic device used by user A.
  • the image of user A displayed on the screen of the electronic device used by user B may be images processed by the electronic device described in the embodiments of the present application, and the processed images will be presented until user A and Used for B to terminate the video call or user A to close the image processing service.
  • FIG. 1 shows a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application.
  • the electronic device 100 may specifically be a chip or a chip set or a circuit board equipped with a chip or a chip set, or an electronic device including the circuit board, but it is not used to limit the embodiment.
  • the specific electronic device is as described above, which is omitted here.
  • the chip or chip set or the circuit board on which the chip or chip set is mounted can be driven by necessary software.
  • Electronic device 100 includes one or more processors, such as ISP 102 and AI processor 101 .
  • the one or more processors can be integrated in one or more chips, and the one or more chips can be regarded as a chipset, when one or more processors are integrated in the same chip
  • the chip is also called a system on a chip (SOC).
  • the electronic device 100 also includes one or more other necessary components, such as a memory 103 .
  • the memory 103 may be located in the same system-on-chip as the AI processor 101 and the ISP 102 in the electronic device 100 , that is, the memory 103 is integrated in the SOC as shown in FIG. 1 above. At this time, the memory 103 may include an on-chip random access memory (RAM, Random Access Memory).
  • RAM Random Access Memory
  • the AI processor 101 may include a neural network processor (Neural-network Processing Unit, NPU) and other special-purpose neural processors, including but not limited to a convolutional neural network processor, a tensor processor, or a neural processing unit. engine.
  • the AI processor can be used alone as a component or integrated in other digital logic devices, including but not limited to: CPU (Central Processing Unit, Central Processing Unit), GPU (Graphics Processing Unit, Graphics Processing Unit) or DSP ( Digital Signal Processor, Digital Signal Processing).
  • CPU Central Processing Unit
  • GPU Graphics Processing Unit
  • DSP Digital Signal Processor, Digital Signal Processing
  • the CPU, GPU and DSP are all processors within a system-on-chip.
  • the AI processor 101 may perform one or more image processing operations, which may include, but are not limited to, noise removal, black level correction, shadow correction, white balance correction, demosaicing, chromatic aberration correction Or gamma correction.
  • the AI processor 101 may run one or more image processing models, each of which is used to perform a specific image processing operation. For example, a noise-cancelling image processing model is used to perform a noise-cancelling image processing operation, and a demosaicing image processing model is used to perform a demosaicing image processing operation.
  • Each image processing model may be obtained by using a traditional neural network training method and training a neural network with training samples, which will not be repeated in this embodiment of the present application.
  • the ISP 102 can set up a plurality of hardware modules or run necessary software programs to process images or communicate with the AI processor 101 .
  • the ISP 102 and the AI processor 101 can communicate by means of direct hardware connection (for example, the relevant descriptions in the embodiments shown in FIG. 3 , FIG. 4 , and FIG. 7 ), or they can communicate by means of signal forwarding by the controller, wherein
  • the relevant description of the communication between the ISP 102 and the AI processor 101 by means of the controller performing signal forwarding refer specifically to the relevant description of the controller 104 shown in FIG. 8 .
  • the image data obtained from the camera 105 may undergo multiple image processing processes to generate a final image processing result, and the multiple image processing processes may include, but are not limited to: noise removal, black level correction, shadowing Correction, white balance correction, demosaicing, chromatic aberration correction, gamma correction or RGB to YUV domain.
  • the AI processor 101 can execute one or more of the above image processing processes, that is, corresponding to the above one or more image processing operations, the ISP 102 can also execute one or more of the above image processing processes.
  • the AI processor 101 and the ISP 102 may perform different image processing processes.
  • the AI processor 101 and the ISP 102 may also perform the same image processing process, such as performing further enhancement processing, which is not limited in this embodiment.
  • the image processing performed by the AI processor 101 can be used as an enhancement or supplement to the image processing process.
  • the AI processor 101 and the ISP 102 perform the process of noise removal simultaneously, the ISP 102 is used to perform the primary noise removal, and the AI processor 101 is used to perform the secondary noise removal based on the primary noise removal by the ISP 102 . Therefore, the entire image processing flow includes multiple processing procedures and is assigned to the AI processor 101 and ISP102 as tasks, which is equivalent to the AI processor 101 replacing the traditional ISP to perform part of the processing before the ISP102 completes all the processing procedures.
  • the processing result is output by ISP102.
  • the ISP 102 performs the remaining image processing processes.
  • the AI processor 101 may be used instead of the ISP 102 to perform part of the processing, avoiding The information of the original image data is lost due to the insufficient processing capability of the ISP102 in the image processing process, and the image processing effect is improved.
  • the image processing process performed by the AI processor 101 can be set as one of the multiple image processing processes performed by the ISP 102. (as shown in Figure 4 to Figure 7 of the embodiment) to replace the ISP to perform the image processing process to achieve a preset effect and improve the flexibility of the combination of the AI processor and the ISP.
  • FIG. 2 shows a schematic flowchart of an image processing method 200 provided by an embodiment of the present application.
  • the image processing method 200 is applied to the electronic device 100 shown in FIG. 1 or FIG. 3 .
  • the image processing method 200 includes the following image processing steps: Step 201 , the camera 105 provides the collected image data to the ISP 102 .
  • Step 202 the ISP 102 processes the image data to generate an image signal A.
  • Step 203 the ISP 102 provides the image signal A to the AI processor 101 .
  • Step 204 the AI processor 101 performs image processing on the image signal A to generate the image signal B.
  • step 205 the AI processor 101 provides the image signal B to the ISP 102.
  • step 206 the ISP 102 performs image processing on the image signal B to obtain an image processing result.
  • FIG. 3 shows another schematic structural diagram of the electronic device 100 provided by the embodiment of the present application.
  • the ISP 102 may include multiple cascaded image processing modules, the multiple cascaded image processing modules include image processing module 01 , image processing module 02 , image processing module 03 . . .
  • image processing module 01 is used to perform image processing of black level correction
  • image processing module 02 is used to perform image processing of shading correction
  • image processing module 03 is used to perform image processing of shading correction...
  • image processing module N+1 Used to perform RGB to YUV processing.
  • any one of the above-mentioned multiple cascaded image processing modules may be provided with an output port and an input port, the output port is used to send the image signal A to the AI processor 101, and the input port For acquiring the image signal B from the AI processor 101, FIG. 3 schematically shows that the image processing module 02 is provided with an output port V po1 , and the image processing module 03 is provided with an input port V pi1 .
  • FIG. 3 schematically shows that the image processing module 02 is provided with an output port V po1
  • the image processing module 03 is provided with an input port V pi1 .
  • the ISP 102 is connected to the camera 105 to acquire image data from the camera 105 .
  • the electronic device 100 is further provided with an on-chip RAM, which is integrated with the ISP 102 and the AI processor 101 in a chip in the electronic device 100 , and the on-chip RAM is used to store the image signal A and the image signal B shown in FIG. 2 .
  • the on-chip RAM is also used to store intermediate data generated during the running of the AI processor 101 and weight data of each network node in the neural network run by the AI processor 101 .
  • an interruption signal Z1 is transmitted between the AI processor 101 and the image processing module 02 in the ISP 102 through an electronic circuit connection L1, and the interruption signal Z1 is used to instruct the image processing module 02 to convert the image signal A is stored in the on-chip RAM;
  • an interruption signal Z2 is transmitted between the AI processor 101 and the image processing module 03 in the ISP 102 through an electronic circuit connection L2, the interruption signal Z2 is used to instruct the AI processor 101 to store the image signal B in the on-chip RAM.
  • the AI processor shown in FIG. 3 may include a task scheduler 1011 and multiple computing units 1012 . Each of the components in the task scheduler 1011 and the plurality of computing units 1012 may include a plurality of logic devices or circuits.
  • the task scheduler 1011 can realize the above-mentioned electronic circuit connection L1 through the input port V ai2 of the AI processor 101 and the output end Vp o2 of the image processing module 02, and the output port V ao2 of the AI processor 101 and the input end of the image processing module 03.
  • Vpi2 implements the aforementioned electronic circuit connection L2.
  • each computing unit in the plurality of computing units can realize electronic circuit connection with the above-mentioned on-chip RAM through the input port V ai1 , so as to read the image signal A from the on-chip RAM; electronic circuit connection with the above-mentioned on-chip RAM through the output port V ao1 , to write the image signal B to the on-chip RAM.
  • the ISP 102 obtains image data from the camera device 105, and the image data is processed by the image processing module 01 and the image processing module 02 in sequence to perform shadow correction and white balance correction processing to generate an image signal A and store it in the on-chip RAM.
  • the image processing module 02 sends the interrupt signal Z1 to the AI processor 101 through the electronic circuit connection L1 after storing the image signal A in the on-chip RAM.
  • the AI processor 101 acquires the image signal A from the on-chip RAM in response to the interrupt signal Z1.
  • the AI processor 101 performs demosaic processing on the image signal A to generate the image signal B, and stores the image signal B in the on-chip RAM.
  • the AI processor 101 stores the image signal B in the on-chip RAM and sends the above-mentioned interrupt signal Z2 to the image processing module 03.
  • the image processing module 03 reads the image signal B from the on-chip RAM in response to the interrupt signal Z2, and the image signal B passes through the image processing module 03..., the image processing module N and the image processing module N+1 in the ISP102 to perform chromatic aberration correction,...Gamma in turn After correction and RGB to YUV domain processing, the final image processing result is generated.
  • more image processing modules may be included before the image processing module 01, so that the ISP 102 performs more image processing processes on the image data.
  • the electronic circuit connection between the AI processor 101 and the ISP 102 is also called a physical connection or an interrupted connection, and the AI processor 101 and the ISP 102 implement the transmission and reception of an interrupt signal through this connection, so that the interrupt signal does not need to pass through other
  • the forwarding of the processor, such as the CPU does not require the CPU to participate in the relevant control, which can improve the transmission speed of the interrupt signal.
  • the image output delay can be reduced, which is conducive to improving the user experience.
  • the interrupted connection includes an interrupted signal processing hardware circuit for realizing the functions of transmitting and receiving the interrupted signal and a connection line for transmitting the signal, so as to realize the sending and receiving of the interrupting signal.
  • Interrupt signal processing hardware circuits include, but are not limited to, conventional interrupt controller circuits. For the specific implementation scheme of the interrupt signal processing hardware circuit, reference may be made to the relevant description of the interrupt controller in the prior art, which will not be repeated here.
  • the image processing process performed by the AI processor 101 is arranged between a plurality of image processing processes performed by the ISP 102 to replace or supplement some intermediate images performed by the ISP 102 processing.
  • the AI processor 101 may directly acquire image data from the camera device 105 and execute the front-end image processing process.
  • the AI processor 101 can replace some image processing modules in the front end of the ISP 102 to execute the corresponding image processing process.
  • the AI processor 101 can directly communicate with the image processing modules behind the ISP 102.
  • FIG. 4 for the hardware structure of this implementation, please refer to FIG. 4 , which shows another schematic diagram of the hardware structure provided by the embodiment of the present application.
  • the structure of the ISP 102 and the structure of the AI processor 101 are the same as the structure of the ISP 102 and the structure of the AI processor 101 shown in FIG. 3 .
  • the AI processor 101 skips the image processing module 01 and the image processing module 02 and communicates with the image processing module 03 .
  • the input port V pi1 of the image processing module 03 and the output port V ao1 of the AI processor 101 and the on-chip RAM can be connected by electronic circuits.
  • An interruption signal Z3 is transmitted between the AI processor 101 and the image processing module 03 in the ISP 102 through an electronic circuit connection L3, and the interruption signal Z3 is used to instruct the AI processor 101 to store the image signal C in the on-chip RAM.
  • the AI processor 101 acquires image data from the camera 105 , and then performs noise removal on the image data to generate an image signal C and store it in the on-chip RAM.
  • the AI processor 101 sends the interrupt signal Z3 to the image processing module 03 through the electronic circuit connection L3 after storing the image signal C in the on-chip RAM.
  • the image processing module 03 acquires the image signal C from the on-chip RAM in response to the interrupt signal Z3.
  • the image signal C is generated after the image processing module 03..., the image processing module N, and the image processing module N+1 in the ISP102 sequentially perform black level correction, shading correction, white balance correction...RGB to YUV domain and other image processing processes.
  • the final image processing result is generated after the image processing module 03..., the image processing module N, and the image processing module N+1 in the ISP102 sequentially perform black level correction, shading correction, white balance correction...RGB to YUV domain and other image processing processes.
  • FIG. 5 shows another flowchart of the image processing method 500 provided by the embodiment of the present application.
  • the image processing method 500 is applied to the electronic device 100 shown in FIG. 1 or FIG. 4 .
  • the image processing method 500 includes the following image processing steps: Step 501 , the camera device 105 provides the collected image data to the AI processor 101 .
  • the AI processor 101 processes the image data to generate an image signal C.
  • the AI processor 101 provides the image signal C to the ISP 102.
  • the ISP 102 processes the image signal C to obtain an image processing result.
  • the AI processor 101 may perform multiple consecutive image processing processes to process image data or image signals, such as The embodiments shown in Figures 2-5. In addition, in some other implementations, the AI processor 101 may also perform multiple discontinuous image processing processes to process image data or image signals.
  • FIG. 6 shows another flowchart of the image processing method 600 provided by the embodiment of the present application.
  • the image processing method 600 is applied to the electronic device 100 shown in FIG. 1 or FIG. 7 .
  • the image processing method 600 includes the following image processing steps: Step 601 , the camera 105 provides the collected image data to the ISP 102 .
  • Step 602 the ISP 102 processes the image data to generate an image signal D.
  • Step 603 the ISP 102 provides the image signal D to the AI processor 101 .
  • Step 604 the AI processor 101 processes the image signal D to generate the image signal E.
  • the AI processor 101 provides the image signal E to the ISP 102.
  • Step 606 the ISP 102 processes the image signal E to generate the image signal F.
  • Step 607 the ISP 102 provides the image signal F to the AI processor 101 .
  • Step 608 the AI processor 101 processes the image signal F to generate the image signal G.
  • step 609 the AI processor 101 provides the image signal G to the ISP 102.
  • step 610 the ISP 102 processes the image signal G to obtain an image processing result.
  • FIG. 7 shows another schematic structural diagram of the electronic device 100 provided by the embodiment of the present application.
  • the electronic device includes an AI processor 101, an ISP 102, and an on-chip RAM.
  • the ISP102 in the electronic device shown in FIG. 7 also includes a cascaded image processing module 01 , an image processing module 02 , an image processing module 03 . . . an image processing module N and an image processing module N+ 1.
  • the structure and function of each module are the same as the structure and function of each module in the ISP 102 shown in FIG. 3 , and will not be repeated here.
  • the on-chip RAM is used to store the image signal D, the image signal F, the image signal F, and the image signal G.
  • the on-chip RAM is used to store the image signal D, the image signal F, the image signal F, and the image signal G.
  • two of the above-mentioned multiple cascaded image processing modules are provided with output ports, and the two image processing modules are provided with output ports, as shown in FIG. 7 . Show.
  • An output port V po1 and an output port V po2 are respectively set at the output end of the image processing module 02 and the output end of the image processing module 03 , and an input port is respectively set at the input end of the image processing module 03 and the input end of the image processing module N V pi1 and input port V pi2 .
  • the output ports of the image processing module 02 and the image processing module 03 are respectively used to output the image signal D and the image signal F as shown in FIG. 6; the input port of the image processing module 03 and the input port of the image processing module N are respectively used to input the The image signal E and the image signal G shown in FIG. 6 .
  • the output port V po1 of the image processing module 02 and the output port V po2 of the image processing module 03 are respectively connected with the on-chip RAM by electronic circuits, and the input port V pi1 of the image processing module 03 and the image processing module N
  • An electronic circuit connection is made between the input port V pi2 and the on-chip RAM; the input port V ai1 and the output port V ao1 of the AI processor 101 can also be electronically connected to the on-chip RAM.
  • an interrupt signal Z4 is transmitted between the AI processor 101 and the image processing module 02 in the ISP 102 through an electronic circuit connection L4, and the interrupt signal Z4 is used to instruct the image processing module 02 to store the image signal D in the on-chip RAM; the AI processor 101
  • the interrupt signal Z5 and the interrupt signal Z6 are transmitted through the electronic circuit connection L5 with the image processing module 03 in the ISP102.
  • the interrupt signal Z5 is used to instruct the AI processor 101 to store the image signal E in the on-chip RAM
  • the interrupt signal Z6 is used to instruct the
  • the image processing module 03 stores the image signal F in the on-chip RAM
  • the AI processor 101 and the image processing module N in the ISP102 transmit an interrupt signal Z7 through the electronic circuit connection L6, and the interrupt signal Z7 is used to instruct the AI processor 101 to store the image.
  • Signal G is stored in on-chip RAM.
  • FIG. 3 to FIG. 7 above schematically illustrate the image processing methods shown in the embodiments of the present application and the hardware structure of the electronic device corresponding to each image processing method.
  • the ISP 102 in the electronic device shown in the embodiments of the present application may further include setting more output ports and input ports, so that images processed by more processing procedures are processed between the AI processor 101 and the ISP 102 transmission, so that the AI processor 101 can perform more image processing procedures at intervals. That is to say, the AI processor 101 and the ISP 102 can perform processing alternately, so that both parties can jointly complete the image processing process to obtain the processing result, instead of the image processing process of the traditional ISP.
  • the ISP 102 and the AI processor 101 are in the form of image blocks to the on-chip shown in FIG. 3 , FIG. 4 or FIG. 7 . stored in RAM.
  • the image block may be a partial image signal in one frame of image signal (for example, one frame of image signal has 1280 lines of pixels, and an image signal formed by 320 lines of pixels is stored in the on-chip RAM).
  • each image processing module in the ISP102 performs image processing in units of one row of pixels, and the ISP102 stores the processed image signals in the on-chip RAM row by row.
  • the ISP102 When the ISP102 has finished storing the image signal stored in the on-chip RAM (for example, the number of lines of the stored image signal reaches the preset threshold, the size of the written image signal reaches the preset threshold, or the last address in the storage address assigned to the ISP101 is stored There is an image signal), stop continuing to transmit the image signal to the on-chip RAM, and send an interrupt signal Z1 to the AI processor through the electronic circuit connection L1 as shown in Figure 3.
  • the AI processor 101 can store the image signal in the on-chip RAM through the same storage method as the ISP 102, and after storing the image signal in the on-chip RAM, send the interrupt signal Z2 to the AI processor through the electronic circuit connection L2 as shown in FIG. 3 .
  • the on-chip RAM can be logically divided into a first storage area and a second storage area, the ISP102 stores the image signal in the form of image blocks in the first storage area, and the AI processor 101 stores the image signal in the form of image blocks in the first storage area. Second memory. In this way, the image processing by the ISP 102 and the image processing by the AI processor 101 can be performed in parallel, the waiting time of the AI processor 101 and the ISP 102 can be reduced, and the transmission rate of the image signal can be improved.
  • the ISP102 can also send to the AI processor 101 a message indicating that the current image block is the starting position of a frame of image. indication signal.
  • the image processing module 02 in the ISP 102 has an electronic circuit connection L7 with the AI processor 101 , and sends an interrupt signal Z8 to the AI processor 101 .
  • the interrupt signal Z8 is used to indicate that the image signal A is the starting position of a frame of image.
  • the electronic device further includes an off-chip memory 106, as shown in FIG. 8 .
  • the off-chip memory 106 may store multiple frames of images, and the multiple frames of images may be the previous frame image, the previous two frame images, or the previous multi-frame images before the current image. Since the off-chip memory 106 has a larger storage space, it can replace the on-chip RAM and be used to store larger units of image data.
  • the image signal stored in the off-chip memory 106 may be an image signal processed by the AI processor 101 , or an image signal provided to the AI processor 101 for processing by the AI processor 101 .
  • the AI processor 101 When the AI processor 101 processes the current image signal, it can also obtain the image information of the previous frame of the current image signal or the previous several frames of the image signal from the off-chip memory 106, and then based on the previous frame of the image signal or the previous several frames. The image information of the image signal processes the current image signal. In addition, the AI processor 101 can also store the processed image signal in the off-chip memory 106 .
  • the off-chip memory 106 may include random access memory (RAM), which may include volatile memory (eg, SRAM, DRAM, DDR (Double Data Rate SDRAM), or SDRAM, etc.) and non-volatile memory.
  • the off-chip memory 106 may store an executable program of the image processing model running in the AI processor 101, and the AI processor loads the executable program to run the image processing model.
  • the memory 103 such as an on-chip RAM
  • the memory 103 may be used to store a single-frame image or image blocks in a single-frame image.
  • the off-chip memory 106 is used to store multiple frames of images. Therefore, for the method shown in FIG. 2 , FIG. 5 or FIG. 7 , the image signal transmitted between the AI processor 101 and the AI processor 101 is divided by unit size, and the image information of a larger unit is transmitted through the off-chip memory 106 , make up for the lack of on-chip RAM space, and effectively utilize the faster transmission speed of on-chip RAM to achieve performance optimization.
  • the electronic device 100 may further include a controller 104, as shown in FIG. 8 .
  • the controller 104 may be an integrated controller located on the same chip where the AI processor 101 , the ISP 102 and the memory 103 are located.
  • the controller 104 runs the necessary software programs or software plug-ins to drive the controller 101 to control the operation of the ISP 102, control the operation of the AI processor 101, and control the communication between the ISP 102 and the AI processor 101, so as to realize FIG. 2, FIG. 5 or
  • the image processing method shown in FIG. 6 replaces the electronic circuit connection between the AI processor 101 and the ISP 102 as a communication medium.
  • the controller 104 may be various digital logic devices or circuits, including but not limited to: CPU, GPU, microcontroller, microprocessor or DSP, and so on.
  • the controller 104 may also be provided separately from the AI processor 101 , the ISP 10 and the memory 10 , which is not limited in this embodiment.
  • the controller 104 and the AI processor 101 may also be integrated in the same logic operation device (eg, CPU), and the same logic operation device can implement the controller 104 and the AI processor 101 described in the embodiments of the present application. function performed.
  • the controller 104 can determine whether to use the ISP 102 in combination with the AI processor 101 to determine whether to use the ISP 102 in combination with the AI processor 101 based on various information (such as light intensity information, white balance information, or mosaic information, etc.) indicated by the image data collected by the camera 105 .
  • the image data is processed, and the image processing model selected when the ISP 102 is selected to be combined with the AI processor 101 to process the image data is determined.
  • the control of the above components and the processing of images by the controller 104 will be described in detail below by taking the light intensity information as an example.
  • the controller 104 Based on the light intensity information indicated by the image data, the controller 104 compares the information with the preset image light intensity threshold, and based on the comparison result, determines that the image light intensity threshold cannot be reached after the ISP102 is used to process the image light intensity.
  • the AI processor 101 processes the image data or the light intensity of the image signal.
  • the AI processor 101 can run a variety of image processing models for light intensity processing.
  • the controller determines which image processing model to use based on the difference between the light intensity information indicated by the image data and the image light intensity threshold. After the controller 104 determines the selected image processing model, it can issue the storage address information of the algorithm program, parameters or instructions in the above-mentioned memory 103 or the off-chip memory 106 for storing the image processing model to the AI processor 101.
  • the controller may also send sequence information or priority information indicating the operation of the multiple image processing models to the AI processor 101 . It should be noted that, in the embodiment of the present application, the controller 104 may detect various information indicated by the image data in real time or periodically, and when it is determined that the currently running image detection model is not applicable based on the detection results, the controller 104 may promptly replace all kinds of information indicated by the image data.
  • the selected image processing model (for example, when the ambient light intensity changes from strong to weak, replace the image processing model used to process the light intensity), and then send the storage address information of the executable program of the replaced image processing model to the AI processor 101, so that the AI processor 101 runs the replaced image processing model when performing image processing in the next image processing cycle. Therefore, the electronic device described in the embodiments of the present application can dynamically adjust the adopted image processing model based on changes in the external environment or changes in the collected image data, so that the user can use the electronic device described in the embodiments of the present application.
  • changing scenes for example, changing from outdoor to indoor or from a strong light area to a weak light area
  • the collected images are processed in a targeted manner to improve the image processing effect and help improve the user experience.
  • the image data generally has a preset clock cycle T from the start of processing to the generation of the final image processing result.
  • the AI processor takes different time periods in the clock cycle T when it executes the image processing procedures.
  • the clock cycle T from the start of image data processing to the generation of the final image processing result is 33.3ms, while the AI processor performs image processing for 18ms, that is to say, the AI processor runs at half the clock cycle T idle for a period of time.
  • the controller 104 may also determine the running duration of the adopted image processing model, when the clock period T is determined When the running time of the internal image processing model is less than the preset threshold, the controller 104 may also allocate the idle time of the AI processor to other AI services.
  • the other AI services may include, but are not limited to, biometrics (eg, facial recognition or fingerprint recognition) services, and services for adding special effects to images (eg, adding objects to images).
  • the executable program or parameters of the AI model for executing the other AI service may also be stored in the above-mentioned off-chip memory 106 . Further, when the storage capacity of the memory 103 is large enough, the executable programs or parameters of the AI model of the other AI service can also be stored in the memory 103 .
  • the controller 104 may include multiple independent controllers, each of which may be a digital logic device (eg, including but not limited to: GPU or DSP) .
  • the multiple independent controllers include an ISP controller for controlling the operation of each component in the ISP 102 and an AI controller for controlling the operation of each component in the AI processor 101 .
  • the AI controller can be integrated inside the AI processor 101 .
  • the ISP controller and the AI controller can transmit information by means of inter-core communication. For example, after determining the selected image processing model based on the image data, the ISP controller can send various configuration information to the AI controller before the image processing starts.
  • the configuration information can include but is not limited to: the executable program of the image processing model is stored in the memory 103 or the off-chip memory 106, or the priority information of each image processing model in the plurality of image processing models, and the like.
  • the information transmission between the AI processor 101 and the ISP 102 can also be realized through the communication between the ISP controller and the AI controller. For example, when there is no electronic circuit connection between the AI processor 101 and the ISP102 to transmit the interrupt signal, the ISP102 can store the image signal in the on-chip RAM and notify the ISP controller, and the ISP controller will instruct the image signal to be stored in the on-chip RAM.
  • the AI controller controls the computing unit in the AI processor 101 to read the image signal from the on-chip RAM for image processing; the computing unit in the AI processor 101 stores the image signal in the on-chip RAM and notifies the AI controller.
  • the AI controller sends the information indicating that the image signal is stored in the on-chip RAM to the ISP controller, and the ISP controller ISP102 reads the image signal from the on-chip RAM for processing.
  • the electronic device 100 may further include a communication unit (not shown in the figure), where the communication unit includes but is not limited to a short-range communication unit or a cellular communication unit.
  • the short-range communication unit performs information exchange with a terminal located outside the mobile terminal for accessing the Internet by running a short-range wireless communication protocol.
  • the short-range wireless communication protocol may include, but is not limited to, various protocols supported by radio frequency identification technology, Bluetooth communication technology protocols, or infrared communication protocols.
  • the cellular communication unit is connected to the Internet by running the cellular wireless communication protocol and the wireless access network, so as to realize the information exchange between the mobile communication unit and the server supporting various applications in the Internet.
  • the communication unit may be integrated in the same SOC with the AI processor 101 and the ISP 102 described in the above embodiments, or may be provided separately.
  • the electronic device 100 may optionally include a bus, an input/output port I/O, a memory controller, and the like.
  • the memory controller is used to control the memory 103 and the off-chip memory 106 .
  • the bus, the input/output port I/O, and the storage controller, etc. can be integrated into the same SOC with the above-mentioned ISP 102 and AI processor 101 and the like.
  • the electronic device 100 may include more or less components than those shown in FIG. 1 or FIG. 8 , which is not limited in this embodiment of the present application.
  • the electronic device includes corresponding hardware and/or software modules for executing each function.
  • the steps of each example described in conjunction with the embodiments disclosed herein can be implemented in hardware or in a combination of hardware and computer software. Whether a function is performed by hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functionality for each particular application in conjunction with the embodiments, but such implementations should not be considered beyond the scope of this application.
  • the above-mentioned one or more processors may be divided into functional modules according to the foregoing method examples. For example, different processors may be divided corresponding to each function, or two or more processors of functions may be integrated in in a processor module.
  • the above-mentioned integrated modules can be implemented in the form of hardware. It should be noted that, the division of modules in this embodiment is schematic, and is only a logical function division, and there may be other division manners in actual implementation.
  • FIG. 9 shows a possible schematic diagram of the apparatus 900 involved in the above-mentioned embodiment, and the above-mentioned apparatus can be further expanded.
  • the apparatus 900 may include: an AI processing module 901 and an image signal processing module 902 .
  • the AI processing module 901 is configured to perform first image signal processing on the first image signal to obtain a second image signal, where the first image signal is obtained based on image data output by the image sensor; the image signal processing module 902, for performing second image signal processing on the second image signal to obtain the image processing result.
  • the image signal processing module 902 is further configured to: receive the image data from the image sensor, and perform third image signal processing on the image data to obtain the first image signal.
  • the third image signal processing includes multiple processing procedures, and in two adjacent processing procedures among the multiple processing procedures, the previous processing procedure is used to generate the third image signal , the latter processing process is used to process the fourth image signal; the AI processing module 901 is further configured to perform fourth image signal processing on the third image signal to obtain the fourth image signal.
  • the first image signal processing includes at least one of the following processing procedures: noise removal, black level correction, shadow correction, white balance correction, demosaicing, chromatic aberration correction or gamma correction.
  • the second image signal processing includes at least one of the following processing procedures: noise removal, black level correction, shadow correction, white balance correction, demosaicing, chromatic aberration correction, gamma correction, and chromatic aberration correction Or RGB to YUV domain.
  • the third image signal processing includes at least one of the following processing procedures: noise removal, black level correction, shadow correction, white balance correction or demosaicing.
  • the fourth image signal processing includes at least one of the following processing procedures: black level correction, shadow correction, white balance correction, demosaicing or chromatic aberration correction.
  • the image processing apparatus 900 provided in this embodiment is configured to execute the image processing method executed by the electronic apparatus 100, and can achieve the same effect as the above-mentioned implementation method or apparatus.
  • each module corresponding to the above FIG. 9 may be implemented by software, hardware or a combination of the two.
  • each module may be implemented in the form of software, corresponding to the corresponding processor corresponding to the module in FIG. 1 , for driving the corresponding processor to work.
  • each module may include a corresponding processor and a corresponding driver software, that is, implemented in a combination of software or hardware. Therefore, the image processing apparatus 900 can be considered to logically include the apparatuses shown in FIG. 1 , FIG. 3 , FIG. 4 , FIG. 7 or FIG. 8 , and each module includes at least a driver software program for the corresponding function. This embodiment Do not expand on this.
  • the image processing apparatus 900 may include at least one processor and a memory, with specific reference to FIG. 1 .
  • at least one processor can call all or part of the computer program stored in the memory to control and manage the actions of the electronic device 100, for example, can be used to support the electronic device 100 to perform the steps performed by the above-mentioned modules.
  • the memory may be used to support the execution of the electronic device 100 by storing program codes and data, and the like.
  • At least one processor can implement or execute various exemplary logic modules described in conjunction with the present disclosure, which can be a combination of one or more microprocessors that implement computing functions, such as, but not limited to, those shown in FIG. 1 .
  • the AI processor 101 and the image signal processor 102 are shown.
  • the at least one processor may also include other programmable logic devices, transistor logic devices, or discrete hardware components, or the like.
  • the memory in this embodiment may include, but is not limited to, the off-chip memory 106 or the memory 103 shown in FIG. 8 .
  • This embodiment also provides a computer-readable storage medium, where computer instructions are stored in the computer-readable storage medium, and when the computer instructions are executed on the computer, the computer executes the above-mentioned relevant method steps to realize the image processing in the above-mentioned embodiments. method.
  • This embodiment also provides a computer program product, which when the computer program product runs on a computer, causes the computer to execute the above-mentioned relevant steps, so as to realize the image processing method in the above-mentioned embodiment.
  • the computer-readable storage medium or computer program product provided in this embodiment is used to execute the corresponding method provided above. Therefore, for the beneficial effect that can be achieved, reference may be made to the corresponding method provided above. The beneficial effects will not be repeated here.
  • each functional unit in each embodiment of the present application may be integrated into one product, or each unit may physically exist alone, or two or more units may be integrated into one product.
  • each functional unit in each embodiment of the present application may be integrated into one product, or each unit may physically exist alone, or two or more units may be integrated into one product.
  • FIG. 9 if the above modules are implemented in the form of software functional units and sold or used as independent products, they may be stored in a readable storage medium.
  • the technical solutions of the embodiments of the present application can be embodied in the form of software products in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, which are stored in a storage medium , including several instructions to make a device (which may be a single chip microcomputer, a chip, etc.) or a processor (processor) to execute all or part of the steps of the methods in the various embodiments of the present application.
  • aforementioned readable storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc. that can store program codes. medium.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)

Abstract

本申请实施例提供了一种电子装置和电子装置的图像处理方法，该电子装置包括：人工智能AI处理器，用于对第一图像信号执行第一图像信号处理以得到第二图像信号，所述第一图像信号是基于图像传感器输出的图像数据获得的；图像信号处理器ISP，用于对第二图像信号执行第二图像信号处理以得到图像处理结果，从而可以灵活的将ISP对图像的处理和AI处理器结合，有利于提高图像处理结果。

Description

电子装置和电子装置的图像处理方法 技术领域
本申请实施例涉及电子技术领域，尤其涉及一种电子装置和电子装置的图像处理方法。
背景技术
随着电子科学技术的进步，智能终端集成有越来越多的功能，得益于图像处理技术的发展，越来越多的用户喜爱利用智能终端设备进行拍照、视频录制和视频通话等。
由于受限于智能终端内图像信号处理器(ISP，Image Signal Processor)的算法的运算能力，为了提高图像处理效果，业界提出了一种将传统的图像处理算法与人工智能(AI，Artificial Intelligence)技术相结合。例如，AI处理器被设置于ISP的后端作为对ISP图像处理结果的补充修正，即进行AI后处理。具体实现中，ISP将处理完毕的图像存储至片外存储器，AI处理器从片外存储器读取ISP存储的图像，在ISP图像处理的基础上对图像进行进一步修正，从而生成最终的图像。在这种方案中，由于传统ISP的能力具有瓶颈，导致图像传感器采集的原始图像数据经ISP处理后出现信息丢失，进而降低图像处理效果。AI后处理技术虽然在一定程度上能改善ISP输出的图像处理结果，但效果依然不理想。由此，现有技术中未能充分解决传统ISP的处理效果不佳的问题。
发明内容
本申请提供的电子装置和电子装置的图像处理方法，可以提高图像处理效果。为达到上述目的，本申请采用如下技术方案。
第一方面，本申请实施例提供一种电子装置，所述电子装置包括：人工智能AI处理器，用于对第一图像信号执行第一图像信号处理以得到第二图像信号，所述第一图像信号是基于图像传感器输出的图像数据获得的；图像信号处理器ISP，用于对第二图像信号执行第二图像信号处理以得到图像处理结果。
通过在AI处理器对图像数据图像处理后再由ISP执行其余图像处理，在整个图像信号处理流程中可充分利用AI处理器的处理能力，使得AI处理器代替传统ISP执行部分处理过程，可以提高ISP输出的图像处理结果的质量。
基于第一方面，在一种可能的实现方式中，所述ISP还用于：从所述图像传感器接收所述图像数据，并对所述图像数据执行第三图像信号处理以得到所述第一图像信号。
该实现方式可以使得将AI处理器所执行的第一图像处理过程设置在ISP所执行的多个图像处理过程之间，AI处理器可以替代ISP执行图像处理过程中间的一部分处理过程以达到预设效果，提高AI处理器与ISP结合的灵活性，从而提高图像处理效果。
基于第一方面，在一种可能的实现方式中，所述第三图像信号处理包括多个处理过程，在所述多个处理过程中的两个相邻处理过程中，前一个处理过程用于产生第三图像信号，后一个处理过程用于处理第四图像信号；所述AI处理器，还用于对所述第三图像信号执 行第四图像信号处理以得到所述第四图像信号。该方案进一步提高了利用AI处理器执行图像处理过程的灵活性。
基于第一方面，在一种可能的实现方式中，所述第一图像信号处理包括如下至少一个处理过程：噪声消除、黑电平校正、阴影矫正、白平衡校正、去马赛克、色差校正或者伽马矫正。
基于第一方面，在一种可能的实现方式中，所述第二图像信号处理包括如下至少一个处理过程：噪声消除、黑电平矫正、阴影矫正、白平衡校正、去马赛克、色差矫正、伽马矫正、色差校正或者RGB转YUV域。
基于第一方面，在一种可能的实现方式中，所述第三图像信号处理包括如下至少一个处理过程：噪声消除、黑电平矫正、阴影矫正、白平衡校正或者去马赛克。
基于第一方面，在一种可能的实现方式中，所述第四图像信号处理包括如下至少一个处理过程：黑电平矫正、阴影矫正、白平衡校正、去马赛克或者色差矫正。
基于第一方面，在一种可能的实现方式中，所述电子装置还包括：存储器，耦合于所述AI处理器和所述ISP，用于在所述AI处理器和所述ISP之间转移任一图像信号中的第一图像单元。
基于第一方面，在一种可能的实现方式中，所述存储器、所述AI处理器和所述ISP位于所述电子装置中的一个片上系统内；所述存储器包括片上随机存取存储器RAM。
基于第一方面，在一种可能的实现方式中，所述第一图像单元包括如下任一项：单帧图像或者单帧图像中的图像块。
基于第一方面，在一种可能的实现方式中，所述电子装置还包括：片外存储器，位于所述片上系统之外，用于在所述AI处理器和所述ISP之间转移任一图像信号中的第二图像单元，所述第二图像单元包括多帧图像。
基于第一方面，在一种可能的实现方式中，所述电子装置还包括：控制器，用于触发所述AI处理器执行所述第一图像信号处理，以及控制所述ISP执行所述第二图像信号处理。
基于第一方面，在一种可能的实现方式中，所述AI处理器和所述ISP之间通过电子线路连接传输中断信号。可选地，该电子线路连接包括通过中断控制器连接。通过在AI处理器和ISP之间设置电子线路连接实现中断信号传输，不需要经过其他处理器例如CPU的转发，可以提高信号传输速度，在某些实时播放的视频中，可以降低图像输出时延，有利于提高用户体验。
可选地，该电子装置还包括所述图像传感器。
第二方面，本申请实施例提供一种电子装置的图像处理方法，该图像处理方法包括：控制人工智能AI处理器对第一图像信号执行第一图像信号处理以得到第二图像信号，所述第一图像信号是基于图像传感器输出的图像数据获得的；控制图像信号处理器ISP对第二图像信号执行第二图像信号处理以得到图像处理结果。
基于第二方面，在一种可能的实现方式中，在控制人工智能AI处理器对第一图像信号执行第一图像信号处理之前，还包括：控制ISP从所述图像传感器接收所述图像数据，并对所述图像数据执行第三图像信号处理以得到所述第一图像信号。
基于第二方面，在一种可能的实现方式中，所述第三图像信号处理包括多个处理过 程，在所述多个处理过程中的两个相邻处理过程中，前一个处理过程用于产生第三图像信号，后一个处理过程用于处理第四图像信号；所述方法还包括：控制所述AI处理器对所述第三图像信号执行第四图像信号处理以得到所述第四图像信号。
基于第二方面，在一种可能的实现方式中，所述第一图像信号处理包括如下至少一个处理过程：噪声消除、黑电平校正、阴影矫正、白平衡校正、去马赛克、色差校正或者伽马矫正。
基于第二方面，在一种可能的实现方式中，所述第二图像信号处理包括如下至少一个处理过程：噪声消除、黑电平矫正、阴影矫正、白平衡校正、去马赛克、色差矫正、伽马矫正、色差校正或者RGB转YUV域。
基于第二方面，在一种可能的实现方式中，所述第三图像信号处理包括如下至少一个处理过程：噪声消除、黑电平矫正、阴影矫正、白平衡校正或者去马赛克。
基于第二方面，在一种可能的实现方式中，所述第四图像信号处理包括如下至少一个处理过程：黑电平矫正、阴影矫正、白平衡校正、去马赛克或者色差矫正。
第三方面，本申请实施例提供一种图像处理装置，该图像处理装置包括：AI处理模块，用于对第一图像信号执行第一图像信号处理以得到第二图像信号，所述第一图像信号是基于图像传感器输出的图像数据获得的；图像信号处理模块，用于对第二图像信号执行第二图像信号处理以得到图像处理结果。
基于第三方面，在一种可能的实现方式中，图像信号处理模块还用于：从所述图像传感器接收所述图像数据，并对所述图像数据执行第三图像信号处理以得到所述第一图像信号。
基于第三方面，在一种可能的实现方式中，所述第三图像信号处理包括多个处理过程，在所述多个处理过程中的两个相邻处理过程中，前一个处理过程用于产生第三图像信号，后一个处理过程用于处理第四图像信号；所述AI处理模块，还用于对所述第三图像信号执行第四图像信号处理以得到所述第四图像信号。
基于第三方面，在一种可能的实现方式中，所述第一图像信号处理包括如下至少一个处理过程：噪声消除、黑电平校正、阴影矫正、白平衡校正、去马赛克、色差校正或者伽马矫正。
基于第三方面，在一种可能的实现方式中，所述第二图像信号处理包括如下至少一个处理过程：噪声消除、黑电平矫正、阴影矫正、白平衡校正、去马赛克、色差矫正、伽马矫正、色差校正或者RGB转YUV域。
基于第三方面，在一种可能的实现方式中，所述第三图像信号处理包括如下至少一个处理过程：噪声消除、黑电平矫正、阴影矫正、白平衡校正或者去马赛克。
基于第三方面，在一种可能的实现方式中，所述第四图像信号处理包括如下至少一个处理过程：黑电平矫正、阴影矫正、白平衡校正、去马赛克或者色差矫正。
第四方面，本申请实施例提供一种电子装置，所述电子装置包括存储器和至少一个处理器，所述存储器用于存储计算机程序，所述至少一个处理器被配置用于调用所述存储器存储的全部或部分计算机程序，执行上述第二方面所述的方法。所述至少一个处理器包括所述AI处理器和ISP。可选地，该电子装置还包括所述图像传感器。
第五方面，本申请实施例提供一种片上系统，所述片上系统包括至少一个处理器和 接口电路，所述接口电路用于从所述芯片系统外部获取计算机程序；所述计算机程序被所述至少一个处理器执行时用于实现上述第二方面所述的方法。所述至少一个处理器包括所述AI处理器和ISP。
第六方面，本申请实施例提供一种计算机可读存储介质，所述计算机可读存储介质存储中存储有计算机程序，该计算机程序被至少一个处理器执行时用于实现如第二方面所述的方法。所述至少一个处理器包括所述AI处理器和ISP。
第七方面，本申请实施例提供一种计算机程序产品，当所述计算机程序产品被至少一个处理器执行时用于实现上述第二方面所述的方法。所述至少一个处理器包括所述AI处理器和ISP。
应当理解的是，本申请的第二至七方面与本申请的第一方面的技术方案一致，各方面及对应的可行实施方式所取得的有益效果相似，不再赘述。
附图说明
为了更清楚地说明本申请实施例的技术方案，下面将对本申请实施例的描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。
图1是本申请实施例提供的电子装置的一个硬件结构示意图；
图2是本申请实施例提供的图像处理方法的一个示意性流程图；
图3是本申请实施例提供的电子装置的又一个硬件结构示意图；
图4是本申请实施例提供的电子装置的又一个硬件结构示意图；
图5是本申请实施例提供的图像处理方法的又一个示意性流程图；
图6是本申请实施例提供的图像处理方法的又一个示意性流程图；
图7是本申请实施例提供的电子装置的又一个硬件结构示意图；
图8是本申请实施例提供的电子装置的又一个硬件结构示意图；
图9是本申请实施例提供的电子装置的软件结构示意图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
本文所提及的"第一"、或"第二"以及类似的词语并不表示任何顺序、数量或者重要性，而只是用来区分不同的部分。同样，"一个"或者"一"等类似词语也不表示数量限制，而是表示存在至少一个。"耦合"等类似的词语并非限定于物理的或者机械的直接连接，而是可以包括电性的连接，不管是直接的还是间接的，等同于广义上的联通。
在本申请实施例中，“示例性的”或者“例如”等词用于表示作例子、例证或说明。本申请实施例中被描述为“示例性的”或者“例如”的任何实施例或设计方案不应被解释为比其它实施例或设计方案更优选或更具优势。确切而言，使用“示例性的”或者“例如”等词旨在 以具体方式呈现相关概念。在本申请实施例的描述中，除非另有说明，“多个”的含义是指两个或两个以上。例如，多个处理器是指两个或两个以上的处理器。
本申请实施例提供的电子装置，可以是个电子设备或集成于电子设备内的模组、芯片、芯片组、电路板或部件。该电子设备可以是一个用户设备(User Equipment,UE)，如手机、平板电脑、智能屏幕或者图像拍摄设备等各种类型的设备。该电子设备可以设置有摄像装置，该摄像装置也可以称为图像传感器，以用于采集图像数据。该电子设备还可以安装有诸如摄像类应用、视频通话类应用或者在线视频拍摄类应用等各种用于驱动摄像装置采集图像的软件应用，用户可以通过启动上述各类应用以利用摄像装置拍摄照片或视频。此外，用户还可以通过该类应用进行各种图像美化的个性化设置，以视频通话类应用为例，用户可以在视频通话时选择对屏幕呈现的画面(例如所呈现的面部头像、或所呈现的背景画面)进行自动调节(例如“一键美化”)。当用户启动上述各类应用后或者启动上述各类应用且选择图像美化后，电子设备中对上述各类应用所支持的图像处理服务可以触发电子设备对摄像装置所采集的图像数据进行处理，从而在电子设备的屏幕中呈现处理后的图像，以达到图像美化的效果。上述图像美化例如可以包括但不限于：提高图像局部或者整个画幅的亮度、更改图像的显示颜色、对图像中呈现的面部对象磨皮、调节画面饱和度、调节画面曝光度、调节画面鲜明度、调节画面高光、调节画面对比度、调节画面锐度或者调节画面清晰度等。本申请实施例所述的图像处理可以包括但不限于：噪声去除、黑电平矫正、阴影矫正、白平衡校正、去马赛克、色差矫正、伽马Gamma矫正或者红绿蓝(RGB)转YUV(YCrCb)域，从而达到上述图像美化的效果。基于本申请实施例所述的电子装置，在一个具体的应用场景中，当用户A启动上述图像处理服务后，用户A与用户B之间进行视频通话时，呈现在用户A使用的电子设备屏幕中的图像，以及呈现在用户B使用的电子设备屏幕中的用户A的图像，可以是经过本申请实施例所述的电子装置处理后的图像，并且会一直呈现处理后的图像直到用户A与用于B终止视频通话或者用户A关闭图像处理服务。
基于如上所述的应用场景，请继续参考图1，其示出了本申请实施例提供的电子装置的一个硬件结构示意图。电子装置100例如具体可以是芯片或芯片组或搭载有芯片或芯片组的电路板或包括所述电路板的电子设备，但不用于限定实施例，具体的电子设备如前面的介绍，此处省略。该芯片或芯片组或搭载有芯片或芯片组的电路板可在必要的软件驱动下工作。电子装置100包括一个或多个处理器，例如ISP102和AI处理器101。可选地，所述一个或多个处理器可以集成在一个或多个芯片内，该一个或多个芯片可以被视为是一个芯片组，当一个或多个处理器被集成在同一个芯片内时该芯片也叫片上系统(System on a Chip，SOC)。在所述一个或多个处理器之外，电子装置100还包括一个或多个其他必要部件，例如存储器103。在一种可能的实现方式中，存储器103可以与AI处理器101和ISP102位于电子装置100中的同一个片上系统内，也即存储器103集成于如上图1所示的SOC中。此时，存储器103可以包括片上随机存取存储器(RAM，Random Access Memory)。
在本申请实施例中，AI处理器101可以包括神经网络处理器(Neural-network Processing Unit,NPU)等专用神经处理器，包括但不限于卷积神经网络处理器、张量处理器或神经处理引擎。AI处理器可以单独作为一个部件或集成于其他数字逻辑器件中，该数字 逻辑器件包括但不限于：CPU(中央处理器，Central Processing Unit)、GPU(图形处理器，Graphics Processing Unit)或者DSP(数字信号处理器，Digital Signal Processing)。示例性地，该CPU、GPU和DSP都是片上系统内的处理器。AI处理器101可以执行一种或多种图像处理操作，该一种或多种图处理操作可以包括但不限于：噪声消除、黑电平矫正、阴影矫正、白平衡校正、去马赛克、色差矫正或者Gamma矫正。AI处理器101可以运行一种或多种图像处理模型，其中每一种图像处理模型用于执行某种特定的图像处理操作。例如，噪声消除的图像处理模型用于执行噪声消除的图像处理操作，去马赛克的图像处理模型用于执行去马赛克的图像处理操作。每一种图像处理模型均可以是采用传统神经网络训练方法，利用训练样本对神经网络训练得到，本申请实施例对此不再赘述。ISP102可以设置多个硬件模块或者运行必要的软件程序以对图像进行处理或者与AI处理器101进行通信。其中，ISP102和AI处理器101可以通过硬件直连的方式通信(例如图3、图4和图7所示的实施例中的相关描述)，也可以通过控制器进行信号转发的方式通信，其中ISP102和AI处理器101之间通过控制器进行信号转发的方式进行通信的相关描述具体参考图8中所示的对控制器104的相关描述。
本申请实施例中，从摄像装置105获取的图像数据可以经过多个图像处理过程以生成最终的图像处理结果，该多个图像处理过程可以包括但不限于：噪声去除、黑电平矫正、阴影矫正、白平衡校正、去马赛克、色差矫正、Gamma矫正或者RGB转YUV域。AI处理器101可以执行上述图像处理过程中的一个过程或多个过程，也即对应上述一种或多种图像处理操作，ISP102也可以执行上述图像处理过程中的一个过程或多个过程。其中，AI处理器101可以与ISP102执行不同的图像处理过程，此外，AI处理器101与ISP102也可以执行相同的图像处理过程，例如进行进一步的增强处理，本实施例对此不限定。当AI处理器101和ISP102执行相同的图像处理过程时，AI处理器101所执行的图像处理可以作为对该图像处理过程的增强或补充。例如，当AI处理器101和ISP102同时执行噪声消除的过程时，ISP102用于进行初次去噪，AI处理器101用于在ISP102初次去噪的基础上进行二次去噪。因此，整个图像处理流程包括多个处理过程，并被作为任务分配给AI处理器101和ISP102，相当于在ISP102完成所有处理过程之前，由AI处理器101代替传统ISP执行部分处理过程，最终的处理结果由ISP102输出。
本申请实施例所示的电子装置100，在AI处理器101对图像数据执行一个或多个图像处理过程后再由ISP102执行其余图像处理过程，可以用AI处理器101代替ISP102执行部分处理，避免ISP102在图像处理过程中处理能力不足导致的原始图像数据的信息丢失，提高图像处理效果。此外，在其他一些实现方式中，当传统ISP中的某些图像处理过程无法达到预设效果时，可以将AI处理器101所执行的图像处理过程设置在ISP102所执行的多个图像处理过程之间(如实施例图4-图7)来替代ISP执行该图像处理过程以达到预设效果，提高AI处理器与ISP结合的灵活性。
下面对AI处理器101与ISP102结合进行图像处理的过程以及各图像处理过程对应的电子装置的硬件结构进行详细描述。请参考图2，其示出了本申请实施例提供的图像处理方法200的一个示意性流程图。该图像处理方法200应用于图1或者图3所示的电子装置100。该图像处理方法200包括如下图像处理步骤：步骤201，摄像装置105将采集的图像数据提供至ISP102。步骤202，ISP102对图像数据进行处理，生成图像信号A。步骤203， ISP102将图像信号A提供至AI处理器101。步骤204，AI处理器101对图像信号A进行图像处理，生成图像信号B。步骤205，AI处理器101将图像信号B提供至ISP102。步骤206，ISP102对图像信号B进行图像处理，得到图像处理结果。
基于图2所示的图像处理步骤，请继续参考图3，其示出了本申请实施例提供的电子装置100的又一个结构示意图。在如图3所示的电子装置100中，ISP102可以包括多个级联的图像处理模块，该多个级联的图像处理模块包括图像处理模块01、图像处理模块02、图像处理模块03…图像处理模块N以及图像处理模块N+1，每一个图像处理模块均可以包括多个逻辑器件或电路以执行特定的图像处理功能。例如，图像处理模块01用于执行黑电平矫正的图像处理，图像处理模块02用于执行阴影矫正的图像处理，图像处理模块03用于执行阴影校正的图像处理…，图像处理模块N+1用于执行RGB转YUV的处理。基于对图像的处理需求，上述多个级联的图像处理模块中的任意一个图像处理模块可以设置有输出端口和输入端口，该输出端口用于向AI处理器101发送图像信号A，该输入端口用于从AI处理器101获取图像信号B，图3中示意性的示出了图像处理模块02设置有输出端口V po1、图像处理模块03设置有输入端口V pi1。在图3中，ISP102与摄像装置105连接，以从摄像装置105获取图像数据。电子装置100还设置有片上RAM，与ISP102和AI处理器101集成在电子装置100中的一个芯片中，该片上RAM用于存储图2所示的图像信号A和图像信号B。此外，片上RAM还用于存储AI处理器101运行过程中所产生的中间数据以及AI处理器101所运行的神经网络中各网络节点的权重数据等。在图3中，为了提高信号传输速度，图像处理模块02的输出端口V po1、图像处理模块03的输入端口V pi1以及AI处理器101的输入端口V ai1、输出端口V ao1与片上RAM之间均可以采用电子线路连接。
进一步的，在一种可能的实现方式中，AI处理器101与ISP102中的图像处理模块02之间通过电子线路连接L1传输中断信号Z1，该中断信号Z1用于指示图像处理模块02将图像信号A存储至片上RAM；AI处理器101与ISP102中的图像处理模块03之间通过电子线路连接L2传输中断信号Z2，该中断信号Z2用于指示AI处理器101将图像信号B存储至片上RAM。具体的，如图3所示的AI处理器可以包括任务调度器1011和多个计算单元1012。其中任务调度器1011和多个计算单元1012中的每一个组件均可以包括多个逻辑器件或电路。任务调度器1011可以通过AI处理器101的输入端口V ai2与图像处理模块02的输出端Vp o2实现上述电子线路连接L1，通过AI处理器101的输出端口V ao2与图像处理模块03的输入端Vpi2实现上述电子线路连接L2。此外，多个计算单元中的每一个计算单元可以通过输入端口V ai1与上述片上RAM实现电子线路连接，以从片上RAM读取图像信号A；通过输出端口V ao1与上述片上RAM进行电子线路连接，以向片上RAM写入图像信号B。
在一个具体的场景中，ISP102从摄像装置105获取图像数据，图像数据通过图像处理模块01、图像处理模块02依次执行阴影矫正和白平衡校正处理后生成图像信号A存储至片上RAM。图像处理模块02将图像信号A存储至片上RAM后通过电子线路连接L1向AI处理器101发送中断信号Z1。AI处理器101响应于中断信号Z1从片上RAM获取图像信号A。AI处理器101对图像信号A进行去马赛克处理后生成图像信号B，以及将图像信号B存储至片上RAM。AI处理器101将图像信号B存储至片上RAM后向图像处理 模块03发送上述中断信号Z2。图像处理模块03响应于中断信号Z2，从片上RAM读取图像信号B，图像信号B经过ISP102中的图像处理模块03…、图像处理模块N以及图像处理模块N+1依次执行色差矫正、…Gamma矫正以及RGB转YUV域的处理后生成最终的图像处理结果。需要说明的是，在图像处理模块01之前还可以包括更多的图像处理模块，以使得ISP102对图像数据执行更多的图像处理过程。
在一个实施例中，AI处理器101与ISP102之间的电子线路连接也叫物理连接或中断连接，AI处理器101与ISP102通过该连接实现中断信号的发送和接收，使得中断信号不需要经过其他处理器例如CPU的转发，也不需要CPU参与相关控制，可以提高中断信号传输速度，在某些实时播放的视频中，可以降低图像输出时延，有利于提高用户体验。具体地，该中断连接包括用于实现中断信号发送和接收功能的中断信号处理硬件电路和传输信号的连接线，以实现中断信号的收发。中断信号处理硬件电路包括但不限于传统的中断控制器电路。关于中断信号处理硬件电路的具体实现方案，可以参照现有技术中的中断控制器的相关描述，此处不做赘述。
在图2和图3所示的实施例中，AI处理器101所执行的图像处理过程设置在ISP102所执行的多个图像处理过程之间，来替代或补充ISP102所执行的某些中间的图像处理过程。在其他一些可能的实现方式中，AI处理器101可以直接从摄像装置105获取图像数据，执行前端的图像处理过程。在该实现方式中，AI处理器101可以替代ISP102中前端的某些图像处理模块，执行相应的图像处理过程，此时，AI处理器101可以直接与ISP102后面的图像处理模块通信。该实现方式的硬件结构参考图4，请参考图4，其示出了本申请实施例提供的又一个硬件结构示意图。
在图4中，ISP102的结构和AI处理器101的结构与图3所示的ISP102的结构和AI处理器101的结构相同，具体参考图3所示的实施例的相关描述，在此不在赘述。与图3所示的实施例不同的是，本实施例中，AI处理器101跳过图像处理模块01和图像处理模块02，与图像处理模块03通信。具体的，图像处理模块03的输入端口V pi1以及AI处理器101的输出端口V ao1与片上RAM之间均可以采用电子线路连接。AI处理器101与ISP102中的图像处理模块03之间通过电子线路连接L3传输中断信号Z3，该中断信号Z3用于指示AI处理器101将图像信号C存储至片上RAM。基于图4所示的硬件结构，在一个具体的场景中，AI处理器101从摄像装置105获取图像数据，然后对图像数据进行噪声去除后生成图像信号C存储至片上RAM。AI处理器101将图像信号C存储至片上RAM后通过电子线路连接L3向图像处理模块03发送中断信号Z3。图像处理模块03响应于中断信号Z3从片上RAM获取图像信号C。图像信号C经过ISP102中的图像处理模块03…、图像处理模块N以及图像处理模块N+1的依次执行黑电平校正、阴影校正、白平衡校正…RGB转YUV域等图像处理过程后，生成最终的图像处理结果。
请继续参考图5，其示出了本申请实施例提供的图像处理方法500的又一个流程图。该图像处理方法500应用于如图1或者图4所示的电子装置100。该图像处理方法500包括如下图像处理步骤：步骤501，摄像装置105将采集的图像数据提供至AI处理器101。步骤502，AI处理器101对图像数据进行处理，生成图像信号C。步骤503，AI处理器101将图像信号C提供至ISP102。步骤504，ISP102对图像信号C进行处理，得到图像处理结果。
在本申请实施例中，对图像数据执行多个处理过程的处理以得到图像处理结果的流程中，AI处理器101可以执行多个连续的图像处理过程以对图像数据或者图像信号进行处理，如图2-图5所示的实施例。此外，在其他一些实现方式中，AI处理器101还可以执行多个不连续的图像处理过程以对图像数据或图像信号进行处理。请继续参考图6，其示出了本申请实施例提供的图像处理方法600的又一个流程图。该图像处理方法600应用于图1或者图7所示的电子装置100。该图像处理方法600包括如下图像处理步骤：步骤601，摄像装置105将采集的图像数据提供至ISP102。步骤602，ISP102对图像数据进行处理，生成图像信号D。步骤603，ISP102将图像信号D提供至AI处理器101。步骤604，AI处理器101对图像信号D进行处理，生成图像信号E。步骤605，AI处理器101将图像信号E提供至ISP102。步骤606，ISP102对图像信号E进行处理，生成图像信号F。步骤607，ISP102将图像信号F提供至AI处理器101。步骤608，AI处理器101对图像信号F进行处理，生成图像信号G。步骤609，AI处理器101将图像信号G提供至ISP102。步骤610，ISP102对图像信号G进行处理，得到图像处理结果。
基于图6所示的图像处理步骤，请继续参考图7，其示出了本申请实施例提供的电子装置100的又一个结构示意图。在图7中，电子装置包括AI处理器101、ISP102和片上RAM。与图3所示的ISP102相同，如图7所示的电子装置中的ISP102同样包括级联的图像处理模块01、图像处理模块02、图像处理模块03…图像处理模块N以及图像处理模块N+1，其中各模块的结构以及作用与图3所示的ISP102中的各模块的结构和作用相同，在此不再赘述。图7所示的AI处理器101的内部结构与图3所示的AI处理器101的内部结构相同，在此不再赘述。片上RAM用于存储图像信号D、图像信号F、图像信号F和图像信号G。与图3、图4所示的电子装置不同的是，上述多个级联的图像处理模块中两个图像处理模块设置有输出端口，两个图像处理模块设置有输出端口，具体如图7所示。在图像处理模块02的输出端和图像处理模块03的输出端分别设置有输出端口V po1和输出端口V po2，在图像处理模块03的输入端和图像处理模块N的输入端分别设置有输入端口V pi1和输入端口V pi2。图像处理模块02和图像处理模块03的输出端口分别用于输出如图6所示的图像信号D和图像信号F；图像处理模块03的输入端口和图像处理模块N的输入端口分别用于输入如图6所示的图像信号E和图像信号G。在图7中，图像处理模块02的输出端口V po1和图像处理模块03的输出端口V po2分别与片上RAM之间进行电子线路连接，图像处理模块03的输入端口V pi1和图像处理模块N的输入端口V pi2与片上RAM之间进行电子线路连接；AI处理器101的输入端口V ai1和输出端口V ao1也可以与片上RAM进行电子线路连接。此外，AI处理器101与ISP102中的图像处理模块02之间通过电子线路连接L4传输中断信号Z4，该中断信号Z4用于指示图像处理模块02将图像信号D存储至片上RAM；AI处理器101与ISP102中的图像处理模块03之间通过电子线路连接L5传输中断信号Z5和中断信号Z6，该中断信号Z5用于指示AI处理器101将图像信号E存储至片上RAM，中断信号Z6用于指示图像处理模块03将图像信号F存储至片上RAM；AI处理器101和ISP102中的图像处理模块N之间通过电子线路连接L6传输中断信号Z7，该中断信号Z7用于指示AI处理器101将图像信号G存储至片上RAM。
以上图3-图7示意性的示出了本申请实施例所示的图像处理方法以及与各图像处理方法对应的电子装置的硬件结构。需要说明的是，本申请实施例所示的电子装置中的 ISP102还可以包括设置更多的输出端口和输入端口，以使得更多处理流程处理后的图像在AI处理器101和ISP102之间进行传输，从而可以使得AI处理器101间隔执行更多的图像处理流程。也即是说，AI处理器101和ISP102可以交替执行处理，使得双方共同完成图像处理过程，以得到处理结果，以代替传统ISP的图像处理过程。
基于图3、图4和图7所述的电子装置，在一种可能的实现方式中，ISP102以及AI处理器101是以图像块的形式向如图3、图4或者图7所示的片上RAM中存储。该图像块可以是一帧图像信号中的局部图像信号(例如一帧图像信号1280行像素，片上RAM中存储320行像素形成的图像信号)。具体实现中，ISP102中的各图像处理模块以一行像素为单位进行图像处理，并且ISP102将处理的图像信号逐行存储至片上RAM。当ISP102向片上RAM中存储的图像信号存储完毕时(例如存储的图像信号的行数达到预设阈值、写入的图像信号的大小达到预设阈值或者分配给ISP101的存储地址中的末地址存储有图像信号)，停止向片上RAM继续传输图像信号，并通过如图3所示的电子线路连接L1向AI处理器发送中断信号Z1。AI处理器101可以通过与ISP102相同的存储方式将图像信号存储至片上RAM，以及将图像信号存储至片上RAM后通过如图3所示的电子线路连接L2向AI处理器发送中断信号Z2。进一步的，片上RAM可以逻辑划分为第一存储区和第二存储区，ISP102将图像信号以图像块的形式存储至第一存储区，AI处理器101将图像信号以图像块的形式存储至第二存储器。这样一来，可以使得ISP102对图像的处理以及AI处理器101对图像的处理并行进行，降低AI处理器101和ISP102的等待时间，提高图像信号的传输速率。
此外，当ISP102以图像块的形式向如图3、图4或者图7所示的片上RAM中存储时，ISP102还可以向AI处理器101发送指示当前图像块为一帧图像的起始位置的指示信号。具体的，如图3所示，ISP102中的图像处理模块02与AI处理器101具有电子线路连接L7，向AI处理器101发送中断信号Z8。该中断信号Z8用于指示图像信号A为一帧图像的起始位置。
在本申请实施例一种可能的实现方式中，电子装置还包括片外存储器106，如图8所示。该片外存储器106可以存储有多帧图像，该多帧图像可以为当前图像之前的前一帧图像、前两帧图或者之前的多帧图像。片外存储器106由于具有更大存储空间，可以代替片上RAM，用来存储更大单位的图像数据。其中，片外存储器106中所保存的图像信号可以是AI处理器101处理过的图像信号，或者是提供至AI处理器101供AI处理器101进行处理的图像信号。AI处理器101对当前图像信号进行处理时，还可以从片外存储器106获取当前图像信号的前一帧图像信号或者前几帧图像信号的图像信息，然后基于前一帧图像信号或者前几帧图像信号的图像信息对当前的图像信号进行处理。此外，AI处理器101还可以将处理后的图像信号存储至该片外存储器106。该片外存储器106可以包括随机存取存储器(RAM)，该随机存取存储器可以包括易失性存储器(如SRAM、DRAM、DDR(双倍数据速率SDRAM，Double Data Rate SDRAM)或SDRAM等)和非易失性存储器。此外，该片外存储器106中可以存储有AI处理器101中所运行的图像处理模型的可执行程序，AI处理器通过加载该可执行程序以运行图像处理模型。
示例性地，存储器103，例如是片上RAM可以用于存储单帧图像或者单帧图像中的图像块。片外存储器106则用于存储多帧图像。因此，对于图2、图5或图7所示的方法， AI处理器101和AI处理器101之间传递的图像信号以单位大小划分，将更大单位的图像信息通过片外存储器106进行传输，弥补片上RAM空间的不足，且有效利用了片上RAM更快的传输速度，达到性能优化。
在本申请实施例中，电子装置100还可以包括控制器104，如图8所示。控制器104可以是一个集成控制器，位于AI处理器101、ISP102和存储器103所在的同一个芯片上。控制器104运行有必要的软件程序或软件插件以驱动控制器101控制ISP102的运行、控制AI处理器101运行以及控制ISP102与AI处理器101之间的通信，从而实现如图2、图5或者图6所示的图像处理方法，以代替AI处理器101和ISP102之间的电子线路连接作为通信媒介。具体实现中，控制器104可以为各种数字逻辑器件或电路，包括但不限于：CPU、GPU、微控制器、微处理器或者DSP等。此外，控制器104也可以与AI处理器101、ISP10和存储器10分离设置，本实施例不做限定。进一步的，控制器104也可以与AI处理器101均集成于同一个逻辑运算器件(例如CPU)中，由同一个逻辑运算器件实现本申请实施例所述的控制器104和AI处理器101所执行的功能。在实践中，控制器104可以基于摄像装置105所采集的图像数据所指示的各种信息(例如光强信息、白平衡信息或者马赛克信息等)来确定是否采用ISP102与AI处理器101结合以对图像数据进行处理，以及确定当选用ISP102与AI处理器101结合以对图像数据进行处理时所选用的图像处理模型。下面以光强信息为例，对控制器104对上述各部件的控制以及图像的处理进行详细说明。控制器104基于图像数据所指示的光强信息，将该信息与预设图像光强阈值进行比较，基于比较结果，确定采用ISP102对图像光强处理后无法达到该图像光强阈值时，可以采用AI处理器101对图像数据或者图像信号的光强进行处理。此外，AI处理器101可以运行多种用于进行光强处理的图像处理模型。控制器基于图像数据所指示的光强信息与图像光强阈值之间的差值，确定出选用哪一个图像处理模型。控制器104在确定出所选用的图像处理模型后，可以将上述存储器103中或者片外存储器106中用于存储该图像处理模型的算法程序、参数或指令的存储地址信息下发给AI处理器101，以使的AI处理器101中的各种计算单元从该存储地址获取该算法程序、参数或指令，以运行该图像处理模型，对图像数据或者图像信号进行光强处理。当需要多个图像处理模型对图像数据或图像信号进行处理时，控制器还可以将指示该多个图像处理模型运行的先后顺序信息或者优先等级信息下发给AI处理器101。需要说明的是，本申请实施例中，控制器104可以实时检测或者周期性的检测图像数据所指示的各种信息，基于检测结果确定当前所运行的图像检测模型不适用时，可以及时更换所选用的图像处理模型(例如环境光照强度由强变弱时，更换用于处理光照强度的图像处理模型)，然后将所更换的图像处理模型的可执行程序的存储地址信息下发给AI处理器101，从而使得AI处理器101在下一个图像处理周期进行图像处理时运行所更换的图像处理模型。由此，本申请实施例所述的电子装置，可以基于外部环境的变化或者所采集的图像数据的变化动态调整所采用的图像处理模型，可以使得用户在使用本申请实施例所述的电子装置进行场景更换(例如从户外更换至室内或者从光线强的区域更换至光线弱的区域)时，对所采集的图像有针对性的处理，提高图像处理效果，有利于提高用户体验。
在本申请实施例中，图像数据从开始处理至生成最终的图像处理结果通常具有预设的时钟周期T。AI处理器基于所运行的图像处理模型的不同或者所执行的图像处理流程的 不同，其执行图像处理流程时所占用的时钟周期T内的时长不同。举例来说，图像数据从开始处理至生成最终的图像处理结果的时钟周期T为33.3ms，而AI处理器执行图像处理的时长为18ms，也即是说，AI处理器在时钟周期T的一半时间内处于空闲状态。基于此，为了提高AI处理器的利用率，提高电子装置的运行效率，在一种可能的实现方式中，控制器104还可以确定所采用的图像处理模型的运行时长，当确定出时钟周期T内图像处理模型的运行时长小于预设阈值时，控制器104还可以将AI处理器的空闲时间分配给其他AI服务。该其他AI服务可以包括但不限于：生物识别(例如面部识别或者指纹识别)服务、在图像中添加特效(例如在图像中添加物体)的服务。此时，用于执行该其他AI服务的AI模型的可执行程序或参数也可以存储在上述片外存储器106中。进一步的，当存储器103的存储容量足够大时，该其他AI服务的AI模型的可执行程序或参数也可以存储在存储器103中。
在其他可能的实现方式中，控制器104可以包括多个独立的控制器，该多个独立的控制器中的每一个控制器均可以为数字逻辑器件(例如包括但不限于：GPU或者DSP)。该多个独立的控制器中包括用于控制ISP102中的各部件运行的ISP控制器和用于控制AI处理器101中的各部件运行的AI控制器。此时，AI控制器可以集成于AI处理器101内部。在该种实现方式中，ISP控制器与AI控制器可以采用核间通信的方式传输信息。例如，ISP控制器可以基于图像数据确定选用的图像处理模型后，在图像处理开始之前向AI控制器发送各种配置信息，该配置信息可以包括但不限于：图像处理模型的可执行程序在存储器103中或者片外存储器106中的地址信息、或者多个图像处理模型中的每一个图像处理模型的优先等级信息等。此外，AI处理器101和ISP102之间的信息传输也可以通过ISP控制器和AI控制器之间的通信实现。例如，当AI处理器101和ISP102之间未设置电子线路连接以传输中断信号时，ISP102可以将图像信号存储至片上RAM后通知ISP控制器，ISP控制器将指示图像信号存储至片上RAM的信息发送至AI控制器，AI控制器控制AI处理器101中的计算单元从片上RAM读取图像信号进行图像处理；AI处理器101中的计算单元将图像信号存储至片上RAM后通知AI控制器，AI控制器将指示图像信号存储至片上RAM的信息发送给ISP控制器，ISP控制器ISP102从片上RAM读取图像信号以进行处理。
在本实施例中，电子装置100还可以包括通信单元(图中未示出)，该通信单元包括但不限于短距离通信单元、或蜂窝通信单元。其中，短距离通信单元通过运行短距离无线通信协议与位于移动终端外的用于接入互联网的终端之间进行信息交互。该短距离无线通信协议可以包括但不限于：射频识别技术支持的各种协议、蓝牙通信技术协议、或红外通信协议等。蜂窝通信单元通过运行蜂窝无线通信协议与无线接入网接入互联网，以实现移动通信单元与互联网中对各种应用进行支持的服务器进行信息交互。该通信单元可以与如上各实施例所述的AI处理器101和ISP102等集成于同一SOC中，或者可以分离设置。此外，电子装置100还可选择性地包括总线、输入/输出端口I/O、或存储控制器等。存储控制器用于控制存储器103以及片外存储器106。其中，总线、输入/输出端口I/O、和存储控制器等均可以与上述ISP102和AI处理器101等集成于同一SOC中。应理解，在实际应用中，电子装置100可以包括比图1或者图8所示的更多或更少的部件，本申请实施例不作限定。
可以理解的是，电子装置为了实现上述功能，其包含了执行各个功能相应的硬件和/或软件模块。结合本文中所公开的实施例描述的各示例的步骤，本申请能够以硬件或硬件和计算机软件的结合形式来实现。某个功能究竟以硬件还是计算机软件驱动硬件的方式来执行，取决于技术方案的特定应用和设计约束条件。本领域技术人员可以结合实施例对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
本实施例可以根据上述方法示例对以上一个或多个处理器进行功能模块的划分，例如，可以对应各个功能划分各个不同处理器，也可以将两个或两个以上的功能的处理器集成在一个处理器模块中。上述集成的模块可以采用硬件的形式实现。需要说明的是，本实施例中对模块的划分是示意性的，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式。
在采用对应各个功能划分各个功能模块的情况下，图9示出了上述实施例中涉及的装置900的一种可能的示意图，可以对之前提到的装置进行进一步扩展。如图9所示，该装置900可以包括：AI处理模块901和图像信号处理模块902。其中，AI处理模块901，用于对第一图像信号执行第一图像信号处理以得到第二图像信号，所述第一图像信号是基于图像传感器输出的图像数据获得的；图像信号处理模块902，用于对第二图像信号执行第二图像信号处理以得到图像处理结果。
在一种可能的实现方式中，图像信号处理模块902还用于：从所述图像传感器接收所述图像数据，并对所述图像数据执行第三图像信号处理以得到所述第一图像信号。
在一种可能的实现方式中，所述第三图像信号处理包括多个处理过程，在所述多个处理过程中的两个相邻处理过程中，前一个处理过程用于产生第三图像信号，后一个处理过程用于处理第四图像信号；AI处理模块901，还用于对所述第三图像信号执行第四图像信号处理以得到所述第四图像信号。
在一种可能的实现方式中，所述第一图像信号处理包括如下至少一个处理过程：噪声消除、黑电平校正、阴影矫正、白平衡校正、去马赛克、色差校正或者伽马矫正。
在一种可能的实现方式中，所述第二图像信号处理包括如下至少一个处理过程：噪声消除、黑电平矫正、阴影矫正、白平衡校正、去马赛克、色差矫正、伽马矫正、色差校正或者RGB转YUV域。
在一种可能的实现方式中，所述第三图像信号处理包括如下至少一个处理过程：噪声消除、黑电平矫正、阴影矫正、白平衡校正或者去马赛克。
在一种可能的实现方式中，所述第四图像信号处理包括如下至少一个处理过程：黑电平矫正、阴影矫正、白平衡校正、去马赛克或者色差矫正。
本实施例提供的图像处理装置900，用于执行电子装置100所执行的图像处理方法，可以达到与上述实现方法或装置相同的效果。具体地，以上图9对应的各个模块可以软件、硬件或二者结合实现。例如，每个模块可以以软件形式实现，对应于图1中与该模块对应的相应处理器，用于驱动该相应处理器工作。或者，每个模块可包括对应的处理器和相应的驱动软件两部分，即以软件或硬件结合实现。因此，图像处理装置900可以认为在逻辑上包含了图1、图3、图4、图7或图8所示的装置，每个模块中均至少包含了对应功能的驱动软件程序，本实施例对此不做展开。
示例性地，图像处理装置900可以包括至少一个处理器和存储器，具体参考图1。其中，至少一个处理器可以调用存储器存储的全部或部分计算机程序，对电子装置100的动作进行控制管理，例如，可以用于支持电子装置100执行上述各个模块执行的步骤。存储器可以用于支持电子装置100执行存储程序代码和数据等。至少一个处理器可以实现或执行结合本申请公开内容所描述的各种示例性的多个逻辑模块，其可以是实现计算功能的一个或多个微处理器组合，例如包括但不限于图1所示的AI处理器101和图像信号处理器102。此外，至少一个处理器还可以包括其他可编程逻辑器件、晶体管逻辑器件、或者分立硬件组件等。本实施例所述存储器可以包括但不限于图8所示的片外存储器106或存储器103。
本实施例还提供一种计算机可读存储介质，该计算机可读存储介质中存储有计算机指令，当该计算机指令在计算机上运行时，使得计算机执行上述相关方法步骤实现上述实施例中的图像处理方法。
本实施例还提供了一种计算机程序产品，当该计算机程序产品在计算机上运行时，使得计算机执行上述相关步骤，以实现上述实施例中的图像处理方法。
其中，本实施例提供的计算机可读存储介质或者计算机程序产品均用于执行上文所提供的对应的方法，因此，其所能达到的有益效果可参考上文所提供的对应的方法中的有益效果，此处不再赘述。
通过以上实施方式的描述，所属领域的技术人员可以了解到，为描述的方便和简洁，仅以上述各功能模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能模块完成，即将装置的内部结构划分成不同的功能模块，以完成以上描述的全部或者部分功能。
另外，在本申请各个实施例中的各功能单元可以集成在一个产品中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个产品中。对应于图9，上述模块如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个可读取存储介质中。基于这样的理解，本申请实施例的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该软件产品存储在一个存储介质中，包括若干指令用以使得一个设备(可以是单片机，芯片等)或处理器(processor)执行本申请各个实施例方法的全部或部分步骤。而前述的可读存储介质包括：U盘、移动硬盘、只读存储器(read only memory，ROM)、随机存取存储器(random access memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围为准。

Claims (16)

  1. 一种电子装置，其特征在于，包括：
    人工智能AI处理器，用于对第一图像信号执行第一图像信号处理以得到第二图像信号，所述第一图像信号是基于图像传感器输出的图像数据获得的；
    图像信号处理器ISP，用于对第二图像信号执行第二图像信号处理以得到图像处理结果。
  2. 根据权利要求1所述的电子装置，其特征在于，所述ISP还用于：
    从所述图像传感器接收所述图像数据，并对所述图像数据执行第三图像信号处理以得到所述第一图像信号。
  3. 根据权利要求1或2所述的电子装置，其特征在于，所述第三图像信号处理包括多个处理过程，在所述多个处理过程中的两个相邻处理过程中，前一个处理过程用于产生第三图像信号，后一个处理过程用于处理第四图像信号；
    所述AI处理器，还用于对所述第三图像信号执行第四图像信号处理以得到所述第四图像信号。
  4. 根据权利要求1-3任一项所述的电子装置，其特征在于，所述第一图像信号处理包括如下至少一个处理过程：噪声消除、黑电平校正、阴影矫正、白平衡校正、去马赛克、色差校正或者伽马矫正。
  5. 根据权利要求1-4任一项所述的电子装置，其特征在于，所述第二图像信号处理包括如下至少一个处理过程：噪声消除、黑电平矫正、阴影矫正、白平衡校正、去马赛克、色差矫正、伽马矫正、色差校正或者RGB转YUV域。
  6. 根据权利要求2或3所述的电子装置，其特征在于，所述第三图像信号处理包括如下至少一个处理过程：噪声消除、黑电平矫正、阴影矫正、白平衡校正或者去马赛克。
  7. 根据权利要求2、3或6所述的电子装置，其特征在于，所述第四图像信号处理包括如下至少一个处理过程：黑电平矫正、阴影矫正、白平衡校正、去马赛克或者色差矫正。
  8. 根据权利要求1-7任一项所述的电子装置，其特征在于，所述电子装置还包括：
    存储器，耦合于所述AI处理器和所述ISP，用于在所述AI处理器和所述ISP之间转移任一图像信号中的第一图像单元。
  9. 根据权利要求8所述的电子装置，其特征在于，
    所述存储器、所述AI处理器和所述ISP位于所述电子装置中的一个片上系统内；
    所述存储器包括片上随机存取存储器RAM。
  10. 根据权利要求9所述的电子装置，其特征在于，
    所述第一图像单元包括如下任一项：单帧图像或者单帧图像中的图像块。
  11. 根据权利要求9或10所述的电子装置，其特征在于，所述电子装置还包括：
    片外存储器，位于所述片上系统之外，用于在所述AI处理器和所述ISP之间转移任一图像信号中的第二图像单元，所述第二图像单元包括多帧图像。
  12. 根据权利要求1-11任一项所述的电子装置，其特征在于，所述电子装置还包 括：
    控制器，用于触发所述AI处理器执行所述第一图像信号处理，以及控制所述ISP执行所述第二图像信号处理。
  13. 根据权利要求1-12任一项所述的电子装置，其特征在于，所述AI处理器和所述ISP之间通过电子线路连接传输中断信号。
  14. 一种图像处理方法，其特征在于，所述方法包括：
    控制人工智能AI处理器对第一图像信号执行第一图像信号处理以得到第二图像信号，所述第一图像信号是基于图像传感器输出的图像数据获得的；
    控制图像信号处理器ISP对第二图像信号执行第二图像信号处理以得到图像处理结果。
  15. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质中存储有计算机程序，该计算机程序被至少一个处理器执行时用于实现如权利要求14所述的方法。
  16. 一种计算机程序产品，其特征在于，当所述计算机程序产品被至少一个处理器执行时用于实现如权利要求14所述的方法。
PCT/CN2020/115590 2020-09-16 2020-09-16 电子装置和电子装置的图像处理方法 WO2022056729A1 (zh)

Priority Applications (5)

Application Number Priority Date Filing Date Title
CN202080011168.6A CN114514552A (zh) 2020-09-16 2020-09-16 电子装置和电子装置的图像处理方法
CN202211583539.5A CN115984083B (zh) 2020-09-16 2020-09-16 电子装置和电子装置的图像处理方法
PCT/CN2020/115590 WO2022056729A1 (zh) 2020-09-16 2020-09-16 电子装置和电子装置的图像处理方法
EP20953587.1A EP4198869A4 (en) 2020-09-16 2020-09-16 ELECTRONIC DEVICE, AND IMAGE PROCESSING METHOD FOR ELECTRONIC DEVICE
US18/184,659 US20230214955A1 (en) 2020-09-16 2023-03-16 Electronic apparatus and image processing method of electronic apparatus

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2020/115590 WO2022056729A1 (zh) 2020-09-16 2020-09-16 电子装置和电子装置的图像处理方法

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/184,659 Continuation US20230214955A1 (en) 2020-09-16 2023-03-16 Electronic apparatus and image processing method of electronic apparatus

Publications (1)

Publication Number Publication Date
WO2022056729A1 true WO2022056729A1 (zh) 2022-03-24

Family

ID=80777532

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/115590 WO2022056729A1 (zh) 2020-09-16 2020-09-16 电子装置和电子装置的图像处理方法

Country Status (4)

Country Link
US (1) US20230214955A1 (zh)
EP (1) EP4198869A4 (zh)
CN (2) CN114514552A (zh)
WO (1) WO2022056729A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114727082A (zh) * 2022-03-10 2022-07-08 杭州中天微系统有限公司 图像处理装置、图像信号处理器、图像处理方法和介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103297787A (zh) * 2012-02-24 2013-09-11 宏达国际电子股份有限公司 图像撷取系统及应用于图像撷取系统的图像处理方法
CN103873781A (zh) * 2014-03-27 2014-06-18 成都动力视讯科技有限公司 一种宽动态摄像机实现方法及装置
CN110400250A (zh) * 2019-07-29 2019-11-01 杭州凝眸智能科技有限公司 基于ai的智能图像预处理方法及系统
CN111062894A (zh) * 2020-01-06 2020-04-24 北京都是科技有限公司 人工智能处理器以及人工智能分析设备
WO2020126023A1 (en) * 2018-12-21 2020-06-25 Huawei Technologies Co., Ltd. Image processor
CN111355936A (zh) * 2018-12-20 2020-06-30 杭州凝眸智能科技有限公司 用于人工智能的采集和处理图像数据的方法及系统

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20140105133A (ko) * 2013-02-22 2014-09-01 에스케이하이닉스 주식회사 이미지 신호 프로세싱 시스템
US10148926B2 (en) * 2015-12-07 2018-12-04 Samsung Electronics Co., Ltd. Imaging apparatus and image processing method of thereof
US20170289404A1 (en) * 2016-03-31 2017-10-05 Intel Corporation Joint edge enhance dynamic
CN108376404A (zh) * 2018-02-11 2018-08-07 广东欧珀移动通信有限公司 图像处理方法和装置、电子设备、存储介质
US10735675B2 (en) * 2018-04-13 2020-08-04 Cornell University Configurable image processing system and methods for operating a configurable image processing system for multiple applications
CN109242757A (zh) * 2018-09-29 2019-01-18 南昌黑鲨科技有限公司 智能终端、图像处理方法及计算机可读存储介质
CN111383638A (zh) * 2018-12-28 2020-07-07 上海寒武纪信息科技有限公司 信号处理装置、信号处理方法及相关产品
CN110648273B (zh) * 2019-09-27 2021-07-06 中国科学院长春光学精密机械与物理研究所 实时图像处理装置
CN111402146B (zh) * 2020-02-21 2022-05-10 华为技术有限公司 图像处理方法以及图像处理装置
CN111614884A (zh) * 2020-06-29 2020-09-01 北京百度网讯科技有限公司 人工智能摄像头及监控方法

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103297787A (zh) * 2012-02-24 2013-09-11 宏达国际电子股份有限公司 图像撷取系统及应用于图像撷取系统的图像处理方法
CN103873781A (zh) * 2014-03-27 2014-06-18 成都动力视讯科技有限公司 一种宽动态摄像机实现方法及装置
CN111355936A (zh) * 2018-12-20 2020-06-30 杭州凝眸智能科技有限公司 用于人工智能的采集和处理图像数据的方法及系统
WO2020126023A1 (en) * 2018-12-21 2020-06-25 Huawei Technologies Co., Ltd. Image processor
CN110400250A (zh) * 2019-07-29 2019-11-01 杭州凝眸智能科技有限公司 基于ai的智能图像预处理方法及系统
CN111062894A (zh) * 2020-01-06 2020-04-24 北京都是科技有限公司 人工智能处理器以及人工智能分析设备

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4198869A4 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114727082A (zh) * 2022-03-10 2022-07-08 杭州中天微系统有限公司 图像处理装置、图像信号处理器、图像处理方法和介质
CN114727082B (zh) * 2022-03-10 2024-01-30 杭州中天微系统有限公司 图像处理装置、图像信号处理器、图像处理方法和介质

Also Published As

Publication number Publication date
CN115984083B (zh) 2023-09-29
CN114514552A (zh) 2022-05-17
EP4198869A4 (en) 2023-12-06
CN115984083A (zh) 2023-04-18
EP4198869A1 (en) 2023-06-21
US20230214955A1 (en) 2023-07-06

Similar Documents

Publication Publication Date Title
KR101245485B1 (ko) 개선된 이미지 캡처를 제공하는 방법과 장치 및 프로그램 저장 장치
CN109040576A (zh) 具有用于图像数据统计的基于多帧的窗口的相机控制和图像处理的方法和系统
KR20210073570A (ko) 이미지 처리 방법 및 장치, 및 디바이스
CN101753820B (zh) 信息处理装置、缓冲器控制方法和计算机程序
CN111491102A (zh) 拍照场景的检测方法、系统、移动终端及存储介质
WO2021238522A1 (zh) 多媒体处理芯片、电子设备和图像处理方法
US20230214955A1 (en) Electronic apparatus and image processing method of electronic apparatus
CN104284076A (zh) 一种处理预览图像的方法、装置及移动终端
CN110445986A (zh) 图像处理方法、装置、存储介质及电子设备
US10769416B2 (en) Image processing method, electronic device and storage medium
WO2022151852A1 (zh) 图像处理方法、装置、系统、电子设备以及存储介质
WO2023010755A1 (zh) 一种hdr视频转换方法、装置、设备及计算机存储介质
US20240054751A1 (en) Electronic apparatus and image processing method of electronic apparatus
WO2012167663A1 (zh) 视频图像的亮度调整方法和装置
WO2024051674A1 (zh) 图像处理电路和电子设备
US20220301278A1 (en) Image processing method and apparatus, storage medium, and electronic device
US20230086519A1 (en) Multimedia processing chip, electronic device, and method for dynamic-image processing
US9374526B2 (en) Providing frame delay using a temporal filter
CN117014720A (zh) 图像拍摄方法、装置、终端、存储介质及产品
CN114299014A (zh) 图像处理架构、方法、电子设备及存储介质
CN110072050A (zh) 曝光参数的自适应调整方法、装置及一种拍摄设备
WO2024066521A1 (zh) 内存刷新方法和装置
CN113873141B (zh) 电子设备
WO2023036313A1 (zh) 图像拍摄方法、装置、计算机设备及存储介质
WO2022227200A1 (zh) 一种拍摄分辨率的调整方法及装置、智能设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20953587

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2020953587

Country of ref document: EP

Effective date: 20230314

NENP Non-entry into the national phase

Ref country code: DE