WO2023125518A1 - Procédé de codage d'image et dispositif - Google Patents

Procédé de codage d'image et dispositif Download PDF

Info

Publication number
WO2023125518A1
WO2023125518A1 PCT/CN2022/142277 CN2022142277W WO2023125518A1 WO 2023125518 A1 WO2023125518 A1 WO 2023125518A1 CN 2022142277 W CN2022142277 W CN 2022142277W WO 2023125518 A1 WO2023125518 A1 WO 2023125518A1
Authority
WO
WIPO (PCT)
Prior art keywords
memory
slice
preset
height
storage
Prior art date
Application number
PCT/CN2022/142277
Other languages
English (en)
Chinese (zh)
Inventor
李国怀
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2023125518A1 publication Critical patent/WO2023125518A1/fr

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/20—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using video object coding
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
    • H04N19/423—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation characterised by memory arrangements

Definitions

  • the present application relates to the field of artificial intelligence, and in particular to an image encoding method and device.
  • the image is divided into multiple slices (Grid), and then each slice is encoded, which can improve the compression efficiency of the image and improve the encoding performance, so the process of image slicing is particularly important
  • the present application provides an image coding method and device, which are used to code stored images by means of memory mapping, reduce memory copying, and improve coding efficiency.
  • the present application provides an image encoding method, comprising: first, allocating a storage memory for an input image; dividing the storage memory into a plurality of memory slices; The address of at least one memory slice reads data and performs encoding to obtain encoded data.
  • the address of the data to be encoded can be directly mapped to the encoder, so that the encoder can directly extract the data to be encoded based on the address, so memory copying is not required, and the memory overhead of memory copying can be reduced , and can improve the coding efficiency.
  • the aforementioned dividing the storage memory into multiple memory slices may include: dividing the storage memory into multiple memory slices according to a preset slice size, where the preset slice size includes a preset height and preset width.
  • the division when dividing the storage memory, the division may be performed according to a preset slice size, so that subsequent encoders may perform encoding in units of the preset slice size.
  • the aforementioned allocating storage memory for the input image may include: allocating storage memory for the input image based on a preset slice size.
  • the storage memory may include a first memory and a second memory, the first memory stores the information of each pixel in the input image, and the second memory is used to make the size of the storage memory according to the preset Slice size alignment so that the encoder reads data from the input address in units of the preset slice size.
  • the address of the obtained memory slice can be mapped to the encoder for encoding to obtain the final encoded data, reducing the amount of memory copy , which can reduce memory overhead for memory copying and improve coding efficiency.
  • the storage memory is arranged in an array, the width of the storage memory is divisible by a preset width, and the height of the storage memory is divisible by a preset height.
  • the size of each memory slice is the preset slice size, Avoid the coder reading the address out of bounds, and only need to allocate a small amount of memory, which reduces the amount of memory copying, reduces the memory overhead of memory copying, and improves coding efficiency.
  • the second memory includes a third memory and a fourth memory
  • the first memory and the third memory are combined and arranged in an array
  • the height of the array is divisible by the preset height
  • the last row of the array is divided by the preset height.
  • the remainder after setting the width is not smaller than the preset width after being combined with the fourth memory.
  • the height of the allocated storage memory can be aligned with the preset height.
  • additional memory can be allocated for the last row, so that the array
  • the remainder after dividing the last line by the preset width is not smaller than the preset width after being combined with the fourth memory, to prevent the encoder from crossing the boundary when reading the address, and only need to allocate a small amount of memory, which reduces the amount of memory copy and can reduce the Memory copy memory overhead, and can improve coding efficiency.
  • multiple memory slices are divided into a first memory slice and a second memory slice, the height of the first memory slice is a preset height, and the height of the second memory slice is smaller than the preset height or the second memory slice
  • the last slice of the last line in the array composed of multiple memory slices, the first memory slice and the second memory slice are different memory slices; the above method may also include: allocating the second memory slice for the second memory slice among the multiple memory slices Five memory, and copy the data in the second memory to the fifth memory, the width of the fifth memory is divisible by the preset width, and the height of the fifth memory is divisible by the preset height; the aforementioned at least one of the multiple memory slices
  • Inputting the address of the memory slice to the encoder may include: inputting the address of the first memory slice and the address of the fifth memory among the plurality of memory slices to the encoder.
  • the storage memory occupied by the input image can be allocated based on the size of the input image, and then the storage memory can be divided.
  • the storage memory of the input image is not divisible by the preset slice size, the obtained and The address of the memory that matches the preset slice size is mapped to the encoder for encoding, and the data is copied from the remainder after division and used as the input of the encoder to obtain the final encoded data without copying all the memory in the storage memory.
  • the amount of memory copy is reduced, the memory overhead of memory copy can be reduced, and the coding efficiency can be improved.
  • the height of the storage memory is evenly divisible by the preset height.
  • the height of the allocated memory can be aligned according to the preset height, thereby reducing multiple The number of slices whose dimensions did not match the preset slice size. Then you can map the address of the memory that matches the preset slice size to the encoder for encoding, and copy the data from the remainder after integer division and use it as the input of the encoder to obtain the final encoded data without needing to store the data in the memory. Copy all the memory of the memory, which reduces the amount of memory copy, can reduce the memory overhead of memory copy, and can improve the coding efficiency.
  • an image encoding device including:
  • a division module configured to divide the storage memory into multiple memory slices
  • the encoding module is configured to input the address of at least one memory slice in the plurality of memory slices to the encoder, and the encoder is used to read and encode data based on the input address to obtain encoded data.
  • the division module is specifically configured to divide the storage memory into multiple memory slices according to a preset slice size, where the preset slice size includes a preset height and a preset width.
  • the storage memory includes a first memory and a second memory, the first memory stores the information of each pixel in the input image, and the second memory is used to make the size of the storage memory according to the preset Slice size alignment so that the encoder reads data from the input address in units of the preset slice size.
  • the storage memory is arranged in an array, the width of the storage memory is divisible by a preset width, and the height of the storage memory is divisible by a preset height.
  • the second memory includes a third memory and a fourth memory
  • the first memory and the third memory are combined and arranged in an array
  • the height of the array is divisible by the preset height
  • the last row of the array is divided by the preset height.
  • the remainder after setting the width is not smaller than the preset width after being combined with the fourth memory.
  • multiple memory slices are divided into a first memory slice and a second memory slice, the height of the first memory slice is a preset height, and the height of the second memory slice is smaller than the preset height or the second memory slice It is the last slice of the last line in the array composed of multiple memory slices, the first memory slice and the second memory slice are different memory slices;
  • the allocation module is also used to allocate a fifth memory for the second memory slice in the plurality of memory slices, and copy the data in the second memory to the fifth memory, the width of the fifth memory is divisible by the preset width, and the fifth memory The height of the memory is divisible by the preset height;
  • the encoding module is specifically configured to input the address of the first memory slice and the address of the fifth memory among the plurality of memory slices to the encoder.
  • the height of the storage memory is evenly divisible by the preset height.
  • an embodiment of the present application provides an electronic device, including: a processor and a memory, wherein the processor and the memory are interconnected through a line, and the processor calls the program code in the memory to execute any one of the above-mentioned first aspects. Functions related to processing in the image coding method shown.
  • the electronic device may be a chip.
  • the embodiment of the present application provides an electronic device, which can also be called a digital processing chip or a chip.
  • the chip includes a processing unit and a communication interface.
  • the processing unit obtains program instructions through the communication interface, and the program instructions are processed by the processing unit.
  • the processing unit is configured to execute the processing-related functions in the first aspect or any optional implementation manner of the first aspect.
  • an embodiment of the present application provides a computer-readable storage medium, including instructions, which, when run on a computer, cause the computer to execute the method in the above-mentioned first aspect or any optional implementation manner of the first aspect.
  • the embodiments of the present application provide a computer program product containing instructions, which, when run on a computer, cause the computer to execute the method in the above-mentioned first aspect or any optional implementation manner of the first aspect.
  • FIG. 1 is a schematic structural diagram of an electronic device provided by the present application.
  • FIG. 2 is a schematic structural diagram of another electronic device provided by the present application.
  • FIG. 3 is a schematic flowchart of an image encoding method provided by the present application.
  • FIG. 4 is a schematic structural diagram of a storage memory provided by the present application.
  • FIG. 5 is a schematic structural diagram of another electronic device provided by the present application.
  • FIG. 6 is a schematic flowchart of another image coding method provided by the present application.
  • FIG. 7 is a schematic structural diagram of another storage memory provided by the present application.
  • FIG. 8 is a schematic structural diagram of another storage memory provided by the present application.
  • FIG. 9 is a schematic structural diagram of another storage memory provided by the present application.
  • FIG. 10 is a schematic structural diagram of another storage memory provided by the present application.
  • FIG. 11 is a schematic structural diagram of another storage memory provided by the present application.
  • FIG. 12 is a schematic structural diagram of another storage memory provided by the present application.
  • FIG. 13 is a schematic structural diagram of an image encoding device provided by the present application.
  • FIG. 14 is a schematic structural diagram of another electronic device provided by the present application.
  • the method provided by this application can be deployed in various electronic devices, such as deployed in various terminals or servers.
  • the servers can be various independent servers, distributed servers, or centralized servers.
  • the terminals can include but not Limited to: smart mobile phones, TVs, tablet computers, wristbands, head mounted display devices (Head Mount Display, HMD), augmented reality (augmented reality, AR) devices, mixed reality (mixed reality, MR) devices, cellular phones (cellular phone), smart phone (smart phone), personal digital assistant (personal digital assistant, PDA), tablet computer, vehicle terminal, laptop computer (laptop computer) (or notebook computer or laptop computer, etc.), personal computer (personal computer, PC) and so on.
  • FIG. 1 a specific structure is taken as an example below to illustrate the structure of the electronic device provided in this application.
  • the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, and an antenna 2 , mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and A subscriber identification module (subscriber identification module, SIM) card interface 195 and the like.
  • SIM subscriber identification module
  • the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, bone conduction sensor 180M, motion sensor 180N, etc.
  • the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100 .
  • the electronic device 100 may include more or fewer components than shown in the figure, or combine certain components, or separate certain components, or arrange different components.
  • the illustrated components can be realized in hardware, software or a combination of software and hardware.
  • the processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.
  • application processor application processor, AP
  • modem processor graphics processing unit
  • GPU graphics processing unit
  • image signal processor image signal processor
  • ISP image signal processor
  • controller video codec
  • digital signal processor digital signal processor
  • baseband processor baseband processor
  • neural network processor neural-network processing unit
  • the controller can generate an operation control signal according to the instruction opcode and timing signal, and complete the control of fetching and executing the instruction.
  • a memory may also be provided in the processor 110 for storing instructions and data.
  • the memory in processor 110 is a cache memory.
  • the memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thereby improving the efficiency of the system.
  • processor 110 may include one or more interfaces.
  • the interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous transmitter (universal asynchronous receiver/transmitter, UART) interface, mobile industry processor interface (mobile industry processor interface, MIPI), general-purpose input and output (general-purpose input/output, GPIO) interface, subscriber identity module (subscriber identity module, SIM) interface, and /or universal serial bus (universal serial bus, USB) interface, etc.
  • I2C integrated circuit
  • I2S integrated circuit built-in audio
  • PCM pulse code modulation
  • PCM pulse code modulation
  • UART universal asynchronous transmitter
  • MIPI mobile industry processor interface
  • GPIO general-purpose input and output
  • subscriber identity module subscriber identity module
  • SIM subscriber identity module
  • USB universal serial bus
  • the I2C interface is a bidirectional synchronous serial bus, including a serial data line (serial data line, SDA) and a serial clock line (derail clock line, SCL).
  • processor 110 may include multiple sets of I2C buses.
  • the processor 110 can be respectively coupled to the touch sensor 180K, the charger, the flashlight, the camera 193 and the like through different I2C bus interfaces.
  • the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 and the touch sensor 180K communicate through the I2C bus interface to realize the touch function of the electronic device 100 .
  • the I2S interface can be used for audio communication.
  • processor 110 may include multiple sets of I2S buses.
  • the processor 110 may be coupled to the audio module 170 through an I2S bus to implement communication between the processor 110 and the audio module 170 .
  • the audio module 170 can transmit audio signals to the wireless communication module 160 through the I2S interface, so as to realize the function of answering calls through the Bluetooth headset.
  • the PCM interface can also be used for audio communication, sampling, quantizing and encoding the analog signal.
  • the audio module 170 and the wireless communication module 160 may be coupled through a PCM bus interface.
  • the audio module 170 can also transmit audio signals to the wireless communication module 160 through the PCM interface, so as to realize the function of answering calls through the Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.
  • the UART interface is a universal serial data bus used for asynchronous communication.
  • the bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication.
  • a UART interface is generally used to connect the processor 110 and the wireless communication module 160 .
  • the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to realize the Bluetooth function.
  • the audio module 170 can transmit audio signals to the wireless communication module 160 through the UART interface, so as to realize the function of playing music through the Bluetooth headset.
  • the MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193 .
  • MIPI interface includes camera serial interface (camera serial interface, CSI), display serial interface (display serial interface, DSI), etc.
  • the processor 110 communicates with the camera 193 through the CSI interface to realize the shooting function of the electronic device 100 .
  • the processor 110 communicates with the display screen 194 through the DSI interface to realize the display function of the electronic device 100 .
  • the GPIO interface can be configured by software.
  • the GPIO interface can be configured as a control signal or as a data signal.
  • the GPIO interface can be used to connect the processor 110 with the camera 193 , the display screen 194 , the wireless communication module 160 , the audio module 170 , the sensor module 180 and so on.
  • the GPIO interface can also be configured as an I2C interface, I2S interface, UART interface, MIPI interface, etc.
  • the USB interface 130 is an interface conforming to the USB standard specification, specifically, it can be a Mini USB interface, a Micro USB interface, a USB Type C interface, and the like.
  • the USB interface 130 can be used to connect a charger to charge the electronic device 100 , and can also be used to transmit data between the electronic device 100 and peripheral devices. It can also be used to connect headphones and play audio through them. This interface can also be used to connect other electronic devices, such as AR devices.
  • the USB interface 130 here can also be replaced with other interfaces, such as Type-c or Lighting, which can realize charging or data transmission, and only the USB interface 130 is used as an example for illustration here.
  • the interface connection relationship between the modules shown in the embodiment of the present invention is only a schematic illustration, and does not constitute a structural limitation of the electronic device 100 .
  • the electronic device 100 may also adopt different interface connection manners in the foregoing embodiments, or a combination of multiple interface connection manners.
  • the charging management module 140 is configured to receive a charging input from a charger.
  • the charger may be a wireless charger or a wired charger.
  • the charging management module 140 can receive charging input from the wired charger through the USB interface 130 .
  • the charging management module 140 may receive a wireless charging input through a wireless charging coil of the electronic device 100 . While the charging management module 140 is charging the battery 142 , it can also supply power to the electronic device through the power management module 141 .
  • the power management module 141 is used for connecting the battery 142 , the charging management module 140 and the processor 110 .
  • the power management module 141 receives the input from the battery 142 and/or the charging management module 140 to provide power for the processor 110 , the internal memory 121 , the display screen 194 , the camera 193 , and the wireless communication module 160 .
  • the power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle times, and battery health status (leakage, impedance).
  • the power management module 141 may also be disposed in the processor 110 .
  • the power management module 141 and the charging management module 140 may also be set in the same device.
  • the wireless communication function of the electronic device 100 can be realized by the antenna 1 , the antenna 2 , the mobile communication module 150 , the wireless communication module 160 , a modem processor, a baseband processor, and the like.
  • Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals.
  • Each antenna in electronic device 100 may be used to cover single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas.
  • Antenna 1 can be multiplexed as a diversity antenna for a WLAN.
  • the antenna may be used in conjunction with a tuning switch.
  • the mobile communication module 150 can provide wireless communication solutions including 2G/3G/4G/5G applied on the electronic device 100 .
  • the mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA) and the like.
  • the mobile communication module 150 can receive electromagnetic waves through the antenna 1, filter and amplify the received electromagnetic waves, and send them to the modem processor for demodulation.
  • the mobile communication module 150 can also amplify the signals modulated by the modem processor, and convert them into electromagnetic waves and radiate them through the antenna 1 .
  • at least part of the functional modules of the mobile communication module 150 may be set in the processor 110 .
  • at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 may be set in the same device.
  • a modem processor may include a modulator and a demodulator.
  • the modulator is used for modulating the low-frequency baseband signal to be transmitted into a medium-high frequency signal.
  • the demodulator is used to demodulate the received electromagnetic wave signal into a low frequency baseband signal. Then the demodulator sends the demodulated low-frequency baseband signal to the baseband processor for processing.
  • the low-frequency baseband signal is passed to the application processor after being processed by the baseband processor.
  • the application processor outputs sound signals through audio equipment (not limited to speaker 170A, receiver 170B, etc.), or displays images or videos through display screen 194 .
  • the modem processor may be a stand-alone device.
  • the modem processor may be independent from the processor 110, and be set in the same device as the mobile communication module 150 or other functional modules.
  • the wireless communication module 160 can provide wireless local area networks (wireless local area networks, WLAN) (such as wireless fidelity (Wireless Fidelity, Wi-Fi) network), bluetooth (bluetooth, BT), global navigation satellite, etc. applied on the electronic device 100.
  • System global navigation satellite system, GNSS
  • frequency modulation frequency modulation, FM
  • near field communication technology near field communication, NFC
  • ultra wideband ultra wideband
  • UWB ultra wideband
  • infrared technology infrared, IR
  • the wireless communication module 160 may be one or more devices integrating at least one communication processing module.
  • the wireless communication module 160 receives electromagnetic waves via the antenna 2 , frequency-modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110 .
  • the wireless communication module 160 can also receive the signal to be sent from the processor 110 , frequency-modulate it, amplify it, and convert it into electromagnetic waves through the antenna 2 for radiation.
  • the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology.
  • the wireless communication technology may include but not limited to: the fifth generation mobile communication technology (5th-Generation, 5G) system, the global system for mobile communications (global system for mobile communications, GSM), general packet radio service (general packet radio service, GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA) ), long term evolution (long term evolution, LTE), bluetooth (bluetooth), global navigation satellite system (the global navigation satellite system, GNSS), wireless fidelity (wireless fidelity, WiFi), near field communication (near field communication, NFC), FM (also called FM radio), Zigbee (Zigbee), radio frequency identification (radio frequency identification, RFID) and/or infrared (infrared, IR) technology, etc
  • the GNSS may include a global positioning system (global positioning system, GPS), a global navigation satellite system (global navigation satellite system, GLONASS), a Beidou navigation satellite system (beidou navigation satellite system, BDS), a quasi-zenith satellite system (quasi -zenith satellite system, QZSS) and/or satellite based augmentation systems (satellite based augmentation systems, SBAS), etc.
  • GPS global positioning system
  • GLONASS global navigation satellite system
  • Beidou navigation satellite system beidou navigation satellite system
  • BDS Beidou navigation satellite system
  • QZSS quasi-zenith satellite system
  • SBAS satellite based augmentation systems
  • the electronic device 100 may also include a wired communication module (not shown in FIG. 1 ), or the mobile communication module 150 or wireless communication module 160 here may be replaced by a wired communication module (not shown in FIG. output), the wired communication module can enable the electronic device to communicate with other devices through a wired network.
  • the wired network may include but not limited to one or more of the following: Optical Transport Network (OTN), Synchronous Digital Hierarchy (SDH), Passive Optical Network (PON), Ethernet network (Ethernet), or flexible Ethernet (flex Ethernet, FlexE), etc.
  • the electronic device 100 realizes the display function through the GPU, the display screen 194 , and the application processor.
  • the GPU is a microprocessor for interface display, and is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering.
  • Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.
  • the display screen 194 is used to display images, videos and the like.
  • the display screen 194 includes a display panel.
  • the display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light emitting diode or an active matrix organic light emitting diode (active-matrix organic light emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diodes (quantum dot light emitting diodes, QLED), etc.
  • the electronic device 100 may include 1 or N display screens 194 , where N is a positive integer greater than 1.
  • the electronic device 100 can realize the shooting function through the ISP, the camera 193 , the video codec, the GPU, the display screen 194 and the application processor.
  • the ISP is used for processing the data fed back by the camera 193 .
  • the light is transmitted to the photosensitive element of the camera through the lens, and the light signal is converted into an electrical signal, and the photosensitive element of the camera transmits the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye.
  • ISP can also perform algorithm optimization on image noise, brightness, and skin color.
  • ISP can also optimize the exposure, color temperature and other parameters of the shooting scene.
  • the ISP may be located in the camera 193 .
  • Camera 193 is used to capture still images or video.
  • the object generates an optical image through the lens and projects it to the photosensitive element.
  • the photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor.
  • CMOS complementary metal-oxide-semiconductor
  • the photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal.
  • the ISP outputs the digital image signal to the DSP for processing.
  • DSP converts digital image signals into image signals in standard RGB camera, YUV and other formats.
  • the electronic device 100 may include 1 or N cameras 193 , where N is a positive integer greater than 1.
  • Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the energy of the frequency point.
  • Video codecs are used to compress or decompress digital video.
  • the electronic device 100 may support one or more video codecs.
  • the electronic device 100 can play or record video in various encoding formats, for example: moving picture experts group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4, etc.
  • the NPU is a neural-network (NN) computing processor.
  • NN neural-network
  • Applications such as intelligent cognition of the electronic device 100 can be realized through the NPU, such as image recognition, face recognition, speech recognition, text understanding, and the like.
  • the external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, so as to expand the storage capacity of the electronic device 100.
  • the external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. Such as saving music, video and other files in the external memory card.
  • the internal memory 121 may be used to store computer-executable program codes including instructions.
  • the internal memory 121 may include an area for storing programs and an area for storing data.
  • the stored program area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.) and the like.
  • the storage data area can store data created during the use of the electronic device 100 (such as audio data, phonebook, etc.) and the like.
  • the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (universal flash storage, UFS) and the like.
  • the processor 110 executes various functional applications and data processing of the electronic device 100 by executing instructions stored in the internal memory 121 and/or instructions stored in a memory provided in the processor.
  • the electronic device 100 can implement audio functions through the audio module 170 , the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor. Such as music playback, recording, etc.
  • the audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal.
  • the audio module 170 may also be used to encode and decode audio signals.
  • the audio module 170 may be set in the processor 110 , or some functional modules of the audio module 170 may be set in the processor 110 .
  • Speaker 170A also referred to as a "horn" is used to convert audio electrical signals into sound signals.
  • Electronic device 100 can listen to music through speaker 170A, or listen to hands-free calls.
  • Receiver 170B also called “earpiece” is used to convert audio electrical signals into sound signals.
  • the receiver 170B can be placed close to the human ear to receive the voice.
  • the microphone 170C also called “microphone” or “microphone” is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can put his mouth close to the microphone 170C to make a sound, and input the sound signal to the microphone 170C.
  • the electronic device 100 may be provided with at least one microphone 170C. In some other embodiments, the electronic device 100 may be provided with two microphones 170C, which may also implement a noise reduction function in addition to collecting sound signals. In some other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify sound sources, and realize directional recording functions, etc.
  • the earphone interface 170D is used for connecting wired earphones.
  • the earphone interface 170D can be a USB interface 130, or a 3.5mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.
  • OMTP open mobile terminal platform
  • CTIA cellular telecommunications industry association of the USA
  • the pressure sensor 180A is used to sense the pressure signal and convert the pressure signal into an electrical signal.
  • the pressure sensor 180A may be located on the display screen 194 .
  • pressure sensors 180A such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors.
  • a capacitive pressure sensor may be comprised of at least two parallel plates with conductive material.
  • the electronic device 100 determines the intensity of pressure according to the change in capacitance.
  • the electronic device 100 detects the intensity of the touch operation according to the pressure sensor 180A.
  • the electronic device 100 may also calculate the touched position according to the detection signal of the pressure sensor 180A.
  • touch operations acting on the same touch position but with different touch operation intensities may correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view short messages is executed. When a touch operation whose intensity is greater than or equal to the first pressure threshold acts on the icon of the short message application, the instruction of creating a new short message is executed.
  • the gyro sensor 180B can be used to determine the motion posture of the electronic device 100 .
  • the angular velocity of the electronic device 100 around three axes may be determined by the gyro sensor 180B.
  • the gyro sensor 180B can be used for image stabilization. Exemplarily, when the shutter is pressed, the gyro sensor 180B detects the shaking angle of the electronic device 100, calculates the distance that the lens module needs to compensate according to the angle, and allows the lens to counteract the shaking of the electronic device 100 through reverse movement to achieve anti-shake.
  • the gyro sensor 180B can also be used for navigation and somatosensory game scenes.
  • the air pressure sensor 180C is used to measure air pressure.
  • the electronic device 100 calculates the altitude based on the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.
  • the magnetic sensor 180D includes a Hall sensor.
  • the electronic device 100 may use the magnetic sensor 180D to detect the opening and closing of the flip leather case.
  • the electronic device 100 when the electronic device 100 is a clamshell machine, the electronic device 100 can detect opening and closing of the clamshell according to the magnetic sensor 180D.
  • features such as automatic unlocking of the flip cover are set.
  • the acceleration sensor 180E can detect the acceleration of the electronic device 100 in various directions (generally three axes).
  • the magnitude and direction of gravity can be detected when the electronic device 100 is stationary. It can also be used to identify the posture of electronic devices, and can be used in applications such as horizontal and vertical screen switching, pedometers, etc.
  • the distance sensor 180F is used to measure the distance.
  • the electronic device 100 may measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 may use the distance sensor 180F for distance measurement to achieve fast focusing.
  • Proximity light sensor 180G may include, for example, light emitting diodes (LEDs) and light detectors, such as photodiodes.
  • the light emitting diodes may be infrared light emitting diodes.
  • the electronic device 100 emits infrared light through the light emitting diode.
  • Electronic device 100 uses photodiodes to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it may be determined that there is an object near the electronic device 100 . When insufficient reflected light is detected, the electronic device 100 may determine that there is no object near the electronic device 100 .
  • the electronic device 100 can use the proximity light sensor 180G to detect that the user is holding the electronic device 100 close to the ear to make a call, so as to automatically turn off the screen to save power.
  • the proximity light sensor 180G can also be used in leather case mode, automatic unlock and lock screen in pocket mode.
  • the ambient light sensor 180L is used for sensing ambient light brightness.
  • the electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness.
  • the ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures.
  • the ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in the pocket, so as to prevent accidental touch.
  • the fingerprint sensor 180H is used to collect fingerprints.
  • the electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access to application locks, take pictures with fingerprints, answer incoming calls with fingerprints, and the like.
  • the temperature sensor 180J is used to detect temperature.
  • the electronic device 100 uses the temperature detected by the temperature sensor 180J to implement a temperature treatment strategy. For example, when the temperature reported by the temperature sensor 180J exceeds the threshold, the electronic device 100 may reduce the performance of the processor located near the temperature sensor 180J, so as to reduce power consumption and implement thermal protection.
  • the electronic device 100 when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 caused by the low temperature.
  • the electronic device 100 boosts the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.
  • the touch sensor 180K is also called “touch device”.
  • the touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a “touch screen”.
  • the touch sensor 180K is used to detect a touch operation on or near it.
  • the touch sensor can pass the detected touch operation to the application processor to determine the type of touch event.
  • Visual output related to the touch operation can be provided through the display screen 194 .
  • the touch sensor 180K may also be disposed on the surface of the electronic device 100 , which is different from the position of the display screen 194 .
  • the bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire the vibration signal of the vibrating bone mass of the human voice. The bone conduction sensor 180M can also contact the human pulse and receive the blood pressure beating signal. In some embodiments, the bone conduction sensor 180M can also be disposed in the earphone, combined into a bone conduction earphone.
  • the audio module 170 can analyze the voice signal based on the vibration signal of the vibrating bone mass of the vocal part acquired by the bone conduction sensor 180M, so as to realize the voice function.
  • the application processor can analyze the heart rate information based on the blood pressure beating signal acquired by the bone conduction sensor 180M, so as to realize the heart rate detection function.
  • the motion sensor 180N can be used to detect moving objects within the range captured by the camera, and collect motion profiles or trajectory of the moving objects.
  • the motion sensor 180N can be an infrared sensor, a laser sensor, a dynamic vision sensor (dynamic vision sensor, DVS), etc.
  • the DVS can specifically include DAVIS (Dynamic and Active-pixel Vision Sensor), ATIS (Asynchronous Time-based Image Sensor ) or sensors such as CeleX sensors.
  • DAVIS Dynamic and Active-pixel Vision Sensor
  • ATIS Asynchronous Time-based Image Sensor
  • CeleX sensors CeleX sensors.
  • DVS draws lessons from the characteristics of biological vision, each pixel simulates a neuron, and independently responds to relative changes in light intensity (hereinafter referred to as "light intensity"). When the relative change of light intensity exceeds the threshold, the pixel will output an event signal, including the pixel's position, time stamp and feature information of light intensity.
  • the keys 190 include a power key, a volume key and the like.
  • the key 190 may be a mechanical key. It can also be a touch button.
  • the electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100 .
  • the motor 191 can generate a vibrating reminder.
  • the motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback.
  • touch operations applied to different applications may correspond to different vibration feedback effects.
  • the motor 191 may also correspond to different vibration feedback effects for touch operations acting on different areas of the display screen 194 .
  • Different application scenarios for example: time reminder, receiving information, alarm clock, games, etc.
  • the touch vibration feedback effect can also support customization.
  • the indicator 192 can be an indicator light, and can be used to indicate charging status, power change, and can also be used to indicate messages, missed calls, notifications, and the like.
  • the SIM card interface 195 is used for connecting a SIM card.
  • the SIM card can be connected and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195 .
  • the electronic device 100 may support 1 or N SIM card interfaces, where N is a positive integer greater than 1.
  • SIM card interface 195 can support Nano SIM card, Micro SIM card, SIM card etc. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards may be the same or different.
  • the SIM card interface 195 is also compatible with different types of SIM cards.
  • the SIM card interface 195 is also compatible with external memory cards.
  • the electronic device 100 interacts with the network through the SIM card to implement functions such as calling and data communication.
  • the electronic device 100 adopts an eSIM, that is, an embedded SIM card.
  • the eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100 .
  • the electronic device may include more or fewer components than those shown in FIG. 1 , which may be adjusted according to actual application scenarios, which is not limited in this application.
  • the electronic device 100 can be logically divided into a hardware layer 21 , an operating system 261 , and an application layer 31 .
  • the hardware layer 21 includes hardware resources such as an application program processor 201 , a microcontroller unit 203 , a modem 207 , a Wi-Fi module 211 , a sensor 214 , and a positioning module 150 .
  • the application layer 31 includes one or more applications (applications, APPs), such as the application program 263, and the application program 263 may be any type of application program such as a social application program, an e-commerce application program, or a browser.
  • the operating system 261 is a software middleware between the hardware layer 21 and the application layer 31, and is a computer program for managing and controlling hardware and software resources.
  • the operating system 261 includes a kernel 23, a hardware abstraction layer (hardware abstraction layer, HAL) 25, libraries and runtime (libraries and runtime) 27, and a framework (framework) 29.
  • the kernel 23 is used to provide underlying system components and services, such as: power management, memory management, thread management, hardware drivers, etc.; hardware drivers include Wi-Fi drivers, sensor drivers, positioning module drivers, etc.
  • the hardware abstraction layer 25 encapsulates the kernel driver, provides an interface to the framework 29, and shields low-level implementation details.
  • the hardware abstraction layer 25 runs in user space, while the kernel driver runs in kernel space.
  • Library and runtime 27 is also called runtime library, which provides required library files and execution environment for executable programs at runtime.
  • the library and runtime 27 includes an Android runtime (Android Runtime, ART) 271 and a library 273, etc.
  • ART 271 is a virtual machine or virtual machine instance capable of converting an application's bytecode into machine code.
  • the library 273 is a program library that provides support for executable programs at runtime, including browser engines (such as webkit), script execution engines (such as JavaScript engines), graphics processing engines, and the like.
  • the framework 27 is used to provide various basic public components and services for applications in the application layer 31, such as window management, location management and so on.
  • the framework 27 may include a phone manager 291, a resource manager 293, a location manager 295, and the like.
  • each component of the operating system 261 described above can be realized by the application program processor 201 executing the programs stored in the memory 205 .
  • the electronic device 100 may include fewer or more components than those shown in FIG. 2.
  • the aforementioned electronic devices have the ability to store images and encode image data.
  • the images can be divided into multiple slices (Grid), and then each slice can be encoded, so that Improve image compression efficiency and improve encoding performance.
  • slicing methods can be divided into two modes: FrameWork layer slicing and slice sinking.
  • the original image is: 4032*3024 high efficiency image file format (high efficiency image file format, HEIF) coded picture (usually called Grid): 512*512;
  • HEIF high efficiency image file format
  • Grid coded picture
  • 512*512 the actual input to the encoder is the storage memory of 4032*3024, and it is divided into 512*512 Grid, that is, it is necessary to allocate memory for the image, copy the image data to the storage memory of 4032*3024, and then encode
  • the device reads data from this part of the memory for encoding, requiring a large number of memory copies, which affects performance and power consumption.
  • a common solution is to use specific instructions (such as NEON instructions) for data parallel processing to speed up the copy process. Or use line-by-line and multi-line copying instead of pixel-by-pixel copying to speed up the copying process.
  • HEVC high efficiency video coding
  • the present application provides an image encoding method.
  • the address of the storage memory of the input image is directly mapped to the encoder, so that the encoder can directly read and encode data based on the mapped address without performing Memory copy, without allocating additional memory to carry slice data, reduces the performance and power consumption overhead caused by memory copy, and improves coding efficiency.
  • FIG. 3 it is a schematic flowchart of an image encoding method provided by the present application, as described below.
  • the input image may be a received image, or may be an image captured by a local camera, and the input image may be stored in a memory for subsequent encoding.
  • the information of the storage memory of the input image may include information such as an address or a length of the storage memory.
  • a continuous memory can be allocated for the input image, and the information of each pixel in the input image can be stored in the memory.
  • the input image can be stored in the memory in the form of an array.
  • a continuous memory is allocated for the input image, and the information of the input image is stored in the allocated memory according to the arrangement of pixels.
  • the input image can be stored in the form of an array in the memory, and information such as the size and address of the storage memory can be recorded.
  • the allocated memory can be used to save the information of each pixel of the input image, such as chrominance value, brightness value and other information.
  • additional memory can be allocated for the input image.
  • a continuous storage memory can be allocated for the input image based on the preset slice size, and the storage memory can be divided into two parts, which are called the first memory and the second memory for easy distinction.
  • the memory is used to save the information of each pixel in the input image, and the second memory is used to align the storage memory according to the preset slice size, so that the subsequent encoder can read data from the input address in units of the preset slice size .
  • the storage memory can be allocated for the input image based on the preset slice size, wherein the storage memory can be understood as two parts, which are called the first memory and the second memory for easy distinction,
  • the first memory is used to save the data of the input image, such as the brightness value, gray value or chrominance value of each pixel in the input image
  • the second memory is used to make the size of the memory allocated for the input image according to the preset
  • the slice size is aligned so that subsequent slices can be sliced according to the preset slice size, so that the encoder can completely read the memory address of each memory slice, and encode in units of each memory slice.
  • the preset slice size when allocating memory for the input image, it can be allocated according to a preset slice size, the preset slice size includes a preset height and a preset width, and the height of the memory allocated for the input image is aligned according to the preset height, The width of the storage memory allocated for the input image is aligned according to the preset width, so that when the subsequent memory slice is performed, the memory slice with the size of the preset slice size can be directly sliced, so that the subsequent encoder can read the memory address avoid crossing the line.
  • the height of the allocated storage memory is aligned according to the preset height, and the width of the allocated storage memory is allocated according to the actual required memory of the input image, that is, the second memory can be divided into multiple
  • the part is called the third memory and the fourth memory for easy distinction.
  • the third memory is combined with the first memory and arranged in an array, that is, the height of the storage memory is aligned with the preset height through the third memory.
  • the last row of the array except Combining the remainder after the preset width with the fourth memory is not less than the preset width. That is, in order to reduce the amount of allocated memory, the width of the array may not be aligned according to the preset width, and an extra section of memory is allocated in the last line of the array to avoid subsequent encoder reading out of bounds.
  • one of the foregoing slicing manners may be pre-deployed in the electronic device, and multiple foregoing slicing manners may also be pre-deployed.
  • one of the slicing methods can be selected from the various slicing methods according to the preset method for slicing, which can be performed according to the actual application scenario. Adjustment.
  • the height of the storage memory can be aligned and allocated only according to the preset height, thereby forming an array, And additionally allocate multiple bytes for the last row of the array, so that the remainder of the last row of the array divided by the preset width is not smaller than the preset width after combining with the additional allocated multiple bytes, so as to avoid the encoder from reading Out of bounds;
  • the available memory of the electronic device is greater than the second threshold but not greater than the third threshold, it can be aligned according to the preset slice size, that is, the height of the storage memory is divisible by the preset height, and the width of the storage memory is divisible by the preset width , so that the follow-up can be sliced according to the preset slice size, so that the encoder performs encoding in units of memory slices without memory copying;
  • the available memory of the electronic device is greater than the third threshold, it means
  • the input image may be segmented to obtain multiple memory slices. It is equivalent to segmenting the storage memory of the input image, so that the storage memory is divided into multiple grids, that is, memory slices.
  • the storage memory may be divided according to a preset size, ie, a preset slice size, to obtain multiple memory slices (Grid). That is, it is equivalent to segmenting the input image, dividing the input image into multiple parts, so that the subsequent encoder can encode the input image.
  • a preset size ie, a preset slice size
  • the preset slice size may include a size preset by the user, a size negotiated between the upper layer and the lower layer of the electronic device, a size set according to the encoding length of the encoder, or a random size, etc., which may be determined according to actual application scenarios.
  • each memory slice After memory slicing is performed, the address of each memory slice is input to an encoder for encoding, thereby obtaining encoded data.
  • the data to be encoded can be extracted from each memory slice based on the input address, thereby encoding the data in each memory slice to obtain encoded data.
  • the encoder may be a pre-deployed encoder in the electronic device, and may perform encoding according to a preset encoding rule. For example, a piece of code can be deployed and run in an electronic device to read data from the address input to the encoder, and encode the image according to a preset encoding method, which can be HEVC encoding or advanced video encoding (advanced video coding, AVC) and other encoding methods, etc., can be adjusted according to actual application scenarios, and this application does not limit the specific encoding methods.
  • Grid is the encoding unit of the encoder, and the encoder encodes the data stored in the memory of a Grid as a unit, and can map the memory address corresponding to each Grid to the encoder, so that the encoder can obtain the memory of the Grid Information, including width, height, Stride, etc.
  • the address of the data to be encoded can be directly input to the encoder, so that the encoder can directly extract the data to be encoded based on the address, thus reducing the memory overhead for memory copying and improving the encoding efficiency. efficiency.
  • the storage memory is not divisible by the preset slice size, then the storage memory is divided to obtain multiple memory slices and the first slice, the first slice is the storage memory according to the preset The slice size is the remainder after slicing. Subsequently, the addresses of multiple memory slices may be mapped to the encoder for encoding, and the first data may be copied from the address corresponding to the first slice, and the first data may be used as an input of the encoder to obtain encoded data.
  • the address of the obtained memory matching the preset slice size can be mapped to the encoder for encoding, and the remaining Partially copy the data and use it as the input of the encoder to obtain the final encoded data without copying all the memory in the storage memory, reducing the amount of memory copying, reducing the memory overhead of memory copying, and improving encoding efficiency.
  • the storage memory is not completely aligned according to the preset slice size when allocating memory for the input image, for example, only the height is aligned but the width is not aligned, or the height and width are not aligned, etc.
  • the storage memory can be divided according to the preset slice size. Then the obtained memory slices can be divided into two types, which are called first memory slices and second memory slices for easy distinction.
  • the height of the first memory slice is the preset height, and the height of the second memory slice is smaller than the preset height, or
  • the second memory slice is the last slice in the last row, and the first memory slice and the second memory slice are different memory slices, that is, the type of the same memory is either the first memory slice or the second memory slice.
  • the fifth memory can be allocated for the second memory slice, and the data in the second memory can be copied to the fifth memory.
  • the width of the fifth memory is divisible by the preset width, and the height of the fifth memory is divisible by the preset height.
  • the data stored in the second memory slice is copied to the fifth memory.
  • the memory slice of the last row and last column can be used as the second memory slice, and the remaining memory slices can be divided into the first memory slice .
  • a fifth memory is allocated for the second memory slice, the size of the fifth memory may be a preset slice size, and the data stored in the second memory slice is copied to the fifth memory. Then, the addresses of the first memory slice and the fifth memory are input to the encoder, so that the encoder reads and encodes data from the input addresses to obtain final encoded data.
  • the part whose height is not divisible by the preset height that is, the part corresponding to the last row in the storage memory
  • the memory slice acts as the second memory slice. And allocate a fifth memory for the second memory slice, and copy the data stored in the second memory slice to the fifth memory. Then, the addresses of the first memory slice and the fifth memory are input to the encoder, so that the encoder reads and encodes data from the input addresses to obtain the final encoded data, avoiding the situation that the encoder reads out of bounds.
  • the system architecture can be based on the MediaCodec/OMX framework and the HEVC encoder to implement HEIC encoding, and can include multiple layers, such as the application layer, FrameWork layer and HAL/vendor layer, etc.
  • APP can be installed in the application layer, and can be implemented by the APP. Trigger the encoder to encode.
  • some APPs that can obtain input images can be installed in the electronic device, such as cameras or communication software that can generate or receive images.
  • the camera can be used to capture images and trigger the step of saving images; Images sent by other electronic devices, and trigger steps to save images, etc.
  • HEIF encoding In Android FrameWork, HEIFWriter and HEIFEncoder have been implemented. HEIFWriter is used to provide an interface for the application layer (APK) for the application layer to input original image data. HEIFWriter can call the interface of HEIFEncoder. HEIFWriter is used to complete the file packaging and generate . HEIC file; HEIFEncoder can call MediaCodec interface to create HEIC encoder and start HEIF encoding.
  • API application layer
  • HEIFWriter can call the interface of HEIFEncoder.
  • HEIFWriter is used to complete the file packaging and generate .
  • HEIC file; HEIFEncoder can call MediaCodec interface to create HEIC encoder and start HEIF encoding.
  • MediaCodec is a class for encoding and decoding audio and video, and realizes the function of encoding and decoding by accessing the underlying codec. It is part of the media infrastructure of electronic devices.
  • OMX OpenMAX (Open Media Acceleration)
  • OpenMAX Open Media Acceleration
  • the HEIC encoder as the OMX IL (OpenMAX Integration Layer, interface between the OMX framework layer and the component layer) layer, completes the docking with the OMX framework.
  • OMX IL OpenMAX Integration Layer, interface between the OMX framework layer and the component layer
  • its core function is to complete HEIF slicing.
  • Gralloc Graphic Buffer Allocator
  • Android Graphic Buffer Allocator
  • HEVC encoder Its input is the memory information of each Grid obtained after slicing by the HEIC encoder, such as the width of the grid corresponding to the picture, the height of the grid corresponding to the picture, stride and other information, used to read data from the input Grid memory information, And perform HEVC encoding on the data in each Grid respectively.
  • the process of encoding an image may be as shown in FIG. 6 .
  • the usage is set by the HEIC encoder, and the usage is an identifier for identifying whether the encoding type of the input image is HEIC encoding.
  • the HEIC encoder sets specific attributes for the Input Buffer, such as marking whether to slice and encode according to the method provided by this application through the graphic buffer usage. It can be understood that, in the process of negotiating Buffer attributes between the HEIC encoder module and OMX, this application adds a specific graphic buffer usage, which is used to identify the slice pre-alignment attribute of the HEIC encoding slice scene, that is, which slice method is used for alignment And slice. Which method is selected from the slicing methods in step 603 below for slicing.
  • OMX can read the graphic buffer usage, and after identifying the type of usage, it can know that the current HEIC encoding scene is. At this point, you can set a new usage, which also needs to be globally unique, to identify the HEIC encoding scene, as an attribute of the Input Buffer, and can be recognized by Gralloc, or directly pass the graphic buffer usage to Gralloc, so that Gralloc recognizes the current HEIC Code the scene.
  • Gralloc allocates memory based on usage
  • one or more slicing methods can be pre-deployed. After Gralloc gets the usage, it can recognize the current HEIC encoding scene and allocate corresponding memory.
  • Gralloc can recognize the current HEIC encoding scene after getting the usage, and then perform memory allocation.
  • Gralloc gets the usage, it can identify which method to use for slicing based on the usage, and then allocate the corresponding memory.
  • the memory allocated by each slicing method may be the same or may not be the same, and the matching slicing method can be selected according to the actual application scenario, for example, when the available memory of the electronic device is sufficient, it can Randomly select one of the slicing methods.
  • the available memory of the electronic device is insufficient, you can choose the slicing method that consumes the least memory, or when the coding efficiency is high, you can choose the slicing method that consumes less memory.
  • Adjustments may be made according to actual application scenarios, and this application is only an illustration and not a limitation.
  • Gralloc allocates memory based on usage
  • the information of the allocated memory can be passed to the HEIC encoder, and the HEIC encoder will slice it, and transmit the information of each slice obtained after slicing to the HEVC encoder, so that HEVC An encoder reads data from a slice and encodes it.
  • the Android system can be sliced by the FrameWork layer (or sliced by the application layer) or sliced by the encoder (usually called slice sinking).
  • the HEVC encoder is created by the upper layer: the upper layer can negotiate the slice size with the lower layer through Mediacodec/OMX, or preset the slice size, and then the upper layer slices based on the slice size, and the address of each slice is passed through Mediacodec /OMX is passed to the encoder, and the encoder extracts and encodes the data to be encoded based on the incoming address.
  • the encoded data can be carried by the Output Buffer and passed to the upper layer.
  • the upper layer generates and feeds back the end of stream (EOS), which indicates that the encoding is completed. After the upper layer receives the encoded data of all slices of the image, it can be encapsulated into a HEIC file.
  • EOS end of stream
  • the HEIC encoder is created by the upper layer.
  • the upper layer can negotiate the slice size with the lower layer through Mediacodec/OMX, or pre-set the slice size, and then the upper layer transmits the image to the lower layer, and the encoder slices the image and completes the encoding. , after the encoding of the last slice is completed, EOS is reported, and after the upper layer receives the encoded data of all slices of the image, it can be encapsulated into a HEIC file.
  • the upper layer and the lower layer are relative concepts.
  • the layer above the FrameWork layer can be understood as the upper layer
  • the layer below the FrameWork layer can be called the lower layer.
  • it can be divided according to the actual application scenario. Not as limiting.
  • the input image can be divided into multiple parts of the same size, that is, the storage memory corresponding to the input image can be divided into multiple grids of the same size, and then related parameters can be configured as the input of the HEVC encoder.
  • the width of the allocated memory is called stride
  • the height of the allocated memory is called Height
  • the width of each Grid is called Grid width
  • the height of each Grid is called Grid height
  • the Grid width*Grid height can be understood as the aforementioned preset slice size.
  • the input image can be allocated memory and sliced in various ways, which will be introduced as examples below.
  • the stride of the memory can be aligned according to the Grid width, which can be understood as the stride of the memory can be divisible by the Grid width, and the height of the memory can be aligned according to the Grid height.
  • the size of the memory occupied by the data in the input image may not be divisible by the preset slice size, and the size of each Grid usually needs to be consistent.
  • the size of the memory occupied by the data in the input image is not divisible by the size of the Grid, then when allocating memory for the Input Buffer, the stride of the allocated memory is aligned according to the Grid width, which can be the last A column of Grid and the last row of Grid allocate memory, and the memory information of each Grid (including the storage memory address allocated for the data included in the input image and the address of additional allocated memory) is passed to the encoder, so that the encoder can read When fetching memory, you can read data from the memory information corresponding to each Grid, so as to avoid reading illegal memory and causing the program to crash or the encoder to fail to work properly. And without memory copying, the encoder can directly read the input image from the input address, which reduces the process of memory copying, reduces memory overhead, and improves coding efficiency.
  • the encoder can encode in the unit of Grid, and the storage memory of the input image is usually a continuous memory, so the encoder can read the data of each Grid according to the starting address of each Grid, based on memory mapping
  • the encoder can read the data of each Grid according to the starting address of each Grid, based on memory mapping
  • N bytes can be allocated for the last Grid, that is, the width of the last line of the last Grid is Grid width, so that the encoder can read the data of a complete grid, thereby avoiding reading illegal memory , so that the encoder generates encoding errors or affects the encoding process of the encoder, etc.
  • the Height of the allocated memory is aligned according to the Grid height, and the Stride does not perform Grid width alignment; on this basis, more N bytes of memory are allocated.
  • N N ⁇ ((Grid width)–(picture width%Grid width))*4
  • the obtained N value is the minimum value to ensure that the encoder does not cross the boundary when reading the memory, so as to avoid reading Access to illegal memory, so that the encoder produces encoding errors or affects the encoding process of the encoder, etc.
  • the Input Buffer allocates memory
  • the Height is aligned according to the size of the Grid
  • the Stride is valued according to the width of the storage memory (or conventionally, aligned according to 16 bytes, 32 bytes, or 64 bytes, and the specific number is determined by the system itself. )
  • the Grid in the lower right corner of Figure 9 is mapped to the encoder, therefore, more N bytes of memory can be allocated to avoid reading illegal memory, which will cause encoding errors in the encoder or affect encoding The encoding process of the device, etc.
  • Method 3 Some parts use memory address mapping, and the rest use memory copy
  • the address of the part of the storage memory of the input image that can be divisible by the size of the Grid can be mapped to the encoder, and the remaining part can be input to the encoder by means of memory copy.
  • the memory occupied by the data of the input image may be divided into many situations, for example, the height is divisible by the height of the Grid, or the height is not divisible by the height of the Grid.
  • the storage memory can be allocated according to the actual memory required by the input image, and the storage memory can be divided. It is the first memory slice and the second memory slice, as shown in Figure 10, the second memory slice is the last memory slice in the last line, its width is not divisible by the Grid width, and the rest are the first memory slice.
  • the memory copy of the last slice of the last line can be performed, that is, additional memory is allocated for the last memory slice (ie, the fifth memory slice) ), and then copy the data in the last slice to the additionally allocated memory. Then both the address of the first memory slice and the address of the fifth memory are used as the input of the encoder.
  • the storage memory of the input image can be divided according to the preset Grid size, and the memory of the divisible part can be transferred to the encoder through memory address mapping, while the rest of the non-divisible part can be allocated to it.
  • the memory height required by the input image data is not divisible by the grid height
  • no additional memory may be allocated for the input image
  • the storage memory may be allocated according to the actual size of the input image.
  • the storage memory is directly divided according to the size of the Grid.
  • the obtained memory slices can be divided into the first memory slice and the second memory slice.
  • the first memory slice is the slice whose height is the Grid height
  • the second memory slice is the slice whose height is smaller than the Grid height slice.
  • allocate additional memory that is, the fifth memory
  • copy the data in the second memory slice to the additionally allocated memory.
  • both the address of the first memory slice and the address of the fifth memory are used as the input of the encoder.
  • the divisible part of the storage memory can be transmitted to the encoder through address mapping, and it is only necessary to allocate memory for the part that is not divisible for memory copying, which reduces the amount of memory copying and Memory overhead, improving coding efficiency.
  • the HEVC encoder After the HEVC encoder receives the configuration information of each Grid, such as the starting address and length of each Grid, it can perform HEVC encoding.
  • the configuration information may include information such as the start address, width or length of each Grid, so that the encoder can read and encode data in each Grid.
  • the received configuration information is as follows:
  • Stride stride of Input Buffer
  • Width Grid width
  • Stride represents the width of the memory
  • Width represents the width of each Grid
  • Height represents the height of each Grid.
  • the encoding is performed by means of memory copy and memory address mapping provided by this application, and the comparison is made by calculating the encoding time.
  • HEIC encoding time the moment when HEIC completes encoding (all grids complete HEVC encoding and report EOS) - the moment when HEIC starts encoding (the moment when the HEIC encoder receives the original image).
  • the memory copy method takes more than 80ms, while the encoding method provided by this application only needs 45ms at least, which greatly improves the encoding efficiency.
  • the above is based on a certain chip test, and the time-consuming data of different hardware may be different.
  • the storage memory of the image can be segmented, and the grid address obtained by the segmentation can be mapped to the encoder, so that additional allocated memory can be reduced, memory overhead can be reduced, and encoding efficiency can be improved. Furthermore, when encoding, in order to prevent the encoder from reading illegal memory addresses, a certain amount of memory can be allocated in various ways, which can prevent the encoder from reading illegal memory, thereby causing encoding errors or affecting the encoder encoding process, etc.
  • FIG. 13 it is a schematic structural diagram of an image coding device provided by the present application, and the image coding device is used to execute the method steps corresponding to the aforementioned FIGS. 3-12 .
  • the image encoding device includes:
  • An allocation module 1301, configured to allocate storage memory for the input image
  • the encoding module 1303 is configured to input the address of at least one memory slice in the plurality of memory slices to the encoder, and the encoder is configured to read and encode data based on the input address to obtain encoded data.
  • the division module 1302 is specifically configured to divide the storage memory into multiple memory slices according to a preset slice size, where the preset slice size includes a preset height and a preset width.
  • the division module 1302 is specifically configured to: allocate storage memory for the input image based on a preset slice size.
  • the storage memory includes a first memory and a second memory, the first memory stores the information of each pixel in the input image, and the second memory is used to make the size of the storage memory according to the preset Slice size alignment so that the encoder reads data from the input address in units of the preset slice size.
  • the storage memory is arranged in an array, the width of the storage memory is divisible by a preset width, and the height of the storage memory is divisible by a preset height.
  • the second memory includes a third memory and a fourth memory
  • the first memory and the third memory are combined and arranged in an array
  • the height of the array is divisible by the preset height
  • the last row of the array is divided by the preset height.
  • the remainder after setting the width is not smaller than the preset width after being combined with the fourth memory.
  • multiple memory slices are divided into a first memory slice and a second memory slice, the height of the first memory slice is a preset height, and the height of the second memory slice is smaller than the preset height or the second memory slice It is the last slice of the last line in the array composed of multiple memory slices, the first memory slice and the second memory slice are different memory slices;
  • the allocation module 1301 is further configured to allocate a fifth memory for the second memory slice among the plurality of memory slices, and copy the data in the second memory to the fifth memory.
  • the width of the fifth memory is divisible by the preset width.
  • the height of the five memory is divisible by the preset height;
  • the encoding module 1303 is specifically configured to input the address of the first memory slice and the address of the fifth memory in the plurality of memory slices to the encoder.
  • the height of the storage memory is evenly divisible by the preset height.
  • FIG. 14 is a schematic structural diagram of another electronic device provided by the present application, as described below.
  • the electronic device may include the aforementioned wearable device, terminal, or vehicle, etc., and the electronic device may include a processor 1401 , a memory 1402 , and a transceiver 1403 .
  • the processor 1401 and the memory 1402 are interconnected by wires. Wherein, program instructions and data are stored in the memory 1402 .
  • the memory 1402 stores program instructions and data corresponding to the aforementioned steps in FIGS. 3-12 .
  • the processor 1401 is configured to execute the method steps performed by the first device or the electronic device shown in any one of the foregoing embodiments in FIG. 3 to FIG. 12 .
  • the transceiver 1403 is configured to execute the step of receiving or sending data performed by the first device or the electronic device shown in any one of the embodiments in FIG. 3 to FIG. 12 .
  • the embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium is stored with a program for generating the driving speed of the vehicle.
  • the illustrated embodiments describe steps in methods.
  • the aforementioned electronic device shown in FIG. 14 is a chip.
  • the embodiment of the present application also provides an electronic device, which can also be called a digital processing chip or a chip.
  • the chip includes a processing unit and a communication interface.
  • the processing unit obtains program instructions through the communication interface.
  • the program instructions are executed by the processing unit.
  • the unit is configured to execute the method steps executed by the electronic device shown in any one of the foregoing embodiments in FIG. 3 to FIG. 12 .
  • the embodiment of the present application also provides a digital processing chip.
  • the digital processing chip integrates circuits and one or more interfaces for realizing the above processor 1401 or the functions of the processor 1401 .
  • the digital processing chip can complete the method steps in any one or more of the foregoing embodiments.
  • no memory is integrated in the digital processing chip, it can be connected to an external memory through a communication interface.
  • the digital processing chip implements the actions performed by the electronic device in the above-mentioned embodiments according to the program code stored in the external memory.
  • the embodiment of the present application also provides a computer program product that, when running on a computer, causes the computer to execute the steps performed by the electronic device in the method described in the foregoing embodiments shown in FIGS. 3-12 .
  • the electronic device provided in the embodiment of the present application may be a chip, and the chip includes: a processing unit and a communication unit, the processing unit may be, for example, a processor, and the communication unit may be, for example, an input/output interface, a pin, or a circuit.
  • the processing unit can execute the computer-executed instructions stored in the storage unit, so that the chip in the server executes the device search method described in the embodiments shown in FIGS. 3-12 above.
  • the storage unit is a storage unit in the chip, such as a register, a cache, etc.
  • the storage unit may also be a storage unit located outside the chip in the wireless access device, such as only Read-only memory (ROM) or other types of static storage devices that can store static information and instructions, random access memory (random access memory, RAM), etc.
  • ROM Read-only memory
  • RAM random access memory
  • the aforementioned processing unit or processor may be a central processing unit (central processing unit, CPU), a network processor (neural-network processing unit, NPU), a graphics processing unit (graphics processing unit, GPU), digital signal processing (digital signal processor, DSP), application specific integrated circuit (ASIC) or field programmable logic gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
  • a general-purpose processor may be a microprocessor or any conventional processor or the like.
  • the processor mentioned in any of the above-mentioned places may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the program execution of the above-mentioned methods in FIGS. 3-12 .
  • the device embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, and the components shown as units may or may not be A physical unit can be located in one place, or it can be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • the connection relationship between the modules indicates that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines.
  • the essence of the technical solution of this application or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product is stored in a readable storage medium, such as a floppy disk of a computer , U disk, mobile hard disk, read only memory (read only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disk, etc., including several instructions to make a computer device (which can be a personal A computer, server, or network device, etc.) executes the methods described in various embodiments of the present application.
  • a computer device which can be a personal A computer, server, or network device, etc.
  • all or part of them may be implemented by software, hardware, firmware or any combination thereof.
  • software When implemented using software, it may be implemented in whole or in part in the form of a computer program product.
  • the computer program product includes one or more computer instructions.
  • the computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices.
  • the computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website, computer, server, or data center Transmission to another website site, computer, server, or data center by wired (eg, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (eg, infrared, wireless, microwave, etc.).
  • wired eg, coaxial cable, optical fiber, digital subscriber line (DSL)
  • wireless eg, infrared, wireless, microwave, etc.
  • the computer-readable storage medium may be any available medium that can be stored by a computer, or a data storage device such as a server or a data center integrated with one or more available media.
  • the available medium may be a magnetic medium (such as a floppy disk, a hard disk, or a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (solid state disk, SSD)), etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)

Abstract

La présente invention concerne un procédé de codage d'image et un dispositif, qui sont utilisés pour coder une image stockée par un moyen de mappage de mémoire interne, réduire la copie de mémoire interne, et améliorer l'efficacité de codage. Le procédé consiste à : d'abord allouer une mémoire interne de stockage pour une image d'entrée ; diviser la mémoire interne de stockage en une pluralité de sections de grille de mémoire interne ; puis lire des données à partir de l'adresse d'au moins une section de grille de mémoire interne parmi la pluralité de sections de grille de mémoire interne et effectuer un codage au moyen d'un codeur, et obtenir des données codées.
PCT/CN2022/142277 2021-12-29 2022-12-27 Procédé de codage d'image et dispositif WO2023125518A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111640001.9A CN116418994A (zh) 2021-12-29 2021-12-29 一种图像编码方法以及装置
CN202111640001.9 2021-12-29

Publications (1)

Publication Number Publication Date
WO2023125518A1 true WO2023125518A1 (fr) 2023-07-06

Family

ID=86997926

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/142277 WO2023125518A1 (fr) 2021-12-29 2022-12-27 Procédé de codage d'image et dispositif

Country Status (2)

Country Link
CN (1) CN116418994A (fr)
WO (1) WO2023125518A1 (fr)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117094879B (zh) * 2023-10-18 2024-03-26 南京砺算科技有限公司 数据拷贝方法及装置、计算机可读存储介质、电子设备

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100061464A1 (en) * 2008-06-03 2010-03-11 Fujitsu Limited Moving picture decoding apparatus and encoding apparatus
CN104216835A (zh) * 2014-08-25 2014-12-17 杨立群 一种实现内存融合的方法及装置
CN108496364A (zh) * 2017-06-21 2018-09-04 深圳市大疆创新科技有限公司 视频传输方法、视频处理器、网络处理器和视频传输设备
CN111538677A (zh) * 2020-04-26 2020-08-14 西安万像电子科技有限公司 数据处理方法及装置

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100061464A1 (en) * 2008-06-03 2010-03-11 Fujitsu Limited Moving picture decoding apparatus and encoding apparatus
CN104216835A (zh) * 2014-08-25 2014-12-17 杨立群 一种实现内存融合的方法及装置
CN108496364A (zh) * 2017-06-21 2018-09-04 深圳市大疆创新科技有限公司 视频传输方法、视频处理器、网络处理器和视频传输设备
CN111538677A (zh) * 2020-04-26 2020-08-14 西安万像电子科技有限公司 数据处理方法及装置

Also Published As

Publication number Publication date
CN116418994A (zh) 2023-07-11

Similar Documents

Publication Publication Date Title
CN109559270B (zh) 一种图像处理方法及电子设备
US20230247093A1 (en) Data Synchronization Method, Electronic Device, and Computer-Readable Storage Medium
CN113448382B (zh) 多屏幕显示电子设备和电子设备的多屏幕显示方法
WO2021047567A1 (fr) Procédé et dispositif de traitement de flux de rappel
WO2021077878A1 (fr) Procédé et appareil de traitement d'image et dispositif électronique
WO2022100685A1 (fr) Procédé de traitement de commande de dessin et dispositif associé
EP4361954A1 (fr) Procédé de reconstruction d'objet et dispositif associé
CN114461375B (zh) 内存资源管理方法及电子设备
WO2022027972A1 (fr) Procédé de recherche de dispositif et dispositif électronique
WO2022062809A1 (fr) Procédé et appareil de commande de projection d'écran
WO2021057626A1 (fr) Procédé de traitement d'image, appareil, dispositif et support de stockage informatique
WO2023125518A1 (fr) Procédé de codage d'image et dispositif
CN113973398A (zh) 无线网络连接方法、电子设备及芯片系统
CN116389884B (zh) 缩略图显示方法及终端设备
CN115119048B (zh) 一种视频流处理方法及电子设备
WO2023051094A1 (fr) Procédé et appareil de récupération de mémoire, dispositif électronique et support de stockage lisible
WO2022121988A1 (fr) Procédé de synchronisation d'affichage, dispositif électronique et support de stockage lisible
WO2022135195A1 (fr) Procédé et appareil permettant d'afficher une interface de réalité virtuelle, dispositif, et support de stockage lisible
WO2022170866A1 (fr) Procédé et appareil de transmission de données, et support de stockage
CN115482143B (zh) 应用的图像数据调用方法、系统、电子设备及存储介质
WO2023016059A1 (fr) Procédé de commande de transmission de données et appareil associé
WO2024082713A1 (fr) Procédé et appareil de rendu d'image
CN114125805B (zh) 蓝牙回连方法及终端设备
WO2023109569A1 (fr) Procédé de stockage de fichier, procédé d'accès à un fichier, dispositif électronique et support de stockage
WO2024093795A1 (fr) Procédé et appareil de configuration de remplacement de dispositif

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22914771

Country of ref document: EP

Kind code of ref document: A1