KR102638468B1 - Electronic apparatus and operating method thereof - Google Patents

Abstract

Various embodiments of the present disclosure propose a video compression device and method that can increase compression efficiency while maintaining video quality in electronic devices that process multimedia data, including video. To this end, a video compression device that acquires a target image from a memory or a camera, generates metadata corresponding to attribute information of the acquired target image, and compresses the target image based at least on the generated metadata; and A method can be provided.

Description

Electronic device and operating method thereof {ELECTRONIC APPARATUS AND OPERATING METHOD THEREOF}

Various embodiments of the present disclosure relate to electronic devices and methods of operating the same.

Recently, broadcasting services that support various types of multimedia services are converging with communication technology. These video communication services are based on electronic devices capable of rapid information processing as well as a broadband network that supports high-speed information transmission.

Electronic devices that support the video communication service may consume a lot of power due to image processing. In particular, the resolution of the image to be processed can be a major factor in determining the power consumption of electronic devices during display. For example, in portable electronic devices (hereinafter referred to as 'portable terminals'), power consumption during display may increase in proportion to the resolution of the image to be processed.

The increase in image resolution may result in increased bandwidth or increased storage space on the link carrying information about the image to be processed. For example, the bandwidth for an application processor (AP) within one electronic device to transmit a frame to a display device or the bandwidth for one electronic device to transmit multimedia data to another electronic device is determined by the display resolution or the multimedia data to be transmitted. It can increase in proportion to the size of .

In response to user demands for improved picture quality, electronic devices such as mobile terminals have become capable of recording high-definition/large-screen (4K UHD, tens of Mbps) videos. In addition, video codec technology based on HEVC, which has improved compression efficiency, has been applied to electronic devices.

However, electronic devices can use a large amount of resources to process high-definition or large-capacity video (4K UHD video, etc.). For example, in order to perform large amounts of data operations related to conversion or correction of high-definition images, electronic devices require a relatively large amount of memory or processing resources. Additionally, in order to transmit a large amount of video to another electronic device, the electronic device can use a relatively large amount of networking resources to increase the data transmission amount or transmission speed.

Therefore, it was necessary for electronic devices to come up with a method to efficiently compress video while maintaining the image quality of the captured video.

According to various embodiments of the present disclosure, it is possible to provide a video compression device and method that can increase compression efficiency while maintaining video quality in an electronic device that processes multimedia data including video.

According to various embodiments of the present disclosure, an apparatus and method for compressing a video using metadata in an electronic device that processes multimedia data including a video can be provided.

According to various embodiments of the present disclosure, a video compression apparatus and method are provided for removing unnecessary data from a compressed video stream and recompressing the video stream from which the unnecessary data has been removed in an electronic device that processes multimedia data including video. You can.

According to various embodiments of the present disclosure, a compression device and method for restoring and then recompressing a compressed video stream based on metadata in an electronic device that processes multimedia data, including video, can be provided.

According to various embodiments of the present disclosure, a compression device that stores variables (parameters) created when acquiring an image in an electronic device and performs compression using the stored variables when compressing a later acquired image; A method can be provided.

An electronic device according to various embodiments of the present disclosure includes a memory, an image signal processor that acquires a target image from the memory or a functionally connected camera, and acquires metadata corresponding to attribute information of the acquired target image; It may include a processor that compresses the target image based on at least metadata.

A method of compressing a video in an electronic device according to various embodiments of the present disclosure includes a process of acquiring a target image from a memory or a camera, a process of generating metadata corresponding to attribute information of the acquired target image, and the generation of metadata corresponding to attribute information of the acquired target image. Compressing the target image based at least on metadata, wherein the process of compressing the target image includes restoring the compressed target image, and compressing the target image based at least on pre-stored metadata. It may include a process of recompressing.

A computer-readable recording medium of an electronic device according to various embodiments of the present disclosure includes an operation of acquiring a target image from a memory or a camera, an operation of generating metadata corresponding to attribute information of the acquired target image, and the generation of metadata corresponding to attribute information of the acquired target image. A program for executing an operation of compressing the target video based at least on metadata may be recorded.

Apparatus and methods for compressing video according to various embodiments of the present disclosure can not only improve compression efficiency of video without loss of image quality, but also use storage space efficiently. In addition, when compressing an image in a processor, rather than analyzing and compressing the image, the variables created when acquiring the image are stored, and the image is compressed using the stored variables, so the compression operation can be performed quickly. In addition, computational complexity may be lowered.

1 is a diagram illustrating a network environment including a wireless terminal according to various proposed embodiments.
FIG. 2 is a diagram illustrating the block configuration of an electronic device according to various proposed embodiments.
Figure 3 is a diagram illustrating the block configuration of a program module according to various proposed embodiments.
FIG. 4 is a diagram illustrating an example of the configuration of a video compression device that performs video compression in an electronic device according to various proposed embodiments.
FIG. 5 is a diagram illustrating control flow performed by an electronic device according to various embodiments proposed in the present disclosure.
FIG. 6 is a diagram illustrating a control flow in which an electronic device performs compression considering metadata, according to various embodiments proposed in the present disclosure.
FIG. 7 is a diagram illustrating a scenario in which an electronic device performs recompression according to various embodiments proposed in the present disclosure.
FIG. 8 is a diagram illustrating an example of an electronic device compressing or recompressing an image based on metadata, according to various embodiments proposed in the present disclosure.
FIG. 9 is a diagram illustrating a scenario in which an electronic device effectively performs a sample adaptive offset (SAO) process of high efficiency video coding (HEVC) using histogram information, according to various embodiments proposed in this disclosure.
FIG. 10 illustrates an example in which metadata is utilized when an electronic device performs perception-based video compression (perceptual video coding) according to various embodiments proposed in the present disclosure.

Hereinafter, various embodiments of the present disclosure are described with reference to the attached drawings. This is not intended to limit the technology described in this disclosure to specific embodiments, and should be understood to include various modifications, equivalents, and/or alternatives to the embodiments of this disclosure. In connection with the description of the drawings, similar reference numbers may be used for similar components.

In the present disclosure, expressions such as “have,” “may have,” “includes,” or “may include” refer to the presence of corresponding features (e.g., components such as values, functions, operations, or parts). , and does not rule out the existence of additional features.

In the present disclosure, expressions such as “A or B,” “at least one of A or/and B,” or “one or more of A or/and B” may include all possible combinations of the items listed together. . For example, “A or B”, “at least one of A and B”, or “at least one of A or B” (1) includes at least one A, (2) includes at least one B, or (3) it may refer to all cases including both at least one A and at least one B.

As used in this disclosure, expressions such as “first”, “second”, “first”, or “second” may modify various elements regardless of order and/or importance, and may refer to one element It is only used to distinguish from other components and does not limit the components. For example, a first user device and a second user device may represent different user devices regardless of order or importance. For example, a first component may be renamed a second component without departing from the scope of rights set forth in the present disclosure, and similarly, the second component may also be renamed to the first component.

A component (e.g., a first component) is “(operatively or communicatively) coupled with/to” another component (e.g., a second component). When referred to as being “connected to,” it should be understood that any component may be directly connected to the other component or may be connected through another component (e.g., a third component). On the other hand, when a component (e.g., a first component) is said to be “directly connected” or “directly connected” to another component (e.g., a second component), the component and the It may be understood that another component (eg, a third component) does not exist between other components.

The expression “configured to” used in the present disclosure may mean, for example, “suitable for,” “having the capacity to,” depending on the situation. It can be used interchangeably with ", "designed to," "adapted to," "made to," or "capable of." The term “configured (or set to)” may not necessarily mean “specifically designed to” in hardware. Instead, in some contexts, the expression “a device configured to” may mean that the device is “capable of” working with other devices or components. For example, the phrase "processor configured (or set) to perform A, B, and C" refers to a processor dedicated to performing the operations (e.g., an embedded processor), or by executing one or more software programs stored on a memory device. , may refer to a general-purpose processor (e.g., CPU or application processor) capable of performing the corresponding operations.

Terms used in the present disclosure are merely used to describe specific embodiments and may not be intended to limit the scope of other embodiments. Singular expressions may include plural expressions, unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as generally understood by a person of ordinary skill in the technical field described in this disclosure. Among the terms used in this disclosure, terms defined in general dictionaries may be interpreted to have the same or similar meaning as the meaning they have in the context of related technology, and unless clearly defined in this disclosure, have an ideal or excessively formal meaning. It is not interpreted as In some cases, even terms defined in the present disclosure cannot be interpreted to exclude embodiments of the present disclosure.

Electronic devices according to various embodiments of the present disclosure include, for example, a smartphone, a tablet personal computer, a mobile phone, a video phone, an e-book reader, Desktop PC (desktop personal computer), laptop PC (laptop personal computer), netbook computer, workstation, server, PDA (personal digital assistant), PMP (portable multimedia player), MP3 player, mobile medical It may include at least one of a device, a camera, or a wearable device. According to various embodiments, the wearable device may be an accessory (e.g., a watch, ring, bracelet, anklet, necklace, glasses, contact lenses, or a head-mounted-device (HMD)), a fabric or clothing piece ( It may include at least one of a body-attached type (e.g., an electronic garment), a body-attachable type (e.g., a skin pad or a tattoo), or a bioimplantable type (e.g., an implantable circuit).

In some embodiments, the electronic device may be a home appliance. Home appliances include, for example, televisions, DVD (digital video disk) players, stereos, refrigerators, air conditioners, vacuum cleaners, ovens, microwave ovens, washing machines, air purifiers, set-top boxes, and home automation controls. Home automation control panel, security control panel, TV box (e.g. Samsung HomeSyncTM, Apple TVTM, or Google TVTM), game console (e.g. XboxTM, PlayStationTM), electronic dictionary, electronic key, camcorder It may include at least one of a (camcorder), or an electronic picture frame.

In another embodiment, the electronic device may include various medical devices (e.g., various portable medical measurement devices (such as blood sugar monitors, heart rate monitors, blood pressure monitors, or body temperature monitors), magnetic resonance angiography (MRA), magnetic resonance imaging (MRI), CT (computed tomography), imaging device, or ultrasound device, etc.), navigation device, GPS receiver (global positioning system receiver), EDR (event data recorder), FDR (flight data recorder), automobile infotainment device, marine Electronic equipment (e.g. marine navigation systems, gyrocompasses, etc.), avionics, security devices, head units for vehicles, industrial or home robots, ATMs (automatic teller's machines) in financial institutions, POS in stores (point of sales), or internet of things (e.g. light bulbs, various sensors, electric or gas meters, sprinkler systems, fire alarms, thermostats, street lights, toasters, exercise equipment, It may include at least one of a hot water tank, heater, boiler, etc.).

According to some embodiments, the electronic device may be a piece of furniture or part of a building/structure, an electronic board, an electronic signature receiving device, a projector, or various measuring devices (e.g., It may include at least one of water, electricity, gas, or radio wave measuring devices, etc.). In various embodiments, the electronic device may be one or a combination of more than one of the various devices described above. An electronic device according to some embodiments may be a flexible electronic device. Additionally, electronic devices according to embodiments of the present disclosure are not limited to the above-described devices and may include new electronic devices according to technological developments.

In order to improve video compression efficiency, various embodiments proposed in this disclosure will provide a method of compressing or recompressing a target image using metadata in an electronic device capable of image processing.

According to one embodiment, to increase real-time compression efficiency, an electronic device may compress images captured through an external or internal camera using metadata. In this case, the electronic device may generate metadata corresponding to the captured image during image signal processing or image sensing. The generated metadata can be used to compress the target image. That is, the electronic device can compress the target image using the generated metadata. The electronic device may record the generated metadata corresponding to the target image in a separate storage area.

According to one embodiment, for efficient use of time and/or resources, an electronic device may recompress images stored in external or internal memory using metadata. The image stored in the external or internal memory may be an image compressed using an existing or proposed method. In this case, it may be desirable for the electronic device to perform recompression in a preset standby state. The standby state may be a state in which the electronic device can perform recompression with minimal impact on other operations. For example, the standby state may be a display off state or a charging state, or a display off state and a charging state.

For example, the electronic device may restore a compressed image stored in a memory and recompress the restored image using metadata corresponding to the compressed image. Metadata corresponding to the compressed video may be created in real time or may be created and stored in advance. During the recompression, the electronic device stores the target image in units of GOP (group of picture) while maintaining the user's subjective image quality or objective image quality (PSNR (peak signal to noise ratio), SSIM (structural similarity index)) above a certain level. Recompression can be performed. The electronic device can store the recompressed compressed image instead of the existing image stored in the memory. Here, the PSNR may be a value that can be expressed using numbers, units, or numbers that enable quantitative recognition of the difference between two different images. The SSIM means structural similarity index. For example, the SSIM is a method of measuring similarity to the original image with respect to distortion caused by compression and transformation, and may enable more accurate comparison than methods using MSE (mean square error) or PSNR.

According to one embodiment, the electronic device may compress or recompress the target image with the help of an external electronic device in a preset standby state. To this end, the electronic device may provide the target image and metadata corresponding to the target image to the external electronic device. The electronic device can receive a compressed image that has been compressed or recompressed using metadata from the external electronic device. The electronic device can record compressed images received from the external electronic device in a designated storage area or store them instead of existing images stored in memory.

The target image may be an image captured in real time by a camera or an image stored in memory. For example, the image stored in the memory may be an image compressed using an existing or proposed method.

The standby state may be a state in which an electronic device can communicate with an external electronic device with minimal influence on other operations. For example, the standby state may be a state in which the display is off but communication is possible, or a state in which the display is off but communication is possible during charging.

The external electronic device may be a cloud server, home server, home host computer, personal computer, etc.

According to various embodiments proposed in this disclosure, metadata referenced to compress the target image may be generated based on attribute information of the target image. For example, the attribute information of the target image includes color, brightness, exposure, histogram, region of interest, sharpness, complexity, texture, and skin tone corresponding to the target image. ), bitrate information, depth, or a combination thereof.

Hereinafter, electronic devices according to various embodiments will be described with reference to the accompanying drawings. In this disclosure, the term user may refer to a person using an electronic device or a device (eg, an artificial intelligence electronic device) using an electronic device.

FIG. 1 is a diagram illustrating an example of a network environment 100 according to various embodiments proposed in the present disclosure.

Referring to FIG. 1, the electronic device 101 in the network environment 100 includes a bus 110, a processor 120, a memory 130, an image processing module 140, an input/output interface 150, and a display 160. , and may include a communication interface 170. In some embodiments, the electronic device 101 may omit at least one of the components or may additionally include another component.

Bus 110 may include, for example, circuitry that connects components 120-170 to each other and carries communications (e.g., control messages and/or data) between components 120-170. You can.

The processor 120 may be a central processing unit (CPU), an application processor (AP), a communication processor (CP), or an image signal processor (ISP). It may include one or more of the following. The processor 120 may, for example, perform operations or data processing related to control, image signal processing, and/or communication of at least one other component of the electronic device 101.

When the ISP is included in the processor 120, the processor 120 acquires a target image, generates metadata corresponding to attribute information of the acquired target image, and at least based on the generated metadata. It may be set to compress or recompress the target image. The processor 120 may determine a method to compress or recompress the target image based at least on metadata.

The processor 120 may acquire a target image using the memory 130 or a functionally connected camera. In addition, the processor 120 may acquire the target image through a streaming service. The streaming service may include a service that receives video data from the external electronic devices 102 and 104 or the server 106 through the communication interface 170.

The processor 120 may generate metadata using attribute information of the acquired target image. The attribute information of the acquired target image includes color, brightness, exposure, histogram, region of interest, sharpness, complexity, texture, and skin tone corresponding to the target image. ), bitrate information, depth, or a combination thereof.

According to one embodiment, in order to increase compression efficiency during real-time video compression, the processor 120 may utilize metadata generated by an ISP (or image sensor, etc.) before compression when compressing video.

According to one embodiment, in order to efficiently create a compressed video stream, the processor 120 performs recompression of the target image while maintaining the user's subjective image quality or objective image quality (PSNR, SSIM) above a certain level. You can. In this case, video compression efficiency can be improved by using metadata (rate control, mode decision, texture information, etc.) obtained during image signal processing performed before video compression.

The processor 120 may perform operations of acquiring a target image and generating metadata in parallel. The processor 120 may check status information about the electronic device 101 before compressing the target image and determine whether the confirmed status information satisfies a specified condition. That is, the processor 120 can perform an operation of compressing the target image only when the status information about the electronic device 101 satisfies specified conditions. For example, the status information may include the remaining amount of memory 130, image signal processing performance, data amount of image signals that can be processed, charging status of the electronic device 101, external electronic device connected to the electronic device 101, or a combination thereof. may include.

For example, the processor 120 determines whether the remaining memory amount is within a specified range based on the remaining memory amount information included in the status information, and compresses the target image only when the remaining memory amount falls within the specified range. there is.

For example, as at least part of the operation of compressing the target image, the processor 120 may configure the resolution, image bit rate, image bit depth, image frame rate, image color space, and audio bit for at least a portion of the target image. It may be set to change at least one of the rate, voice bit depth, and voice sample rate, and determine the compression rate for at least some sections based on at least information according to the change.

For example, the processor 120 may be set to compress the first section of the target image using a first method (scheme), and may be set to compress the second section of the target image using the second method. In this case, the processor 120 may generate metadata using first metadata corresponding to the first section and second metadata corresponding to the second section. The processor 120 may generate first metadata with reference to a first method and generate second metadata with reference to a second method.

As at least part of the operation of compressing the target image, the processor 120 may determine a first method or a second method so that the compression result of the first section or the second section satisfies a preset image quality. For example, the processor 120 identifies a section containing a region of interest among at least some sections of the target image, and converts the target image using a bit rate at a level specified for the section containing the identified region of interest. You can decide how to compress.

Memory 130 may include volatile and/or non-volatile memory. For example, the memory 130 may store commands or data related to at least one other component of the electronic device 101. According to one embodiment, memory 130 may store software and/or program 180. The program 180 may include, for example, a kernel 181, a middleware 183, an application programming interface (API) 185, and/or an application program (or “application”) 187, etc. may include. At least a portion of the kernel 181, middleware 183, or API 185 may be referred to as an operating system (OS).

According to one embodiment, the memory 130 may store metadata and/or compressed or uncompressed image data in an area designated by the processor 120. For example, the memory 130 may store metadata as at least part of the target image.

The memory 130 may read images and/or metadata stored in a designated area in response to a request from the processor 120 and provide them to the processor 120. The memory 130 may record images recompressed by the processor 120 instead of previously recorded images.

If the processor 120 does not include an ISP, the electronic device 101 may be separately equipped with an image processing module 140. In this case, the image processing module 140 may perform the image signal processing operation on behalf of the processor 120.

In FIG. 1, the image processing module 140 is shown as a component independent of the processor 120 and the memory 130, but various embodiments are not limited thereto. For example, the image processing module 140 may be implemented integrated with the processor 120, and may be stored in the memory 130 in software form and implemented in a form that can be executed on the processor 120. You can. Additionally, the image processing module 140 may be implemented distributed over, for example, the processor 120 and the memory 130. In this case, the image processing module 140 may be implemented to perform an operation of generating metadata for image signal processing, and the processor 120 may be implemented to perform an operation of compressing the target image based at least on the generated metadata. You can.

Kernel 181, for example, provides system resources (e.g., middleware 183, API 185, or application program 187) used to execute operations or functions implemented in other programs (e.g., : Bus 110, processor 120, or memory 130, etc.) can be controlled or managed. In addition, the kernel 181 provides an interface for controlling or managing system resources by accessing individual components of the electronic device 101 in the middleware 183, API 185, or application program 187. You can.

The middleware 183 may, for example, perform an intermediary role so that the API 185 or the application program 187 can communicate with the kernel 181 to exchange data.

Additionally, the middleware 183 may process one or more work requests received from the application program 187 according to priority. For example, the middleware 183 may use system resources (e.g., bus 110, processor 120, or memory 130, etc.) of the electronic device 101 for at least one of the application programs 187. Priority can be given. For example, the middleware 183 may process one or more work requests according to the priority given to at least one application program, thereby performing scheduling or load balancing in response to the one or more work requests.

The API 185 is, for example, an interface for the application program 187 to control functions provided by the kernel 181 or middleware 183, such as file control, window control, image processing, or It may include at least one interface or function (e.g., command) for character control, etc.

For example, the input/output interface 150 may serve as an interface that can transmit commands or data input from a user or another external device to other component(s) of the electronic device 101. Additionally, the input/output interface 150 may output commands or data received from other component(s) of the electronic device 101 to the user or other external device. For example, the input/output interface 150 may include multiple image sensors. A plurality of image sensors included in the input/output interface 150 may have different characteristics. The input/output interface 150 can transmit images captured by a plurality of image sensors to the image processing module 140, memory 130, display 160, communication interface 170, etc. through the bus 110. . The captured images may have different image characteristics. This may be due to differences in the characteristics of the image sensor, conditions set for shooting, etc.

Display 160 may be, for example, a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting diode (OLED) display, or It may include a micro electromechanical systems (MEMS) display, or an electronic paper display. For example, the display 160 may display various contents (e.g., text, images, videos, icons, or symbols). The display 160 may include a touch screen and may receive, for example, a touch, gesture, proximity, or hovering input using an electronic pen or a part of the user's body.

The communication interface 170, for example, establishes communication between the electronic device 101 and an external device (e.g., the first external electronic device 102, the second external electronic device 104, or the server 106). You can. For example, the communication interface 170 may be connected to the network 162 through wireless or wired communication and communicate with an external device (eg, the second external electronic device 104 or the server 106).

Wireless communication is, for example, a cellular communication protocol, for example, LTE (long-term evolution), LTE-A (LTE Advance), CDMA (code division multiple access), WCDMA (wideband CDMA), UMTS (universal) At least one of mobile telecommunications system), WiBro (wireless broadband), or GSM (global system for mobile communications) may be used. Additionally, wireless communication may include, for example, short-range communication 164. The short-range communication 164 may include at least one of, for example, WiFi (wireless fidelity), Bluetooth (bluetooth), near field communication (NFC), magnetic stripe transmission (MST), or Zigbe. Wireless communication may use, for example, a global positioning system (GPS) or a global navigation satellite system (GNSS). Wired communication may include at least one of, for example, universal serial bus (USB), high definition multimedia interface (HDMI), recommended standard 232 (RS-232), or plain old telephone service (POTS). The network 162 may include at least one of a telecommunications network, for example, a computer network (eg, LAN or WAN), the Internet, or a telephone network.

According to one embodiment, the communication interface 170 may request compression of the target image from the external electronic devices 102 and 104 or the server 106 in response to control from the processor 120. To this end, the communication interface 170 may provide the target image and/or metadata corresponding to the target image to the external electronic device. The communication interface 170 may receive compressed video provided from an external electronic device 102 or 104 or a server 106, and transmit the received compressed video to the processor 120.

Each of the first and second external electronic devices 102 and 104 may be of the same or different type as the electronic device 101. According to one embodiment, server 106 may include one or more groups of servers. According to various embodiments, all or part of the operations performed in the electronic device 101 may be executed in one or more electronic devices (eg, the electronic devices 102 and 104, or the server 106). According to one embodiment, when the electronic device 101 needs to perform a certain function or service automatically or upon request, the electronic device 101 additionally performs the function or service instead of executing the function or service on its own. At least some of the related functions may be requested from another device (eg, the electronic device 102 or 104, or the server 106). Another electronic device (e.g., electronic device 102, 104, or server 106) may execute the requested function or additional function and transmit the result to electronic device 101. The electronic device 101 may provide the requested function or service by processing the received result as is or additionally. For this purpose, for example, cloud computing, distributed computing, or client-server computing technologies may be used.

FIG. 2 is a diagram illustrating the configuration of an electronic device 201 according to various embodiments proposed in the present disclosure.

Referring to FIG. 2, the electronic device 201 may include, for example, all or part of the electronic device 101 shown in FIG. 1. The electronic device 201 includes one or more processors (e.g., AP (application processor)) 210, a communication module 220, a subscriber identification module 224, a memory 230, a sensor module 240, and an input device 250. ), display 260, interface 270, audio module 280, camera module 291, power management module 295, battery 296, indicator 297, and motor 298. there is.

The processor 210, for example, can run an operating system or application program to control a number of hardware or software components connected to the processor 210, and can perform various data processing and calculations. The processor 210 may be implemented, for example, as a system on chip (SoC). According to one embodiment, the processor 210 may further include a graphic processing unit (GPU) and/or an image signal processor. The processor 210 may include at least some of the components shown in FIG. 2 (eg, the cellular module 221). The processor 210 may load commands or data received from at least one of the other components (eg, non-volatile memory) into the volatile memory, process them, and store various data in the non-volatile memory.

According to various embodiments proposed in the present disclosure, the processor 210 may be set to perform all operations performed by the processor 120 and/or the image processing module 140 described with reference to FIG. 1. there is. Since the detailed description for this is the same as that made with reference to FIG. 1, redundant description will be omitted.

The communication module 220 may have the same or similar configuration as the communication interface 170 of FIG. 1. The communication module 220 includes, for example, a cellular module 221, a WiFi module 223, a Bluetooth module 225, a GPS module 227, an NFC module 228, and a radio frequency (RF) module 229. may include.

For example, the cellular module 221 may provide voice calls, video calls, text services, or Internet services through a communication network. According to one embodiment, the cellular module 221 may use the subscriber identification module (eg, SIM card) 224 to distinguish and authenticate the electronic device 201 within the communication network. According to one embodiment, the cellular module 221 may perform at least some of the functions that the processor 210 can provide. According to one embodiment, the cellular module 221 may include a CP.

Each of the WiFi module 223, Bluetooth module 225, GPS module 227, or NFC module 228 may include, for example, a processor for processing data transmitted and received through the corresponding module. According to some embodiments, at least some (e.g., two or more) of the cellular module 221, WiFi module 223, Bluetooth module 225, GPS module 227, or NFC module 228 are one integrated chip. (IC) or may be included within an IC package.

For example, the RF module 229 may transmit and receive communication signals (eg, RF signals). The RF module 229 may include, for example, a transceiver, a power amp module (PAM), a frequency filter, a low noise amplifier (LNA), or an antenna. According to another embodiment, at least one of the cellular module 221, WiFi module 223, Bluetooth module 225, GPS module 227, or NFC module 228 transmits and receives RF signals through a separate RF module. You can.

According to one embodiment, the communication module 220 may be set to perform the same operations performed by the communication interface 170 in FIG. 1. That is, the communication module 220 may request compression of the target image from an external electronic device in response to control from the processor 210. To this end, the communication module 220 may provide the target image and/or metadata corresponding to the target image to the external electronic device. The communication module 220 may receive compressed video provided from an external electronic device and transmit the received compressed video to the processor 210.

Subscriber identity module 224 may include, for example, a card and/or an embedded SIM that includes a subscriber identity module and may include unique identification information (e.g., an integrated circuit card identifier (ICCID)) or May include subscriber information (e.g. IMSI (international mobile subscriber identity)).

Memory 230 (eg, memory 130) may include, for example, internal memory 232 or external memory 234. The memory 230 may record the target image and/or metadata corresponding to the target image in a designated area in response to the control of the processor 210. The memory 230 reads a specific target image and/or metadata corresponding to the specific target image in response to the control of the processor 210, and reads the metadata corresponding to the read specific target image and/or the specific target image. Data may be provided to the processor 210.

The built-in memory 232 may include, for example, volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM), etc.), non-volatile memory (e.g., OTPROM (one time programmable ROM), PROM (programmable ROM), EPROM (erasable and programmable ROM), EEPROM (electrically erasable and programmable ROM), mask ROM, flash ROM, flash memory (e.g. NAND flash or NOR flash, etc.), It may include at least one of a hard drive or a solid state drive (SSD).

The external memory 234 may be a flash drive, for example, CF (compact flash), SD (secure digital), Micro-SD (micro secure digital), Mini-SD (mini secure digital), xD (extreme digital), MMC (multi-media card), or memory stick, etc. may be further included. The external memory 234 may be functionally and/or physically connected to the electronic device 201 through various interfaces.

For example, the sensor module 240 may measure a physical quantity or sense the operating state of the electronic device 201 and convert the measured or sensed information into an electrical signal. The sensor module 240 includes, for example, a gesture sensor 240A, a gyro sensor 240B, an atmospheric pressure sensor 240C, a magnetic sensor 240D, an acceleration sensor 240E, a grip sensor 240F, and a proximity sensor ( 240G), color sensor (240H) (e.g. RGB (red, green, blue) sensor), biometric sensor (240I), temperature/humidity sensor (240J), illuminance sensor (240K), or UV (ultra violet) sensor ( 240M) may include at least one. Additionally or alternatively, the sensor module 240 may include, for example, an olfactory sensor (E-nose sensor), an EMG sensor (electromyography sensor), an EEG sensor (electroencephalogram sensor), an ECG sensor (electrocardiogram sensor) , may include an IR (infrared) sensor, an iris sensor, and/or a fingerprint sensor. The sensor module 240 may further include a control circuit for controlling at least one sensor included therein. In some embodiments, the electronic device 201 further includes a processor configured to control the sensor module 240, either as part of the processor 210 or separately, so that the sensor module 240 is configured to control the sensor module 240 while the processor 210 is in a sleep state. The module 240 can be controlled.

The input device 250 may include, for example, a touch panel 252, a (digital) pen sensor 254, a key 256, or an ultrasonic input device 258. It can be included. The touch panel 252 may use at least one of, for example, a capacitive type, a resistive type, an infrared type, or an ultrasonic type. Additionally, the touch panel 252 may further include a control circuit. The touch panel 252 may further include a tactile layer to provide a tactile response to the user.

The (digital) pen sensor 254 may be, for example, part of a touch panel or may include a separate recognition sheet. Keys 256 may include, for example, physical buttons, optical keys, or key pads. The ultrasonic input device 258 can detect ultrasonic waves generated by the input tool through the microphone 288 and check data corresponding to the detected ultrasonic waves.

Display 260 may include a panel 262, a holographic device 264, or a projector 266.

The panel 262 may include the same or similar configuration as the display 160 of FIG. 1 . The panel 262 may be implemented as flexible, transparent, or wearable, for example. The panel 262 may be composed of the touch panel 252 and one module.

According to one embodiment, panel 262 may include at least one sensor. For example, panel 262 may include a pressure sensor (or force sensor (interchangeably used hereinafter)). The pressure sensor may be a sensor capable of measuring the intensity of pressure in response to a user's touch. The pressure sensor may be implemented integrally with the touch panel 252, or may be implemented as one or more sensors separately from the touch panel 252.

The hologram device 264 can display a three-dimensional image in the air using light interference. The projector 266 can display an image by projecting light onto the screen. The screen may be located, for example, inside or outside the electronic device 201. According to one embodiment, the display 260 may further include a control circuit for controlling the panel 262, the hologram device 264, or the projector 266.

The interface 270 is, for example, a high-definition multimedia interface (HDMI) 272, a universal serial bus (USB) 274, an optical interface 276, or a D-sub (D-subminiature). )(278). Interface 270 may be included in, for example, communication interface 170 shown in FIG. 1 . Additionally or alternatively, the interface 270 may be, for example, a mobile high-definition link (MHL) interface, a secure digital (SD) card/multi-media card (MMC) interface, or an infrared data association (IrDA) standard interface. may include.

The audio module 280 can, for example, convert sound and electrical signals in two directions. At least some components of the audio module 280 may be included in, for example, the input/output interface 150 shown in FIG. 1 . The audio module 280 may process sound information input or output through, for example, a speaker 282, a receiver 284, an earphone 286, or a microphone 288.

The camera module 291 is, for example, a device that can capture still images and moving images. According to one embodiment, the camera module 291 includes one or more image sensors (e.g., a front sensor or a rear sensor), a lens, and an image signal processor (ISP). , or a flash (e.g., LED or xenon lamp, etc.). That is, the camera module 291 may include image sensors with different characteristics. The different characteristics are requirements that determine the characteristics of the captured image, and may be characteristics related to the type of image (black and white, color, etc.), resolution, angle of view, etc.

The power management module 295 may manage the power of the electronic device 201, for example. According to one embodiment, the power management module 295 may include a power management integrated circuit (PMIC), a charger integrated circuit (IC), a battery 296, or a fuel gauge. PMIC may have wired and/or wireless charging methods. The wireless charging method includes, for example, a magnetic resonance method, a magnetic induction method, or an electromagnetic wave method, and may further include an additional circuit for wireless charging, for example, a coil loop, a resonance circuit, or a rectifier. there is. The battery gauge may measure, for example, the remaining amount of the battery 296, voltage, current, or temperature during charging. Battery 296 may include, for example, a rechargeable battery and/or a solar battery.

The indicator 297 may display a specific state of the electronic device 201 or a part thereof (e.g., the processor 210), for example, a booting state, a message state, or a charging state. The motor 298 can convert electrical signals into mechanical vibration and generate vibration or haptic effects. Although not shown, the electronic device 201 may include a processing unit (eg, GPU) to support mobile TV. A processing device for supporting mobile TV may process media data according to standards such as, for example, digital multimedia broadcasting (DMB), digital video broadcasting (DVB), or mediaFloTM.

Each of the components described in this disclosure may be composed of one or more components, and the names of the components may vary depending on the type of electronic device. In various embodiments, an electronic device may be configured to include at least one of the components described in this disclosure, some components may be omitted, or other additional components may be included. Additionally, by combining some of the components of an electronic device according to various embodiments to form a single entity, the functions of the corresponding components before being combined can be performed in the same manner.

Figure 3 is a diagram illustrating the configuration of a program module according to various embodiments proposed in the present disclosure.

Referring to FIG. 3, the program module 310 (e.g., program 180) is an operating system (OS) that controls resources related to an electronic device and/or various applications (e.g., running on the operating system). It may include an application program (187). The operating system may be, for example, android, iOS, windows, symbian, tizen, or bada.

The program module 310 may include a kernel 320, middleware 330, an application programming interface (API) 360, and/or an application 370. At least a portion of the program module 310 may be preloaded into the electronic device or downloaded from an external electronic device (eg, the electronic devices 102 and 104, the server 106, etc.).

As examples of each of the kernel 320, middleware 330, API 360, and application 370 included in the program module 310, the kernel 181 and middleware ( 183), API (185), and application (187).

The kernel 320 may include, for example, a system resource manager 321 and/or a device driver 323. The system resource manager 321 may control, allocate, or retrieve system resources. According to one embodiment, the system resource manager 321 may include a process management unit, a memory management unit, or a file system management unit. The device driver 323 may include, for example, a display driver, camera driver, Bluetooth driver, shared memory driver, USB driver, keypad driver, WiFi driver, audio driver, or IPC (inter-process communication) driver, etc. there is.

For example, the middleware 330 provides functions commonly required by the application 370 or provides various functions through the API 360 so that the application 370 can efficiently use limited system resources inside the electronic device. Functions may be provided as an application 370. According to one embodiment, the middleware 330 includes a runtime library 335, an application manager 341, a window manager 342, a multimedia manager 343, and a resource manager ( resource manager (344), power manager (345), database manager (346), package manager (347), connectivity manager (348), notification manager ( It may include at least one of a notification manager (349), a location manager (350), a graphic manager (351), or a security manager (352).

The runtime library 335 may include, for example, a library module used by a compiler to add new functions through a programming language while the application 370 is running. The runtime library 335 may perform functions such as input/output management, memory management, or arithmetic functions.

The application manager 341 may, for example, manage the life cycle of at least one application among the applications 370 . The window manager 342 can manage GUI resources used on the screen. The multimedia manager 343 can identify the format required to play various media files and encode or decode the media file using a codec suitable for the format. The resource manager 344 may manage resources such as source code, memory, or storage space of at least one of the applications 370.

For example, the power manager 345 may operate in conjunction with a basic input/output system (BIOS) to manage batteries or power and provide power information necessary for the operation of electronic devices. The database manager 346 may create, search, or change a database to be used by at least one of the applications 370. The package manager 347 can manage the installation or update of applications distributed in the form of package files.

The connection manager 348 can manage wireless connections, such as WiFi or Bluetooth. The notification manager 349 can display or notify events such as arrival messages, appointments, and proximity notifications in a way that does not disturb the user. The location manager 350 can manage location information of the electronic device. The graphics manager 351 can manage graphic effects or user interfaces related thereto to be provided to the user. The security manager 352 can provide various security functions necessary for system security or user authentication. According to one embodiment, when the electronic device includes a phone function, the middleware 330 may further include a telephony manager for managing the voice or video call function of the electronic device.

Middleware 330 may include a middleware module that forms a combination of various functions of the above-described components. The middleware 330 may provide specialized modules for each type of operating system to provide differentiated functions. Additionally, the middleware 330 may dynamically delete some existing components or add new components.

The API 360 is, for example, a set of API programming functions and may be provided in different configurations depending on the operating system. For example, in the case of Android or iOS, one API set can be provided for each platform, and in the case of Tizen, two or more API sets can be provided for each platform.

The application 370 includes, for example, home 371, dialer 372, SMS/MMS (373), instant message (IM) 374, browser 375, camera 376, and alarm 377. , one or more applications that can perform functions such as contact 378, voice dial 379, email 380, calendar 381, media player 382, album 383, or clock 384. It can be included. The application 370 is an application that can perform functions such as health care (e.g., measuring exercise amount or blood sugar level) or providing environmental information (e.g., providing atmospheric pressure, humidity, or temperature information). It may also include .

According to one embodiment, the application 370 is an application that supports information exchange between an electronic device (e.g., the electronic device 101) and an external electronic device (hereinafter referred to as an “information exchange application” for convenience of explanation). It can be included. The information exchange application may include, for example, a notification relay application for delivering specific information to an external electronic device, or a device management application for managing an external electronic device.

For example, the notification delivery application may include a function to deliver notification information generated from another application of the electronic device (e.g., SMS/MMS application, email application, health management application, or environmental information application, etc.) to an external electronic device. there is. Additionally, the notification delivery application may, for example, receive notification information from an external electronic device and provide it to the user.

The device management application may, for example, control at least one function of an external electronic device in communication with the electronic device, such as turning on/off the external electronic device itself (or some component part) or brightness of a display (or , resolution) adjustment), applications running on an external electronic device, or services provided by an external electronic device (e.g., call service or message service, etc.) can be managed (e.g., installation, deletion, or update).

According to one embodiment, the application 370 may include an application designated according to attribute information of an external electronic device (eg, a health care application for a mobile medical device, etc.). According to one embodiment, the application 370 may include an application received from an external electronic device. According to one embodiment, the application 370 may include a preloaded application or a third party application that can be downloaded from a server. The names of components of the program module 310 according to the illustrated embodiment may vary depending on the type of operating system.

According to various embodiments, at least a portion of the program module 310 may be implemented as software, firmware, hardware, or a combination of at least two or more thereof. At least a portion of the program module 310 may be implemented (eg, executed) by, for example, a processor (eg, the processor 210). At least a portion of the program module 310 may include, for example, modules, programs, routines, sets of instructions, or processes for performing one or more functions.

The term “module” used in this disclosure may mean, for example, a unit including one or a combination of two or more of hardware, software, or firmware. “Module” may be used interchangeably with terms such as unit, logic, logic block, component, or circuit, for example. A “module” may be the smallest unit of integrated parts or a part thereof. “Module” may be the minimum unit or part of one or more functions. A “module” may be implemented mechanically or electronically. For example, a “module” is an application-specific integrated circuit (ASIC) chip, field-programmable gate arrays (FPGAs), or programmable-logic device, known or to be developed, that performs certain operations. It can contain at least one.

At least a portion of the device (e.g., modules or functions thereof) or method (e.g., operations) according to various embodiments may be stored in, for example, a computer-readable storage media in the form of a program module. It can be implemented with instructions stored in . When the instruction is executed by a processor (eg, processor 120), the one or more processors may perform the function corresponding to the instruction. The computer-readable storage medium may be, for example, memory 130.

Computer-readable recording media include hard disks, floppy disks, magnetic media (e.g. magnetic tape), optical media (e.g. CD-ROM (compact disc read only memory), DVD (e.g. digital versatile disc, magneto-optical media (e.g., floptical disk), hardware devices (e.g., read only memory (ROM), random access memory (RAM), or flash memory, etc. ), etc. In addition, program instructions may include not only machine language code such as that created by a compiler, but also high-level language code that can be executed by a computer using an interpreter, etc. The above-described hardware devices include a variety of It may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

A module or program module according to various embodiments may include at least one of the above-described components, some of them may be omitted, or may further include other additional components. Operations performed by modules, program modules, or other components according to various embodiments may be executed sequentially, in parallel, iteratively, or in a heuristic manner. Additionally, some operations may be executed in a different order, omitted, or other operations may be added. Additionally, the embodiments disclosed in this disclosure are presented for explanation and understanding of technical content, and do not limit the scope of the technology described in this disclosure. Accordingly, the scope of the present disclosure should be construed as including all changes or various other embodiments based on the technical idea of the present disclosure.

FIG. 4 is a diagram illustrating a configuration for image compression in an electronic device according to various embodiments proposed in the present disclosure.

Referring to FIG. 4, the electronic device may include a processor 410, a camera module 420, and a memory 430 for image compression. The camera module 420 may include a sensor 422 and an ISP 424.

The processor 410 may receive an image to be compressed or recompressed from the camera module 420 or the memory 430. The processor 410 may receive an image to be compressed or recompressed in a preset unit (for example, GOP). The processor 410 may receive an image to be compressed or recompressed from an external electronic device or external storage medium based on a streaming service.

The processor 410 may perform real-time compression on the target image provided from the camera module 420 based at least on metadata. The processor 410 may restore the target image provided from the memory 430 and then recompress the restored target image based on at least metadata. The metadata may be provided from the camera module 420, memory 430, external memory, or an external device.

According to one embodiment, the processor 410 obtains metadata to be considered for compressing the target image from the sensor 422 (for example, an image sensor) or the ISP 424 constituting the camera module 420. It may also be generated using attribute information of the target image obtained from the camera module 420.

The processor 410 may perform real-time compression on the target image based on a constant bitrate (CBR) method or a variable bitrate (VBR) method.

The CBR method is a method that performs compression using the same bit rate from the beginning to the end of the bit stream to be compressed. Generally, the CBR method is applied when compressing video, and in this case, compression can be performed in a short time, but when compressing video with a lot of movement, image quality may deteriorate.

The VBR method largely supports three types of compression. For example, it can be divided into average/maximum/min. In general, compression can be performed using an average bit rate, if there is a lot of motion, compression can be performed using the maximum bit rate, and if there is little motion, compression can be performed using the minimum bit rate. In other words, the VBR method performs rough compression (hereinafter referred to as 'simple VBR compression') on the entire target image, and then performs additional detailed compression (hereinafter referred to as 'additional VBR compression') once again. there is.

Therefore, compared to the CBR method, the VBR method can generate compressed images with relatively small capacity and good image quality. However, not only is the compression operation of the VBR method relatively more complicated than the CBR method, but the time required for the compression operation may increase.

According to one embodiment, the processor 410 may perform recompression on a compressed image that has been compressed according to the CBR method or simple VBR compressed according to the VBR method, based at least on metadata. If the target compressed image for recompression is generated by simple VBR compression, the processor 410 may perform recompression by additional VBR compression. In this case, even if compression is performed using the VBR method, additional VBR compression can be performed in a standby state, thereby solving the problem of increased consumption time due to application of the VBR method.

According to the above, the processor 410 can be set to compress or recompress the image by referring to real-time or pre-generated metadata without the need to analyze and recompress the compressed image.

The processor 410 may generate metadata using attribute information of the acquired target image. The attribute information of the acquired target image includes color, brightness, exposure, histogram, region of interest, sharpness, complexity, texture, and skin tone corresponding to the target image. ), bitrate information, depth, or a combination thereof.

According to one embodiment, in order to increase compression efficiency when compressing real-time video, the processor 410 acquires data from the sensor 422 (image sensor, etc.) or the ISP 424 constituting the camera module 420 before compression. Metadata or metadata obtained from the memory 430 or an external memory or external device can be used when compressing an image.

According to one embodiment, in order to efficiently create a compressed video stream, the processor 410 may recompress the target image while maintaining the user's subjective image quality or objective image quality (PSNR, SSIM) above a certain level. In this case, video compression efficiency can be improved by using metadata (rate control, mode decision, texture information, etc.) obtained during image signal processing performed before video compression.

The processor 410 may perform operations of acquiring a target image and generating metadata in parallel. This is to enable metadata to be created in real time.

The processor 410 may check status information about the electronic device before compressing the target image and determine whether the checked status information satisfies a specified condition. This is to determine whether to perform additional VBR compression using metadata on compressed images obtained by compression using the CBR method or simple VBR compression. The status information can be triggered for additional compression.

For example, the processor 410 may perform an operation of compressing a target image only when status information about the electronic device satisfies specified conditions. The status information may include remaining memory capacity, image signal processing performance, data amount of image signals that can be processed, charging status of the electronic device, external electronic device connected to the electronic device, or a combination thereof.

For example, the processor 410 determines whether the remaining memory amount falls within a specified range based on the remaining memory amount information included in the status information, and compresses the target image only when the remaining memory amount falls within the specified range. there is.

The processor 410 recognizes a server (cloud server, etc.) connected through a network, and provides uncompressed video or compressed video with simple compression (CBR compression or simple VBR compression) to the recognized server. You can. In this case, recompression of the compressed video that has simply been compressed can be performed by the server. The processor 410 can receive the recompressed compressed video from the server through the network.

The processor 410 monitors the status of the memory 430 and triggers a recompression operation when it is determined through the monitoring that the storage area is insufficient.

For example, as at least part of the operation of compressing the target image, the processor 410 may configure the resolution, image bit rate, image bit depth, image frame rate, image color space, and audio bit for at least a portion of the target image. It may be set to change at least one of the rate, voice bit depth, and voice sample rate, and determine the compression rate for at least some sections based on at least information according to the change.

The processor 410 may divide the target image into a first section and a second section and apply different compression rates to the divided first section and the second section.

To this end, the processor 410 can be set to compress the first section using a first method (scheme), and can be set to compress the second section using a second method. In this case, the processor 410 may generate metadata using first metadata corresponding to the first section and second metadata corresponding to the second section. The processor 410 may generate first metadata with reference to a first method and generate second metadata with reference to a second method.

As at least part of the operation of compressing the target image, the processor 410 may determine a first method or a second method so that the compression result of the first section or the second section satisfies a preset image quality. This will ensure that the quality of the compressed video corresponding to the first section and the second section is maintained at a similar level.

The processor 410 identifies a section containing a region of interest among at least some sections of the target image, and compresses the target image using a bit rate at a level specified for the section containing the identified region of interest. You can decide how to do it.

Memory 430 may include volatile and/or non-volatile memory. The memory 430 may store metadata and/or compressed or uncompressed image data in an area designated by the processor 410. For example, the memory 430 may store metadata as at least part of the target image. At this time, all or part of the target image may be stored in the memory 430, and all or part of the metadata may also be stored in the memory 430.

The memory 430 may read images and/or metadata stored in a designated area in response to a request from the processor 410 and provide them to the processor 410. The memory 430 may record images recompressed by the processor 410 instead of previously recorded images. The metadata may be recorded in the memory 430 as part of the target image.

FIG. 5 is a diagram illustrating control flow performed by an electronic device according to various embodiments proposed in the present disclosure.

Referring to FIG. 5, the electronic device can receive a source bit stream in step 510. The source bit stream may indicate a bit stream corresponding to the target image for compression or recompression.

According to one embodiment, the source bit stream may be captured by an internal or external camera and provided through an image sensor, selected from bit streams recorded in an internal or external memory, or provided through a network based on a streaming service. The source bit stream may be provided in preset units (eg, GOP) for compression or recompression.

In step 520, the electronic device may compress or recompress the source bit stream based on at least metadata. The compression is performed in real time and may correspond to CBR compression or simple VBR compression.

According to one embodiment, an electronic device may perform real-time compression on an image captured by a camera by considering metadata. For example, metadata may be generated during image signal processing for a source bit stream provided from a camera or may be generated by a sensor (image sensor). The generated metadata may be processed for efficient use in real-time compression. The processing may utilize metadata to exclude overlapping operations among operations performed during image signal processing and operations performed during compression. That is, if the electronic device can obtain information through metadata among the information obtained through the compression operation, the corresponding operation can be omitted. As a result, the amount of computation during compression can be reduced.

The electronic device may perform an operation of acquiring a target image and an operation of generating metadata in parallel. This is to enable metadata to be created in real time.

Additionally, the electronic device can maintain the image quality of the compressed video by controlling the bit rate based on the VBR method during real-time compression.

To be more specific, the electronic device may receive metadata to be considered for compressing the source bit stream from a sensor (for example, an image sensor) constituting the camera module, or may generate it using attribute information of the source bit stream.

The attribute information of the source bit stream includes color, brightness, exposure, histogram, region of interest, sharpness, complexity, texture, and skin tone corresponding to the source bit stream. ), bitrate information, depth, or a combination thereof.

The electronic device can perform real-time compression on the source bit stream based on the CBR method or VBR method. At this time, real-time compression using the VBR method may correspond to simple VBR compression.

According to one embodiment, the electronic device may perform recompression on compressed video provided through a memory or streaming service by considering metadata. At this time, the metadata may be created during compression and stored in advance. For the recompression, the electronic device can obtain a restored image by performing restoration on the source bit stream.

The electronic device may evaluate the image quality level of the reconstructed image and detect a high-quality encoding area in the reconstructed image based on the result. The electronic device can detect a high-definition encoding area with image quality higher than a preset standard through image quality measurement on the restored image. For example, the electronic device can measure the image quality of the reconstructed image based on the PSNR method or the SSIM method and detect a high-quality encoding area based on this.

The electronic device may set new compression conditions for a target area to be recompressed in the restored image based on the image quality level of the reconstructed image for which the evaluation was made. For example, the electronic device may determine a quantization parameter (QP) as a compression condition. Here, QP represents a value that can be set as an option to adjust the degree of quantization. For example, by adjusting the QP, the quality of the image can be adjusted.

The electronic device may perform recompression on frame data (restored video) in a GOP unit, that is, a source bit stream with a size corresponding to a predetermined unit, based on the newly determined compression conditions and metadata. The electronic device evaluates image quality reception until recompression of the entire frame data in the GOP unit is completed, sets new compression conditions for the area to be recompressed based on this, and performs recompression based on this. will repeat

When recompression of frame data in GOP units is completed, the electronic device determines whether there is a source bit stream in GOP units to be next recompressed. If there is a source bit stream to be recompressed, the electronic device may perform recompression of the source bit stream by repeating the above-described operation.

According to the above, the electronic device can be set to compress or recompress the video by referring to real-time or pre-generated metadata without the need to analyze and recompress the compressed video.

To summarize the above, in the Quality Assessment stage, the electronic device evaluates the image quality level of the video through the analysis results of the restored image and previously restored images in GOP units. At this time, a method of inferring objective image quality, such as PSNR, can be used. The evaluation method may differ depending on implementation. If the evaluated image quality is higher than the standard image quality, the amount of data due to compression can be reduced by adjusting the quantization parameter (QP) on a slice or block basis to remove excessive information from the user's perspective.

According to one embodiment, in order to efficiently create a compressed video stream, the electronic device may perform recompression of the target image while maintaining the user's subjective image quality or objective image quality (PSNR, SSIM) above a certain level. In this case, video compression efficiency can be improved by using metadata (rate control, mode decision, texture information, etc.) obtained during image signal processing performed before video compression.

The electronic device may check state information about the electronic device before compressing the target image and determine whether the confirmed state information satisfies a specified condition. This is to determine whether to perform additional VBR compression using metadata on compressed images obtained by compression using the CBR method or simple VBR compression. The status information can be triggered for additional compression.

For example, the electronic device may perform an operation of compressing a target image only when state information about the electronic device satisfies a specified condition. The status information may include remaining memory capacity, image signal processing performance, data amount of image signals that can be processed, charging status of the electronic device, external electronic device connected to the electronic device, or a combination thereof.

For example, the electronic device may determine whether the remaining memory amount falls within a specified range based on the remaining memory amount information included in the status information, and compress the target image only when the remaining memory amount falls within the specified range.

The electronic device recognizes a server (cloud server, etc.) connected through a network, and provides uncompressed video or compressed video with simple compression (CBR compression or simple VBR compression) to the recognized server. . In this case, recompression of the compressed video that has simply been compressed can be performed by the server. The electronic device can receive the recompressed compressed video from the server through the network.

The electronic device monitors the memory status and triggers a recompression operation when it is determined through the monitoring that the storage area is insufficient.

For example, as at least part of the operation of compressing the target image, the electronic device may include resolution, image bit rate, image bit depth, image frame rate, image color space, and audio bit rate for at least a portion of the target image. It may be set to change at least one of voice bit depth and voice sample rate, and determine a compression rate for at least some sections based on at least information according to the change.

The electronic device may divide the target image into a first section and a second section and apply different compression rates to the divided first section and second section.

To this end, the electronic device can be set to compress the first section using a first method (scheme), and the second section can be set to compress using the second method. In this case, the electronic device may generate metadata using first metadata corresponding to the first section and second metadata corresponding to the second section. The electronic device may generate first metadata with reference to a first method and generate second metadata with reference to a second method.

As at least part of the operation of compressing the target image, the electronic device may determine a first method or a second method so that the compression result of the first section or the second section satisfies a preset image quality. This will ensure that the quality of the compressed video corresponding to the first section and the second section is maintained at a similar level.

The electronic device identifies a section containing a region of interest among at least some sections of the target image, and compresses the target image using a bit rate at a level specified for the section containing the identified region of interest. can be decided.

The electronic device may store the compressed bit stream generated by compression in step 530. For example, the electronic device may store the compressed bit stream in at least one of internal memory, external memory, and an external device. The stored compressed bit stream may be restored by the electronic device or provided upon request from an external device.

FIG. 6 is a diagram illustrating a control flow in which an electronic device performs compression considering metadata, according to various embodiments proposed in the present disclosure.

Referring to FIG. 6, in step 610, the electronic device inputs an image to be compressed. For example, the electronic device may generate an image to be compressed through a camera module, or may generate an image to be compressed by restoring a compressed image provided through a memory or streaming service. The electronic device can perform initial compression when the image to be compressed is generated through a camera module, and can perform recompression when the image to be compressed is generated through restoration.

In step 620, the electronic device can measure the partial or entire image quality of the image to be compressed. For example, the electronic device may evaluate the image quality level of the restored or generated image to be compressed and detect a high-definition encoding area in the image to be compressed based on the result. That is, the electronic device can detect a high-definition encoding area with image quality higher than a preset standard through image quality measurement for the image to be compressed. For example, the electronic device may measure the image quality of the video to be compressed based on the PSNR method or the SSIM method and detect a high-definition encoding area based on this.

In step 614, the electronic device may determine whether the partial or total image quality of the compressed image measured satisfies preset conditions. For example, the electronic device may preset a threshold value corresponding to the reference image quality for determining whether the image to be compressed requires recompression as a decision condition. However, it goes without saying that the preset conditions can be used or set as various conditions for judgment depending on the user's request.

The electronic device may proceed to step 616 when the image quality measured from the image to be compressed satisfies a preset condition, and may proceed to step 618 when the image quality measured from the image to be compressed does not satisfy the preset condition.

In step 616, the electronic device may compress part or all of the image to be compressed according to the first compression condition. In step 618, the electronic device may compress part or all of the image to be compressed according to the second compression condition.

For example, the electronic device may preset the first compression condition in consideration of the compression rate to be obtained through recompression, and may preset the second compression condition in consideration of the compression rate to be obtained through initial compression. . For example, the initial compression may be to compress the image generated by the camera module for the first time, and the recompression may be to recompress the restored image provided through a memory or streaming service. In this case, it would be desirable to set the first compression condition to obtain a relatively high compression ratio compared to the second compression condition. In addition, the first compression condition and the second compression condition may include metadata to be referenced for compressing the corresponding compression target image. Alternatively, the meta data may be provided through a separate procedure. Since the method by which the electronic device considers metadata for compression has already been sufficiently described, the description will be omitted.

According to one embodiment, the first and second compression conditions may determine QP. The QP represents a value that can be set as an option to adjust the degree of quantization. For example, by adjusting the QP, the quality of the image can be adjusted.

The electronic device may record an image compressed by a first compression condition or an image compressed by a second compression condition to an internal or external memory or transmit it to an external device.

FIG. 7 is a diagram illustrating a scenario in which an electronic device performs recompression according to various embodiments proposed in the present disclosure.

According to FIG. 7, it can be confirmed that the electronic device processes recompression for each predetermined unit section (GOP). In this way, recompression is performed for each predetermined unit section (GOP) in order to cope with a situation in which compression is suddenly interrupted. The electronic device may apply different compression to the closed GOP (closed-GOP) section and the open GOP (open-GOP) section that constitutes the GOP.

For example, the electronic device can perform recompression including an infrastructure frame (infra frame, I-frame, or I-slice) in the closed-GOP section. In this case, I-frame (or I-slice), predicted frame (previous frame or predicted frame, P-frame or P-slice) and bidirectional frame (bidirectional frame, B-frame or B-slice) in closed-GOP interval. You can see that 10Mb information consisting of ) is compressed to 5Mb. And in the open-GOP section, recompression can be performed on the P-frame (or P-slice) and B-frame (or B-slice) while maintaining the I-frame (or I-slice).

In general, GOP is a basic unit for compressing images and can refer to a collection of frames from one key frame to the next key frame. There are two options for the GOP, one is a scene detection technology and the other is closed-GOP. For example, in the closed-GOP section, compression can be performed by excluding the B frame before the next I frame in the GOP configuration.

According to the above, the electronic device can perform compression using closed-GOP or clean random access (CRA) when initially compressing the video to facilitate video recompression. The CRA can be set to exclude specific frames when compressing. Frames (I-frame, B-frame, P-frame) designated by the CRA can be referenced during compression. That is, at least one frame that exists between a specific frame and a frame designated as CRA will be excluded during compression.

According to various embodiments proposed in this disclosure, the reason the compression procedure is processed in GOP units is to enable independent processing without affecting the video stream when the video processing process is suddenly stopped. And as previously stated, when recompressing in open-GOP, the electronic device can only perform recompression for P-frames and B-frames (P/B-slice).

According to various embodiments proposed in this disclosure, methods of utilizing metadata when compressing images in an electronic device can be provided in various ways depending on how the metadata is configured.

FIG. 8 is a diagram illustrating an example of an electronic device compressing or recompressing an image based on metadata, according to various embodiments proposed in the present disclosure.

Referring to FIG. 8, when an electronic device captures a video, the ISP can generate various metadata. Among them, images can be efficiently compressed or recompressed by using histogram, sharpness, exposure information, skin tone information, etc.

FIG. 9 is a diagram illustrating a scenario in which an electronic device effectively performs a sample adaptive offset (SAO) process of high efficiency video coding (HEVC) using histogram information, according to various embodiments proposed in this disclosure.

Referring to FIG. 9, HEVC's SAO consists of a band offset (BO) and an edge offset (EO). The BO and EO define an adaptive sample offset calculation method. The adaptive sample offset corresponds to an in-loop filter technique proposed to calculate an offset value appropriate for each region and use this to reduce quantization error. The BO and the EO define the type of adaptive sample offset. The BO can be an efficient offset calculation method to use when a difference in value from the original pixel occurs in a specific brightness band. The EO can be a more efficient offset calculation method in areas containing boundary information.

Therefore, the electronic device may need to select the SAO type as BO or EO in order to increase the compression efficiency of SAO. For this purpose, the electronic device can use histogram information of the image.

Since the electronic device has already obtained the histogram information from the ISP, there is no need to calculate the histogram information repeatedly for image compression. This ultimately reduces the computational complexity for image compression.

FIG. 10 illustrates an example in which an electronic device utilizes metadata when performing perception-based video compression (perceptual video coding) according to various embodiments proposed in the present disclosure.

Referring to FIG. 10, the human eye has a contrast sensitivity function (CSF) that has the characteristics of a band pass filter depending on spatial/temporal frequency. Therefore, the sensitivity perceived by the human eye is different depending on the brightness and texture of the area in the image. When compressing an image using these characteristics, if more data is allocated to areas that are more sensitive to the human eye and less data is allocated to less sensitive areas, better image quality can be obtained even when compressed at the same compression rate. In particular, perceived image quality can be improved by setting an area that humans perceive more sensitively, such as the face, as a ROI (region on interest) and applying bit rate control (rate-control) for each area.

Meanwhile, although specific embodiments have been described in the detailed description of the present disclosure, various modifications may be made by those skilled in the art without departing from the scope of the various embodiments proposed in the present disclosure. Of course it can be done. Therefore, the scope according to the present disclosure should not be limited to the described embodiments, but should be determined not only by the scope of the patent claims described later, but also by the scope of the patent claims and their equivalents. In addition, these modified implementations should not be understood individually from the technical spirit or perspective of the present disclosure.

Claims (20)

In electronic devices,
A camera including an image sensor;
a processor functionally connected to the camera; and
An image signal processor (ISP) functionally connected to the camera and the processor and configured to generate metadata containing attribute information of the image streams obtained by the image sensor while capturing video containing the image streams. Including,
The processor:
In response to receiving a user request to capture the video, control the camera to capture the video,
Obtain the video along with the metadata corresponding to the video, wherein the metadata is generated by the ISP while capturing the video,
storing the acquired video and the metadata,
Compressing the obtained video in a constant bit rate (CBR) manner using the metadata generated by the ISP while capturing the video,
Replace the stored video with a video compressed using the CBR method, and
If the state information of the electronic device satisfies a predetermined condition, after restoring the video compressed by the CBR method, variable bit rate (VBR) is generated using the metadata generated by the ISP while capturing the video. ) configured to be recompressed in a manner
Electronic devices.
According to claim 1,
The attribute information of the image streams includes color, brightness, exposure, histogram, region of interest, sharpness, complexity, texture, and skin tone corresponding to the image streams. , an electronic device that includes bitrate information, depth, or a combination thereof.
delete delete According to claim 1,
The status information includes the remaining amount of memory of the electronic device, the performance of the ISP, the amount of data processed by the ISP, the charging state of the electronic device, an external electronic device connected to the electronic device, or a combination thereof. .
delete delete The method of claim 1, wherein the processor:
At least as part of the operation of compressing the video, at least one of resolution, video bit rate, video bit depth, video frame rate, video color space, audio bit rate, audio bit depth, or audio sample rate for at least a portion of the video An electronic device further configured to change one, and determine a compression rate for the at least some sections based on the change.
The method of claim 1, wherein the processor:
compressing a first section of the video by a first scheme and compressing a second section of the video by a second method, and
The ISP is,
Generate first metadata corresponding to the first section with reference to the first method,
Generate second metadata corresponding to the second section by referring to the second method, and
further configured to configure the metadata to include the first metadata and the second metadata.
Electronic devices.
delete The method of claim 1, wherein the processor:
Identify a section including a region of interest among at least some sections of the video, and further compress the video using a bit rate at a level specified for the identified section including the region of interest. Consisting of electronic devices.
The method of claim 1, wherein the processor:
An electronic device configured to store the metadata as at least part of the image stream in a memory of the electronic device.
As a method of compressing video in an electronic device,
Method performed by the electronic device of any one of claims 1, 2, 5, 8, 9, 11, and 12
delete delete delete delete delete delete A computer-readable recording medium recording a program for executing an operation performed by the electronic device of any one of claims 1, 2, 5, 8, 9, 11, and 12. .

Images