CN113316017B - Method for detecting resolution and electronic device - Google Patents
Method for detecting resolution and electronic device Download PDFInfo
- Publication number
- CN113316017B CN113316017B CN202010122937.1A CN202010122937A CN113316017B CN 113316017 B CN113316017 B CN 113316017B CN 202010122937 A CN202010122937 A CN 202010122937A CN 113316017 B CN113316017 B CN 113316017B
- Authority
- CN
- China
- Prior art keywords
- frame
- sub
- resolution
- image data
- frames
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 52
- 238000013473 artificial intelligence Methods 0.000 claims abstract description 60
- 238000012545 processing Methods 0.000 claims description 9
- 238000004364 calculation method Methods 0.000 description 17
- 230000015654 memory Effects 0.000 description 14
- 238000013527 convolutional neural network Methods 0.000 description 6
- 238000001514 detection method Methods 0.000 description 6
- 238000013528 artificial neural network Methods 0.000 description 4
- 239000000872 buffer Substances 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000013341 scale-up Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/44008—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N17/00—Diagnosis, testing or measuring for television systems or their details
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Multimedia (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Mathematical Physics (AREA)
- Health & Medical Sciences (AREA)
- Signal Processing (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Medical Informatics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Molecular Biology (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a method for detecting resolution of image data and an electronic device. The method is applied to an electronic device, and a computing circuit of the electronic device executes an artificial intelligence model. The image data comprises a plurality of frames, and each frame comprises a plurality of sub-frames. The artificial intelligence model processes a plurality of pixel data to produce an intermediate resolution corresponding to the pixel data. The method comprises the following steps: (A) Generating a target sub-frame, wherein the number of pixels of the target sub-frame is smaller than that of any image frame; (B) Inputting the target sub-frame into the artificial intelligence model to obtain the intermediate resolution; (C) storing the intermediate resolution; (D) Repeating steps (a) through (C) to obtain a plurality of intermediate resolutions; and (E) determining the resolution of the image data based on the intermediate resolutions.
Description
Technical Field
The present invention relates to image processing, and more particularly to detecting resolution of image data (video material).
Background
Some current electronic devices (e.g. televisions, screens, multimedia players, set-top boxes, mobile phones, computers, notebook computers, tablet computers, etc.) can play or process Ultra-high-definition (UHD) image data, but these high-resolution image data may be amplified (scale up) from lower-resolution image data, in other words, these high-resolution image data may have low original resolution.
To improve image quality, the electronic device typically performs different image quality processes (picture quality function) on the image data based on the original resolution (original resolution) of the image data. The conventional method detects the resolution of the image data by monitoring the update frequency of the pixel values of the image data. However, this approach requires specially designed hardware and algorithms, resulting in a lack of flexibility in implementation.
With the vigorous development of artificial intelligence (Artificial Intelligence, AI), more and more image processing related machine learning (machine learning) algorithms or deep learning (deep learning) algorithms have been proposed. Neural Networks (NN) or convolutional Neural networks (Convolutional Neural Network, CNN) are artificial intelligence models that are widely used for image processing. Currently, artificial intelligence models are trained to detect the original resolution of image data (please refer to https:// github. Com/sub/koniq), however, when the nominal resolution of image data is higher, the more image data the artificial intelligence model needs to process, the more time-consuming the process of detecting resolution. This can cause hardware (e.g., central processing units, graphics processors (Graphics Processing Unit, GPUs)) executing the artificial intelligence model to be occupied by programs of detected resolution for a long period of time, reducing the performance of the electronic device.
Disclosure of Invention
In view of the shortcomings of the prior art, an object of the present invention is to provide a method and an electronic device for detecting resolution.
The invention discloses a method for detecting resolution of image data. The method is applied to an electronic device, and a computing circuit of the electronic device executes an artificial intelligence model. The image data comprises a plurality of frames, and each frame comprises a plurality of sub-frames. The artificial intelligence model processes a plurality of pixel data to produce an intermediate resolution corresponding to the pixel data. The method comprises the following steps: (A) Generating a target sub-frame, wherein the number of pixels of the target sub-frame is smaller than the number of pixels of any one of the image frames; (B) Inputting the target sub-frame into the artificial intelligence model to obtain the intermediate resolution; (C) storing the intermediate resolution; (D) Repeating steps (a) through (C) to obtain a plurality of intermediate resolutions; and (E) determining the resolution of the image data based on the intermediate resolutions.
The invention also discloses an electronic device for detecting the resolution of the image data. The image data comprises a plurality of frames, and each frame comprises a plurality of sub-frames. The electronic device comprises a storage circuit and a calculation circuit. The storage circuit stores a plurality of program instructions or program codes. A computing circuit is coupled to the storage circuit and executes the program instructions or program code to execute the artificial intelligence model. The artificial intelligence model processes a plurality of pixel data to produce an intermediate resolution corresponding to the pixel data. The computing circuitry also executes the program instructions or program code to perform the steps of: (A) Generating a target sub-frame, wherein the number of pixels of the target sub-frame is smaller than the number of pixels of any one of the image frames; (B) Inputting the target sub-frame into the artificial intelligence model to obtain the intermediate resolution; (C) storing the intermediate resolution; (D) Repeating steps (a) through (C) to obtain a plurality of intermediate resolutions; and (E) determining the resolution of the image data based on the intermediate resolutions.
The invention also discloses a method for detecting the resolution of the image data. The method is applied to an electronic device comprising a computing circuit and a storage circuit. The storage circuit stores a plurality of program instructions or program code that the computing circuit executes to execute a first artificial intelligence model that includes a plurality of sub-models. The method comprises the following steps: (A) obtaining a frame of the image data; (B) Inputting the frame into a target sub-model of the sub-models to obtain a first intermediate result; (C) storing the first intermediate result; (D) Executing a second artificial intelligence model using the computing circuit; (E) Taking the next sub-model of the target sub-model as the target sub-model; (F) Inputting the first intermediate result into the target sub-model to obtain a second intermediate result or the resolution of the image data; and (G) when the second intermediate result is obtained in the step (F) instead of the resolution of the image data, taking the second intermediate result as the first intermediate result, and repeating the steps (C) to (F) until the resolution of the image data is obtained in the step (F).
Compared with the conventional technology, the method for detecting resolution and the electronic device of the present invention do not occupy the software and hardware resources of the electronic device for a long time, so that the performance of the electronic device can be increased (e.g. the bandwidth requirement of the storage circuit or the memory is reduced, and/or the calculation amount is reduced).
The features, implementation and effects of the present invention will now be described in detail with reference to examples of the invention with reference to the accompanying drawings.
Drawings
FIG. 1 illustrates some of the components of an electronic device of the present invention;
FIG. 2 is a flow chart of a method for detecting resolution of image data according to an embodiment of the present invention;
fig. 3A and 3B each show one frame composed of a plurality of sub-frames;
FIG. 4 is a flowchart illustrating a method for detecting resolution of image data according to another embodiment of the present invention;
FIG. 5 shows a picture frame, sub-frame and memory corresponding to the embodiment of FIG. 4;
FIG. 6 illustrates a frame and sub-frame corresponding to another example of the flow of FIG. 4;
FIG. 7 is a flowchart illustrating a method for detecting resolution of image data according to another embodiment of the present invention;
FIG. 8 shows a picture frame, sub-frame and memory corresponding to the embodiment of FIG. 7;
FIG. 9 is a flowchart illustrating a method for detecting resolution of image data according to another embodiment of the present invention;
FIG. 10 illustrates an artificial intelligence model including a plurality of sub-models; and
fig. 11 shows a sub-step of step S930 (corresponding to the embodiment of fig. 10).
Detailed Description
Technical terms used in the following description refer to terms commonly used in the art, and as used in the specification, some terms are described or defined, and the explanation of the some terms is based on the description or the definition of the specification.
The present disclosure includes methods and electronic devices for detecting resolution. Since some of the components included in the electronic device of the present invention may be known components alone, the details of the known components will be omitted from the following description without affecting the full disclosure and operability of the device. Furthermore, part or all of the flow of the method of detecting resolution of the present invention may be in the form of software and/or firmware and may be performed by the electronic device of the present invention or an equivalent thereof, and the following description of the method invention will focus on the contents of the steps rather than the hardware without affecting the full disclosure and operability of the method invention.
Fig. 1 shows part of the components of the electronic device of the invention. The electronic device 100 includes a computing circuit 110 and a storage circuit 120. The storage circuit 120 may store a portion of the image data, where the image data includes a plurality of frames. In some embodiments, the storage circuit 120 includes a plurality of line buffers (line buffers) to store all or a portion of the frames. The storage circuit 120 may be a dynamic random access memory, a static random access memory, equivalent hardware thereof, or a combination of memories. The storage circuit 120 further stores a plurality of program instructions, artificial intelligence model parameters, and other parameters, and the computing circuit 110 uses these program instructions, artificial intelligence model parameters, and other parameters to execute artificial intelligence models (including but not limited to Neural Networks (NN) or convolutional Neural networks (Convolutional Neural Network, CNN)) and to execute methods for detecting the resolution of image data. The electronic device 100 is, for example, but not limited to, a television, a screen, a multimedia player, a set-top box, a mobile phone, a computer, a notebook computer, a tablet computer, etc.
The computing circuit 110 may be a circuit or electronic component with program execution capabilities, such as a central processing unit, a graphics processor, or an application specific integrated circuit (Application Specific Integrated Circuit, ASIC). The application specific integrated circuit is, for example, an artificial intelligence model accelerator.
FIG. 2 is a flow chart of a method for detecting resolution of image data according to an embodiment of the invention. The electronic device 100 receives image data (step S210), wherein the image data includes a plurality of frames, and each frame includes a plurality of sub-frames. The computing circuit 110 then determines a target frame from the plurality of frames (step S220), such as a frame currently being received by the electronic device 100. The calculation circuit 110 then selects a target sub-frame from the same target sub-frame (step S230).
Fig. 3A and 3B each show one frame composed of a plurality of sub-frames. The frame 310 contains 9 sub-frames (SF 1-1, SF1-2, SF1-3, …, SF 1-9), each of the sub-frames being the same size. Frame 320 includes 8 sub-frames (SF 2-1, SF2-2, SF2-3, …, SF 2-8), the 8 sub-frames having a variety of sizes. In some embodiments, the calculation circuit 110 selects the sub-frame SF1-1 or the sub-frame SF2-1 as the target sub-frame at step S230. In the example of fig. 3A, the size of the target frame selected in step S230 is the same each time. In the example of fig. 3B, the size of the target frame selected in step S230 is not necessarily the same each time. The number of sub-frames in fig. 3A and 3B is merely exemplary, and the present embodiment may determine the number of sub-frames according to the practical application.
Returning to fig. 2, after determining the target sub-frame, the computing circuit 110 inputs the target sub-frame into the artificial intelligence model to obtain an intermediate resolution corresponding to the target sub-frame (step S240). In other words, the artificial intelligence model takes a plurality of pixel data of a target sub-frame as input data, and then calculates or determines the resolution of the target sub-frame based on the pixel data. In some embodiments, the number of pixels of the sub-frame is, for example, greater than or equal to 5% of the number of pixels of the frame.
After the artificial intelligence model generates the intermediate resolution, the computing circuit 110 stores the intermediate resolution (step S250), for example, stores the intermediate resolution in a specific storage space or a specific location of the storage circuit 120. After step S250 is completed, the calculation circuit 110 determines whether the number of existing intermediate resolutions is equal to or greater than a target value (step S260). When the number of intermediate resolutions is equal to or greater than the target value, the calculation circuit 110 analyzes the obtained intermediate resolutions to determine the resolution of the image data (step S270). When the number of intermediate resolutions is smaller than the target value, the calculation circuit 110 selects another target sub-frame from the same target frame (i.e., the current target frame) (step S230), and then repeatedly performs steps S230 to S250 until the determination of step S260 is yes.
In some embodiments, the target value is equal to the number of sub-frames of one frame; in other words, the computation circuit 110 needs to input all sub-frames of a frame into the artificial intelligence model to obtain the resolution of the frame. For the example of fig. 3A and 3B, the target value corresponding to frame 310 is 9, and the target value corresponding to frame 320 is 8.
In other embodiments, the target value is less than the number of sub-frames of a frame; in other words, the computing circuit 110 only needs to input a part of sub-frames, but not all of the sub-frames, of one frame into the artificial intelligence model to obtain the resolution of the frame.
In some embodiments, the intermediate resolution obtained in step S240 may be one of several categories (e.g., high, medium, low), and the user may define the actual resolution of each category by himself. For example, high, medium, and low intermediate resolutions correspond to 4K, 2K, and 1080P, respectively. In some embodiments, the computing circuit 110 takes the mode of the intermediate resolution as the resolution of the frame in step S270. If there is more than one mode, then the computation circuit 110 takes either mode as the resolution of the frame. Since the resolution of the image data is not frequently switched in many cases, the flow of fig. 2 is equivalent to detecting the resolution of the image data. After step S270 is completed, the computing circuit 110 deletes the stored intermediate resolution, and then ends the flow of fig. 2 (i.e. stops detecting the resolution of the image data), or returns to step S220 to continue detecting the resolution of the image data. If the resolution of the image data is continuously detected, the target frame determined by the computing circuit 110 in step S220 is different from the previous target frame.
FIG. 4 is a flowchart illustrating a method for detecting resolution of image data according to another embodiment of the present invention. Fig. 5 shows a picture frame, a sub-frame and a memory corresponding to the embodiment of fig. 4. Step S410 and step S420 in fig. 4 are the same as step S210 and step S220 in fig. 2, respectively, and thus are not described again. In step S430, the calculation circuit 110 determines the region of the target frame. The memory 510 of fig. 5 is part of the storage circuit 120 for storing sub-frames.
Referring to fig. 5, fig. 5 shows that the image data includes a plurality of frames (F1, F2,..fk,..fw., fn, 1. Ltoreq.k. Ltoreq.n), and each frame includes 9 sub-frames (SF 1, SF2, SF3, …, SF 9) (the number of sub-frames is merely for illustration, and not for limiting the invention). Any sub-frame of any frame corresponds to a region of the frame, respectively. Any region may be represented by coordinates where the region is located (i.e., relative) to the frame. In the example of fig. 5, when the target frame determined in step S420 is frame F1, the region determined in step S430 is the region (in gray) corresponding to sub-frame SF 1; when the target frame determined in step S420 is frame F2, the region determined in step S430 is the region corresponding to sub-frame SF 2; when the target frame determined in step S420 is the frame Fk, the region determined in step S430 is the region corresponding to the sub-frame SF 9; and when the target frame determined in step S420 is the frame Fn, the area determined in step S430 is the area corresponding to the sub-frame SF 1. In some embodiments, the two areas determined in step S430 two consecutive times have different positions or coordinates, for example, the position or coordinates of the sub-frame SF1 of the frame F1 is different from the position or coordinates of the sub-frame SF2 of the frame F2.
After determining the region, the computing circuit 110 selects the pixels of the region as the target sub-frame (step S435), in other words, the target sub-frame includes all the pixels of the region (stored in the memory 510), i.e., the target sub-frame is composed of all the pixels of the region. In some embodiments, step S430 and step S435 may be equivalent to selecting one target sub-frame from the target sub-frames. In some embodiments, memory 510 may be designed to store just one sub-frame to conserve hardware resources.
After the step S435 is completed, the computing circuit 110 performs the steps S440 to S470 to obtain the resolution of the image data. Since steps S440, S450, S460, S470 are the same as steps S240, S250, S260, S270, respectively, the description thereof will not be repeated.
In the embodiments of fig. 4 and 5, before obtaining the resolution of the image data, the computing circuit 110 performs several steps S420 (i.e. determining a plurality of target frames, such as the frames F1, F2, and the third order shown in fig. 5) and several steps S430-S435 (i.e. determining a plurality of sub-frames, such as the sub-frame SF1 of the frame F1, and the sub-frame SF2 of the frame F1, and the third order shown in fig. 5). It should be noted that, in the process of generating the resolution of the image data (i.e. performing steps S420-S460 multiple times), at least one of the determined areas in step S430 is different from the previously determined area. For example, as shown in fig. 5, step S430 is performed twice successively to determine the region corresponding to the sub-frame SF1 of the frame F1 and the region corresponding to the sub-frame SF2 of the frame F2.
Fig. 6 shows a frame and sub-frames corresponding to another example of the flow of fig. 4. The frames F1, F2, F3, F4, F5 of fig. 6 are consecutive frames. In this example, the calculation circuit 110 may not select a consecutive frame in step S420, for example, the frame F2 is not selected as the target frame. In this example, the calculation circuit 110 may determine the same region in step S430 two consecutive times, for example, the determined regions of the frame F3 and the frame F4 are the regions corresponding to the sub-frame SF 2.
The embodiment of fig. 4 requires fewer pixels to be stored simultaneously than the embodiment of fig. 2, in other words, the embodiment of fig. 4 occupies fewer storage circuits 120. More specifically, assuming that the embodiment of fig. 2 and the embodiment of fig. 4 use sub-frames of the same size (assuming that q pixels are included) and the target values (assuming that p) are the same, the embodiment of fig. 2 needs to store data of q×p pixels at the same time, whereas the embodiment of fig. 2 only needs to store data of q pixels at the same time.
FIG. 7 is a flowchart illustrating a method for detecting resolution of image data according to another embodiment of the present invention. Fig. 8 shows a picture frame, a sub-frame and a memory corresponding to the embodiment of fig. 7. Steps S710, S740, S750, S760 and S770 of fig. 7 are the same as steps S210, S240, S250, S260 and S270 of fig. 2, respectively, and thus are not described again. In step S720, the computing circuit 110 determines a first target frame and a second target frame, for example, frame F1 and frame F2, respectively. In step S730, the computing circuit 110 determines the first region and the second region, for example, a region corresponding to the sub-frame SF1 (shown in gray) and a region corresponding to the sub-frame SF2 (shown in gray). In step S735, the computing circuit 110 combines the sub-frame SF1 of the frame F1 (i.e., the pixel of the first region of the first target frame) with the sub-frame SF2 of the frame F2 (i.e., the pixel of the second region of the second target frame) to generate the target sub-frame. In other words, the sub-frame of the present embodiment includes a plurality of pixels of the first frame and a plurality of pixels of the second frame.
The memory 810 of fig. 8 is part of the storage circuit 120 for storing sub-frames. In some embodiments, memory 810 may be considered to contain two sub-memories: a sub-memory 812 and a sub-memory 814. The sub-memory 812 is used for storing pixels of the first region of the first frame, and the sub-memory 814 is used for storing pixels of the second region of the second frame.
In some embodiments, the first and second frames are consecutive frames (e.g., frame F1 and frame F2 of fig. 8). In other embodiments, the first and second frames are discontinuous frames (e.g., frame F1 and frame F3 of fig. 6).
On the premise that the target values are the same (the number of sub-frames processed is the same), the embodiment of fig. 7 can refer to pixel values taken from more frames to determine the resolution of the image data, compared to the embodiment of fig. 4. In some applications, it is possible for the embodiment of FIG. 7 to achieve a more accurate resolution than the embodiment of FIG. 4.
In some embodiments, the size of the first region is equal to the size of the second region, and each is half the size of the region employed in the embodiment of fig. 4. As such, the size of the sub-frame of the embodiment of fig. 7 is equal to the size of the sub-frame of the embodiment of fig. 4, i.e., the storage capacity of the memory 810 is the same as that of the memory 510.
Since the artificial intelligence model processes only one sub-frame at a time, rather than a complete frame, in the methods of fig. 2, 4 and 7, the time that the artificial intelligence model is continuously occupied can be shortened to avoid the performance degradation of the electronic device 100 due to the long-term processing of the same operation or task. In addition, since the processes of fig. 2, 4 and 7 may be interrupted and then sequentially performed (e.g., the computing circuit 110 performs another operation or task with the artificial intelligence model after completing step S250/S450/S750, and then sequentially performs the operation of resolution detection from step S260/S460/S760), design or operation flexibility of the electronic device 100 may be increased.
In some cases, in addition to executing an artificial intelligence model (hereinafter referred to as a first artificial intelligence model) for detecting resolution of image data, the electronic device 100 may execute another artificial intelligence model (hereinafter referred to as a second artificial intelligence model) to process tasks with high real-time or importance (hereinafter referred to as primary tasks), such as object recognition (including but not limited to face recognition). FIG. 9 is a flowchart illustrating a method for detecting resolution of image data according to another embodiment of the present invention. Program instructions implementing the flow of fig. 9, first and second artificial intelligence model parameters, and other parameters are also stored in the storage circuit 120 for execution by the computing circuit 110. When the computing circuit 110 determines that the primary task is to be processed (yes in step S910), the computing circuit 110 executes the second artificial intelligence model stored in the storage circuit 120 to process the primary task (step S920). After step S920 is completed, the flow returns to step S910. When the computing circuit 110 determines that there is no main task to be processed (no in step S910), the computing circuit 110 performs one of the sub-operations of detecting the resolution of the image data by using the first artificial intelligence model in the storage circuit 120 (step S930). After step S930 is completed, the calculation circuit 110 determines whether all sub-operations of the operation of the resolution detection have been completed (step S940). When the determination in step S940 is negative, the computing circuit 110 continues to determine whether there is a main task to be processed (step S910). When the determination in step S940 is yes, the computing circuit 110 determines the resolution of the image data (step S950).
In some embodiments, the sub-operation referred to in step S930 may be the operation of generating an intermediate resolution in the embodiments of fig. 2, 4 and 7, the details of step S940 are determining whether the number of stored intermediate resolutions is greater than or equal to the target value, and the details of step S950 are the same as those of step S270. In other words, step S930 may be equivalent to steps S210 to S250 of fig. 2 (where steps S210 and S220 are performed only once), steps S410 to S450 of fig. 4 (where step S410 is performed only once), or steps S710 to S750 of fig. 7 (where step S710 is performed only once). Because step S930 processes a portion of pixels, rather than all pixels, of a frame, the computing circuit 110 may perform the operation of detecting resolution with the same artificial intelligence model accelerator without affecting the primary tasks between performing the two primary tasks. In other words, based on the embodiments of fig. 2, 4 and 7, a single artificial intelligence model accelerator can simultaneously satisfy real-time demand tasks (e.g., primary tasks) and non-real-time demand tasks (e.g., detection resolution operations), increasing the performance and flexibility of use of the electronic device 100.
In another embodiment, the sub-operation of step S930 is a sub-model of the first artificial intelligence model. Referring to FIG. 10, FIG. 10 shows a first artificial intelligence model including a plurality of sub-models. The first artificial intelligence model 1000 includes sub-models 1010_1,1010_2, …,1010_k, …, and 1010_n (2 n, 2k n). The first sub-model 1010_1 receives and processes the frame to obtain an intermediate result R1. The calculation circuit 110 does not directly input the intermediate result R1 into the next submodel 1010_2, but stores it into the storage circuit 120. The last submodel 1010—n takes as input the intermediate result of the previous submodel and generates the resolution of the frame. The intermediate sub-model (i.e., the sub-model other than the first sub-model 1010_1 and the last sub-model 1010_n) takes as input the intermediate result of the previous sub-model and produces an intermediate result. In some embodiments, the intermediate results R1, R2, …, rk-1, rk, …, rn-1 include feature maps, hidden layer outputs, and other parameters generated by the first artificial intelligence model 1000 during operation. In some embodiments, the first artificial intelligence model 1000 may be formed by combining a decoder model (decoder model), an encoder model (encoder model) and a wave network model (wave net model) (please refer to:https://github.com/Rayhane-mamah/ Tacotron-2)。
fig. 11 shows a sub-step corresponding to step S930 of this embodiment. First, the computing circuit 110 determines whether the intermediate result is already stored in the storage circuit 120 (step S1110). When the computing circuit 110 performs the step S930 of fig. 9 for the first time, the storage circuit 120 does not store any intermediate result, so the determination result is no. Next, the computing circuit 110 obtains a frame of the image data (step S1120), then takes the first sub-model of the sub-models as the target sub-model, and inputs the frame into the target sub-model to obtain an intermediate result R1 (step S1130). After the calculating circuit 110 stores the intermediate result R1 in the storing circuit 120 (step S1140), step S930 ends, and the calculating circuit 110 continues to execute step S940.
When the computing circuit 110 does not execute the step S930 for the first time, the determining result of the step S1110 is yes, and the computing circuit 110 selects the next sub-model of the current target sub-model as the target sub-model (step S1150). The current target sub-model, i.e., the target sub-model determined when the calculation circuit 110 executed step S930 (i.e., the flow of fig. 11) last time. For example, when the computing circuit 110 performs step S930 for the second time, the current target sub-model is the sub-model 1010_1, so the computing circuit 110 takes the next sub-model of the sub-model 1010_1 (i.e. the sub-model 1010_2) as the target sub-model in step S1150. When the calculation circuit 110 performs the k-th execution of step S930, the current target sub-model is the sub-model 1010_k-1 (not shown), so the calculation circuit 110 takes the next sub-model of the sub-model 1010_k-1 (i.e., the sub-model 1010_k) as the target sub-model in step S1150.
After determining the new target submodel (i.e., after completing step S1150), the computing circuit 110 reads the previous (i.e., last stored) intermediate result from the storage circuit 120, and then inputs the intermediate result into the target submodel. More specifically, when the calculation circuit 110 performs the k-th step S930, the calculation circuit 110 inputs the submodel 1010—k with the intermediate result Rk-1 in step S1160.
Step S1160 generates an intermediate result (when k+.n) or resolution of the frame (when k=n). More specifically, when the target sub-model determined in step S1150 is not the last sub-model of the first artificial intelligence model 1000, step S1160 generates intermediate results; when the target sub-model determined in step S1150 is the last sub-model of the first artificial intelligence model 1000, step S1160 generates resolution. In other words, after the computing circuit 110 performs the step S930 multiple times to perform the sub-models 1010 of the first artificial intelligence model 1000, the resolution of the frame (i.e. the resolution of the image data) can be obtained. Therefore, when the calculating circuit 110 determines that the output of step S1160 is the intermediate result in step S1170, the calculating circuit 110 stores the intermediate result in the storage circuit 120 (step S1140), and then ends step S930; when the calculating circuit 110 determines that the output of step S1160 is the resolution in step S1170, the calculating circuit 110 stores or outputs the resolution (step S1180), and then ends step S930. In some embodiments, the resolution is, for example, a single value or a string, and the intermediate result is, for example, a collection of feature maps, hidden layer outputs, and/or other parameters.
In the embodiment of fig. 9, the artificial intelligence model accelerator is shared by multiple tasks, in other words, the electronic device 100 may utilize the same set of devices (e.g., the computing circuit 110 and the storage circuit 120) in combination with software (i.e., the first and second artificial intelligence models) to alternately perform resolution detection and object recognition to save costs (e.g., sharing the computing circuit 110 and/or using less storage circuitry). Further, since the operation of resolution detection is split into a plurality of sub-operations to be performed, even if hardware is shared by a plurality of artificial intelligence models, a main task of higher importance (e.g., security-based face recognition) is not affected by the resolution detection operation.
Since those skilled in the art can understand the implementation details and variations of the method of the present invention through the disclosure of the apparatus of the present invention, repeated descriptions are omitted herein for avoiding redundant description without affecting the disclosure requirement and the implementation of the method. It should be noted that the shapes, sizes, proportions of the components, the order of the steps, and the like, are merely illustrative, and are used as a matter of understanding of the present invention by those skilled in the art, and are not intended to limit the present invention.
Although the embodiments of the present invention have been described above, these embodiments are not intended to limit the present invention, and those skilled in the art may make various changes and modifications to the technical features of the present invention according to the explicit or implicit disclosure of the present invention, and all such changes and modifications are included in the present invention, in other words, the scope of the present invention should be defined by the claims of the present specification.
Description of the reference numerals
100 electronic device
110 computing circuit
120 storage circuit
310,320 picture frame
SF1-1, SF1-2, SF1-3, SF1-4, SF1-5, SF1-6, SF1-7, SF1-8, SF1-9, SF2-1, SF2-2, SF2-3, SF2-4, SF2-5, SF2-6, SF2-7, SF2-8: sub-frame
510,810 memory
F1, F2, F3, F4, F5, fk, fn: frames
SF1, SF2, SF3, SF4, SF5, SF6, SF7, SF8, SF9: sub-frame
R1, R2, rk-1, rk, rn-1: intermediate results
812, 814. Sub-memories
1000 first artificial intelligence model
1010_1,1010_2,1010_k, 1010_n. submodel
S210 to S270, S410 to S470, S710 to S770, S910 to S950, S1110 to S1180, steps
Claims (10)
1. A method for detecting resolution of image data, applied to an electronic device, a computing circuit of the electronic device executing an artificial intelligence model, the image data comprising a plurality of frames, each frame comprising a plurality of sub-frames, the method comprising:
(A) Generating a target sub-frame, wherein the number of pixels of the target sub-frame is smaller than the number of pixels of any one of the sub-frames;
(B) Inputting the target sub-frame into the artificial intelligence model, the artificial intelligence model processing a plurality of pixel data of the target sub-frame to produce an intermediate resolution corresponding to the pixel data;
(C) Storing the intermediate resolution;
(D) Repeating steps (a) through (C) to obtain a plurality of intermediate resolutions; and
(E) The resolution of the image data is determined based on the intermediate resolutions.
2. The method of claim 1, wherein step (D) comprises:
and (3) stopping executing the steps (A) to (C) when the number of the intermediate resolutions is larger than or equal to a target value.
3. The method of claim 2, wherein the target value is a number of sub-frames of the same target frame.
4. The method of claim 2, wherein the frames comprise a first frame and a second frame, and the two target sub-frames generated in step (a) two consecutive times are selected from the first frame and the second frame, respectively.
5. The method of claim 4, wherein the two target sub-frames generated in step (a) two consecutive times are respectively selected from a first region of the first frame and a second region of the second frame, the first region corresponding to a position of the first frame being equal to or different from a position of the second region corresponding to the second frame.
6. The method of claim 1, wherein step (E) comprises:
the mode of these intermediate resolutions is taken as the resolution.
7. An electronic device for detecting a resolution of image data, the image data comprising a plurality of frames, each frame comprising a plurality of sub-frames, the electronic device comprising:
a storage circuit storing a plurality of program instructions or program codes;
a computing circuit coupled to the storage circuit, wherein the computing circuit executes the program instructions or program code to execute the artificial intelligence model, the computing circuit further executing the program instructions or program code to perform the steps of:
(A) Generating a target sub-frame, wherein the number of pixels of the target sub-frame is smaller than the number of pixels of any one of the sub-frames;
(B) Inputting the target sub-frame into the artificial intelligence model, the artificial intelligence model processing a plurality of pixel data of the target sub-frame to produce an intermediate resolution corresponding to the pixel data;
(C) Storing the intermediate resolution;
(D) Repeating steps (a) through (C) to obtain a plurality of intermediate resolutions; and
(E) The resolution of the image data is determined based on the intermediate resolutions.
8. The electronic device of claim 7, wherein step (D) comprises:
and (3) stopping executing the steps (A) to (C) when the number of the intermediate resolutions is larger than or equal to a target value.
9. The electronic device of claim 8, wherein the frames comprise a first frame and a second frame, and the two target sub-frames generated in step (a) two consecutive times are selected from the first frame and the second frame, respectively.
10. A method of detecting resolution of image data applied to an electronic device, the electronic device comprising a computing circuit and a storage circuit, the storage circuit storing a plurality of program instructions or program code, the computing circuit executing the program instructions or program code to execute a first artificial intelligence model, the first artificial intelligence model comprising a plurality of sub-models, the method comprising:
(A) Obtaining a frame of the image data;
(B) Inputting the image frames into target sub-models of the sub-models to obtain a first intermediate result;
(C) Storing the first intermediate result;
(D) Executing a second artificial intelligence model using the computing circuitry;
(E) Taking the next sub-model of the target sub-model as the target sub-model;
(F) Inputting the first intermediate result into the target submodel to obtain a second intermediate result or the resolution of the image data; and
(G) When the second intermediate result is obtained in the step (F) instead of the resolution of the image data, taking the second intermediate result as the first intermediate result, and repeating the steps (C) to (F) until the resolution of the image data is obtained in the step (F).
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010122937.1A CN113316017B (en) | 2020-02-27 | 2020-02-27 | Method for detecting resolution and electronic device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010122937.1A CN113316017B (en) | 2020-02-27 | 2020-02-27 | Method for detecting resolution and electronic device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113316017A CN113316017A (en) | 2021-08-27 |
| CN113316017B true CN113316017B (en) | 2023-08-22 |
Family
ID=77370047
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010122937.1A Active CN113316017B (en) | 2020-02-27 | 2020-02-27 | Method for detecting resolution and electronic device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113316017B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20130112574A (en) * | 2012-04-04 | 2013-10-14 | 삼성전자주식회사 | Apparatus and method for improving quality of enlarged image |
| CN104581180A (en) * | 2014-12-31 | 2015-04-29 | 乐视网信息技术（北京）股份有限公司 | Video coding method and device |
| CN105074781A (en) * | 2013-03-15 | 2015-11-18 | 英特尔公司 | Variable resolution depth representation |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9996902B2 (en) * | 2016-01-19 | 2018-06-12 | Google Llc | Image upscaling |
-
2020
- 2020-02-27 CN CN202010122937.1A patent/CN113316017B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20130112574A (en) * | 2012-04-04 | 2013-10-14 | 삼성전자주식회사 | Apparatus and method for improving quality of enlarged image |
| CN105074781A (en) * | 2013-03-15 | 2015-11-18 | 英特尔公司 | Variable resolution depth representation |
| CN104581180A (en) * | 2014-12-31 | 2015-04-29 | 乐视网信息技术（北京）股份有限公司 | Video coding method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113316017A (en) | 2021-08-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11250591B2 (en) | Target detection method, system, and non-volatile storage medium | |
| EP3525138A1 (en) | Operation apparatus, operation execution device, operation excecution method and non-transitory computer-readable storage medium | |
| US20210357759A1 (en) | Task processing method and device based on neural network | |
| US11611715B2 (en) | System and method for event camera data processing | |
| US11783522B2 (en) | Animation rendering method and apparatus, computer-readable storage medium, and computer device | |
| US20210089871A1 (en) | Processing system and method for binary weight convolutional neural network | |
| US20180137407A1 (en) | Convolution operation device and convolution operation method | |
| US7912128B2 (en) | Motion vector estimation system and method thereof | |
| US11257252B2 (en) | Image coding apparatus, probability model generating apparatus and image compression system | |
| JP2006094494A (en) | Accelerating video encoding using graphics processor unit | |
| US8295551B2 (en) | System and method of adaptive vertical search range tracking for motion estimation in digital video | |
| WO2019210546A1 (en) | Data processing method and computing device | |
| Berjon et al. | GPU-based implementation of an optimized nonparametric background modeling for real-time moving object detection | |
| US6668020B2 (en) | Method for motion estimation in video coding | |
| US20130083042A1 (en) | Gpu self throttling | |
| CN113316017B (en) | Method for detecting resolution and electronic device | |
| TWI715448B (en) | Method and electronic device for detecting resolution | |
| CN110289861B (en) | Semi-precision compressed sensing sampling method | |
| US8559518B2 (en) | System and method for motion estimation of digital video using multiple recursion rules | |
| KR20030025058A (en) | A circuit and method for full search block matching | |
| JP4488805B2 (en) | Motion vector detection apparatus and method | |
| US10152766B2 (en) | Image processor, method, and chipset for increasing intergration and performance of image processing | |
| US11409356B1 (en) | Using data correlation to reduce the power consumption of signal processing systems without affecting the precision of computation | |
| JP3094893B2 (en) | Video encoding device | |
| US11836893B2 (en) | Online AI super resolution engine and operations |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |