TWI736060B - High-resolution video image processing method, device, and electronic device - Google Patents

Abstract

A video image processing method includes: (a) acquiring a video image with a high resolution; (b) defining one or a plurality of picture regions of interest in the video image as a ROI, and identifying the plurality of ROI regions to be processed; (c) analyzing and processing the plurality of ROI regions to obtain key information for each ROI region, and obtaining key ROI based on the key information area; and (d) sending the obtained ROI area and analysis result to a display terminal and displaying it on a screen of the display terminal. A video image processing device and an electronic device are also provided.

Description

High-resolution video image processing method, device and electronic equipment

The present invention relates to the technical field of video image processing, in particular to a high-resolution video image processing method, device and electronic equipment.

With the advancement of image sensing technology and codec technology, the application of ultra-high-definition video images (4K/8K) is becoming more and more popular. The ultra-high-definition image details will greatly improve the effects of various applications such as industrial inspection and surveillance. At the same time, artificial intelligence AI technology, including advances in deep learning, makes intelligent automatic analysis and processing of video images (such as classification, defect recognition, etc.) increasingly reliable. However, for ultra-high-definition video images such as 8K, the amount of data is 16 times that of 2K full-HD video. For real-time video processing, the system requires high computing power.

In addition, 8K ultra-high-definition decoding and display can be achieved in large-screen systems such as 8K TVs. But for many portable terminals, such as mobile phones, tablet computers, etc., ultra-high-definition decoding and display capabilities are still insufficient. Therefore, how to apply the results and video images processed by smart AI to devices with limited ultra-high-definition decoding and display capabilities, such as mobile phones, is an important issue currently facing.

In view of the foregoing, it is necessary to provide a high-resolution video image processing method, device, and electronic equipment.

One aspect of the present invention provides a video image processing method, the method comprising: (a) collecting a high-resolution video image; (b) defining one or more of the video images that are of interest and require further processing The screen area is ROI (Region Of Interest), and the complex ROI area that needs to be processed is identified; (c) The complex ROI area is analyzed and processed to obtain the key information of each ROI area, and based on the key information Obtain a further key ROI area; and (d) send the obtained ROI area and analysis result to the corresponding display terminal, and display it on the screen of the display terminal.

As a preferred solution, the method further includes a step of preprocessing the video image before performing step (b).

As a preferred solution, the method further includes the step of judging whether the computing resources are sufficient before performing step (b). When it is judged that the computing resources are sufficient, step (b) is executed; when it is judged that the computing resources are insufficient, the high-resolution The high-resolution video image is processed to convert it into a low-resolution video image, and then step (b) is performed.

As a preferred solution, the method further includes: analyzing and processing the key ROI area according to the obtained key ROI area to obtain the relevant information of the key ROI area, and obtaining further relevant ROI according to the relevant information area.

As a preferred solution, the ROI is a sub-picture of the panoramic picture, the panoramic picture itself, or the panoramic picture with reduced resolution.

Another aspect of the present invention provides a video image processing device, the device comprising: The acquisition module is used to collect high-resolution video images; the first processing module is used to define one or more regions of the video image that are of interest and require further processing as ROI (Region Of Interest) , And identify the complex ROI area that needs to be processed; the second processing module is used to analyze and process the complex ROI area to obtain the key information of each ROI area, and obtain further key information based on the key information ROI area; and a distribution module for sending the obtained ROI area and analysis results to the corresponding display terminal, and displaying it on the screen of the display terminal.

As a preferred solution, the acquisition module is also used to preprocess the video image.

As a preferred solution, the video image processing device further includes a judgment module and a down-resolution module. The judgment module is used to judge whether the computing resources are sufficient. When it is judged that the computing resources are insufficient, the down-resolution module is The group is used to process high-resolution video images to convert them into low-resolution video images.

As a preferred solution, the video image processing device further includes a third processing module for analyzing and processing the key ROI area according to the acquired key ROI area to obtain related information of the key ROI area, and A further associated ROI area is obtained according to the associated information.

As a preferred solution, the ROI is a sub-picture of the panoramic picture, the panoramic picture itself, or the panoramic picture with reduced resolution.

Another aspect of the present invention provides an electronic device that performs the above The video image processing method described in any one of the above items.

The video image processing method and device of the present invention can directly quickly identify high-resolution video images or low-resolution video images. In addition, the video image processing method can dynamically obtain the detailed ROI and analysis process based on the high-resolution image at each stage according to various configuration methods, and can also obtain a low-resolution panoramic preview. For example, the video image processing method can obtain and display the ROI of a smiling face with 8K level of detail on the display terminal of the small screen, the ROI of a crying face, or the ROI of a bleeding part, or browse the panoramic low-resolution pictures Introduction to the distribution and analysis of all human faces, and click to enter the face details browsing. In this way, when an 8K large display screen is not required, using the video image processing method of the present invention, the display terminal can also quickly access the ROI picture that needs attention in the 8K full-frame image, and quickly follow up processing.

100: Video image processing device

11: Acquisition module

13: Judgment module

15: Down-resolution module

16: The first processing module

17: The second processing module

18: The third processing module

19: Distribution module

200: electronic equipment

201: Memory

202: processor

203: Computer Program

205: Photography module

300: Video image processing system

301: display terminal

FIG. 1 is a flowchart of a video image processing method in a preferred embodiment of the present invention.

FIG. 2 is a schematic diagram of application scenarios of the video image processing method shown in FIG. 2.

Fig. 3 is a functional block diagram of a video image processing device in a preferred embodiment of the present invention.

4 is a functional block diagram of the video image processing system in the preferred embodiment of the present invention.

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those with ordinary knowledge in the field without creative work shall fall within the protection scope of the present invention.

It should be noted that when an element is referred to as being "electrically connected" to another element, it can be directly connected to the other element or a central element may also exist. When an element is considered to be "electrically connected" to another element, it can be a contact connection, for example, a wire connection, or a non-contact connection, for example, a non-contact coupling.

Unless otherwise defined, all technical and scientific terms used in this article have the same meanings commonly understood by those with ordinary knowledge in the field. The terms used in the description of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

Hereinafter, some embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the case of no conflict, the following embodiments and the features in the embodiments can be combined with each other.

Please refer to FIG. 1. FIG. 1 is a flowchart of a video image processing method in a preferred embodiment of the present invention. This method is used to effectively process high-resolution video images. It can be understood that, according to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

In step S01, a high-resolution video image is collected in real time, and the video image is preprocessed.

It can be understood that, in this embodiment, a camera module can be used to image the frame in the shooting field of view to obtain a frame composed of digitized original image element data. It can be understood that the picture is usually 4K/8K ultra-high-definition picture data with a large amount of data. In other words, in this embodiment, ultra-high-definition, such as 4K or 8K video images can be captured in real time.

It can be understood that, in this embodiment, a frame of ultra-high-definition picture composed of original primitive data can be periodically obtained from the camera module, and the picture primitive data format of the picture can be converted and pre-processed according to the needs of the application. For example, depending on the configuration, the number of frames collected per second may be 25, 30, 60, 120, etc. The format of the primitive may be UYUV444, YUV444, YUV422, YUV420 Wait. In addition, if necessary, pre-processing operations such as noise reduction or edge sharpening can be performed on the picture as needed.

In step S02, it is judged whether the computing resources are sufficient. If not, go to step S03. If yes, go to step S04.

Step S03, processing the high-resolution video image to convert it into a low-resolution video image.

For example, the 8K image is reduced to a 2K image.

Step S04: Define one or more regions of interest in the video image that need further processing as ROI (Region Of Interest), and identify the multiple ROI regions that need to be processed in the rough mode.

Understandably, the ROI is a sub-picture of the panoramic picture. In addition, the panoramic image itself or the reduced resolution panoramic image can also be regarded as a special kind of ROI image.

It can be understood that, in this embodiment, the face area can be defined as an ROI area, and a fast face recognition algorithm is used to quickly mark the plural face areas in the video image.

Step S05: Analyze and process the multiple ROI regions in the coarse mode screen to execute the fine mode to obtain key information of each ROI region, and obtain further key ROI regions based on the key information.

For example, for each ROI area (for example, each face area), further analysis is performed to obtain key information of each ROI area, such as emotional information of the face area. The emotional information can be joy, peace, crying, or others. Then obtain the corresponding key ROI area according to the key information of each ROI area. For example, according to the crying information in the above emotion information, the corresponding key ROI area is obtained as the face area with crying information.

Step S06: According to the acquired key ROI area, The domain is analyzed and processed to execute the correlation mode to obtain the correlation information of the key ROI area, and obtain the further correlation ROI area based on the correlation information.

For example, in step S05, when the crying face area is found in the analysis, the human contour area related to the face area is further calculated, and the human contour area is analyzed and detected to determine whether there is bleeding, falling posture, Related information such as the number of the jersey.

Step S07: According to the system configuration or the dynamic demand of a display terminal, the ROI image area and analysis result obtained in step S03, step S04, step S05 and/or step S06 are sent to the corresponding display terminal and displayed on the Display on the screen of the terminal.

It can be understood that, in other embodiments, when the correlation area analysis is not required, step S07 can be directly executed, and step S06 is omitted.

It can be understood that, in this embodiment, the video image processing method can directly quickly identify high-resolution video images or low-resolution video images. In addition, the video image processing method can dynamically obtain the detailed ROI and analysis process based on the high-resolution image at each stage according to various configuration methods, and can also obtain a low-resolution panoramic preview. For example, the video image processing method can obtain and display the ROI of a smiling face with 8K level of detail on the display terminal of the small screen, the ROI of a crying face, or the ROI of a bleeding part, or browse the panoramic low-resolution pictures Introduction to the distribution and analysis of all faces, and click to enter the face details browsing (for example, step S03). In this way, when an 8K large display screen is not required, using the video image processing method of the present invention, the display terminal can also quickly access the ROI picture that needs attention in the 8K full-frame image, and quickly follow up processing.

Please also refer to FIG. 2. The following takes the recognition and analysis of the face area as an example to further illustrate the video image processing method.

First of all, first carry out the collection of ultra-high-definition (for example, 8K) video images. Then determine the amount of calculation. When the amount of calculation is low, the ultra-high-definition video image is first reduced in resolution, for example, an 8K image is reduced to a 2K image. Then perform the rough mode to identify the relevant complex ROI regions, for example, identify multiple face regions, including smiling faces, crying faces, and so on. When the amount of calculation is high, the down-resolution operation is skipped directly to identify the relevant complex ROI area in the rough mode screen of the ultra-high-definition video image.

Then, the multiple face regions are further analyzed and processed, that is, a fine mode is executed to obtain key information of each ROI region, and further key ROI regions are obtained according to the key information. For example, the multiple face regions are analyzed and processed to obtain emotional information of each face region. The emotional information can be joy, peace, crying, or others. Then obtain the corresponding key ROI area according to the key information of each ROI area. For example, analyze the expressions or emotions of multiple face regions, and use the crying face region as the key ROI region.

Then, according to the acquired key ROI area, the key ROI area is analyzed and processed to execute the correlation mode, and then the correlation information of the key ROI area is obtained, and further correlation ROI areas are obtained according to the correlation information. For example, when a crying face area is found by analysis, the relevant human contour area related to the face area is further calculated, and analysis is performed to determine whether there is associated information such as bleeding, falling posture, and jersey number.

Finally, according to the system configuration or the dynamic requirements of the display terminal, the ROI images obtained at each stage, such as multiple ROI regions, key ROI regions, and/or associated ROI regions, and the analysis results are sent to the corresponding display terminal, and displayed as described Display on the screen of the display terminal.

Understandably, please refer to FIG. 3 together. Another embodiment of the present invention also provides a video Image processing device 100. The video image processing device 100 includes a collection module 11, a judgment module 13, a resolution reduction module 15, a first processing module 16, a second processing module 17, a third processing module 18, and a distribution module 19 .

Wherein, the acquisition module 11 is used for real-time acquisition of ultra-high-definition, such as 4K or 8K video images and pre-processing the video images.

The judgment module 13 is used to judge whether the computing resources are sufficient.

The down-resolution module 15 is used to process high-resolution video images to convert them into low-resolution video images. For example, the 8K image is reduced to a 2K image.

The first processing module 16 is used to define one or more regions of interest in the video image that need to be further processed as ROI (Region Of Interest), and in the rough mode screen, identify the multiple ROIs that need to be processed area.

Understandably, the ROI is a sub-picture of the panoramic picture. In addition, the panoramic image itself or the reduced resolution panoramic image can also be regarded as a special kind of ROI image.

It can be understood that, in this embodiment, the face area can be defined as an ROI area, and a fast face recognition algorithm is used to quickly mark the plural face areas in the video image.

The second processing module 17 is used to analyze and process the multiple ROI regions in the coarse mode screen to execute the fine mode to obtain key information of each ROI region, and obtain further information based on the key information The key ROI area. For example, for each ROI area (for example, each face area), further analysis is performed to obtain key information of each ROI area, such as emotional information of the face area. The emotional information can be joy, peace, crying, or others. Then obtain the corresponding key ROI area according to the key information of each ROI area. For example, according to the crying information in the above emotion information, the corresponding key ROI area is obtained with crying Weeping information of the face area.

The third processing module 18 is used to analyze and process the key ROI area according to the acquired key ROI area, so as to execute the correlation mode, and then obtain the key ROI area correlation information, and obtain the key ROI area according to the correlation information. Further associate the ROI area.

For example, when a crying face area is found in the analysis, the human contour area related to the face area is further calculated, and the human contour area is analyzed and detected to determine whether there is any association with bleeding, falling posture, jersey number, etc. News.

The distribution module 19 is used to transfer the ROI image area obtained by the first processing module 16, the second processing module 17, and/or the third processing module 18 according to the system configuration or the dynamic demand of the display terminal And the analysis result is sent to the corresponding display terminal and displayed on the screen of the display terminal.

It is understandable that please refer to FIG. 4 together, another embodiment of the present invention also provides an electronic device 200. The electronic device 200 includes a memory 201, a processor 202, and a computer program 203 stored in the memory 201 and running on the processor 202.

The electronic device 200 may further include a camera module 205. The photographing module 205 is used to perform image sensing on the picture within the shooting field of view to obtain a picture composed of digitized original primitive data.

The electronic device 200 may be a device with functions such as ultra-high-definition video image processing, real-time processor, image playback or display, such as an 8K camera, a high-speed server, an 8K TV integrated system, and the like. Those skilled in the art can understand that the schematic diagram is only an example of the electronic device 200, and does not constitute a limitation on the electronic device 200. It may include more or less components than those shown in the figure, or combine certain components, or different ones. part.

The processor 202 is used to execute the computer program 203 to implement the steps in the foregoing embodiment of the video image processing method, such as steps S01-S07 shown in FIG. 1. Alternatively, when the processor 202 executes the computer program 203, the function of each module/unit in the embodiment of the video image processing device 100 is realized, for example, the acquisition module 11, the judgment module 13, and the resolution reduction in FIG. 3 The module 15, the first processing module 16, the second processing module 17, the third processing module 18 and the distribution module 19.

The computer program 203 can be divided into one or more modules/units, and the one or more modules/units are stored in the memory 201 and executed by the processor 202 to complete this invention. The one or more modules/units may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program 203 in the electronic device 200. For example, the computer program 203 can be divided into the acquisition module 11, the judgment module 13, the down-resolution module 15, the first processing module 16, the second processing module 17, and the third processing module in FIG. Group 18 and distribution module 19.

The processor 202 may be a central processing unit (Central Processing Unit, CPU), or other general-purpose processors, digital signal processors (Digital Signal Processors, DSPs), dedicated integrated circuits (Application Specific Integrated Circuits, ASICs), Ready-made programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor 202 may also be any conventional processor, etc. The processor 202 is the control center of the electronic device 200, which uses various interfaces and lines to connect various parts of the entire electronic device 200 .

The memory 201 can be used to store the computer program 203 and/or modules/units. The processor 202 implements various functions of the electronic device 200 by running or executing computer programs and/or modules/units stored in the memory 201 and calling data stored in the memory 201. The memory 201 may mainly include a storage program area and a storage data area. Among them, the storage program area can store the operating system, at least one application program (such as sound playback function, image playback function, etc.) required by at least one function. The data storage area can store data created based on the use of the electronic device 200 (such as video data, audio data, phone book, etc.). In addition, the memory 201 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk drives, memory, plug-in hard disk drives, Smart Media Card (SMC), Secure Digital (SD) card, flash memory card (Flash Card), at least one magnetic disk memory device, flash memory device, or other volatile solid-state memory device.

If the integrated module/unit of the electronic device 200 is realized in the form of a software function module and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiments and methods, and can also be completed by computer programs instructing related hardware. The computer programs described in this section can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of each method embodiment described above. Wherein, the computer program includes computer program code, and the computer program code may be in the form of source code, executable files, or some intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, portable hard disk drive, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read -Only Memory), Random Access Memory (RAM, Random Access Memory), telecommunications and software distribution media, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased in accordance with the requirements of the legislation and patent practice in the jurisdiction, for example, in certain jurisdictions. Jurisdiction, according to legislation and patent practices, computer-readable media does not include electrical carrier signals and telecommunications signals.

Understandably, referring to FIG. 4 again, another embodiment of the present invention also provides a video image processing system 300. The video image processing system 300 includes the electronic device 200 and a display terminal 301. The display terminal 301 may be a terminal device with limited ultra-high-definition decoding and display capabilities such as mobile phones and tablet computers.

In the several embodiments provided by the present invention, it should be understood that the disclosed electronic device and method can be implemented in other ways. For example, the electronic device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division, and there may be other divisions in actual implementation.

In addition, the functional modules in the various embodiments of the present invention may be integrated in the same processing module, or each module may be physically stored separately, or two or more modules may be integrated in the same module. The above-mentioned integrated modules can be realized either in the form of hardware or in the form of hardware plus software functional modules.

For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description, and is therefore intended to fall within the equivalent requirements of the claims. All changes within the meaning and scope are included in the present invention. Any reference signs in the request shall not be regarded as the request item involved in the restriction. In addition, it is obvious that the word "include" does not exclude other modules or steps, and the singular does not exclude the plural. Multiple modules or electronic devices stated in the electronic device request can also be implemented by the same module or electronic device by software or hardware. Words such as first and second are used to denote names, but do not denote any specific order.

The above are only preferred embodiments of the present invention, and are not intended to limit the present invention in any form. In addition, those skilled in the art can also make other changes within the spirit of the present invention. Of course, these changes made according to the spirit of the present invention should all be included in the scope of protection claimed by the present invention.

Claims (9)

A video image processing method, which is improved in that the method includes: (a) collecting a high-resolution video image; (b) judging whether the computing resources are sufficient, and performing step (c) when it is judged that the computing resources are sufficient; When it is judged that the computing resources are insufficient, first process the high-resolution video image to convert it into a low-resolution video image, and then perform step (c); (c) define the video image One or more areas of interest and need to be further processed are ROI (Region Of Interest), and multiple ROI areas that need to be processed are identified; (d) analyzing and processing the multiple ROI areas to obtain each ROI The key information of the area, and further key ROI areas are obtained according to the key information; and (e) the obtained ROI area and the analysis result are sent to the corresponding display terminal and displayed on the screen of the display terminal. The video image processing method according to claim 1, wherein the method further includes a step of preprocessing the video image before performing step (c). The video image processing method according to claim 1, wherein the method further comprises: analyzing and processing the key ROI area according to the obtained key ROI area to obtain the related information of the key ROI area, and according to the obtained key ROI area The associated information obtains a further associated ROI area. The video image processing method according to claim 1, wherein the ROI is a sub-picture of the panoramic picture, the panoramic picture itself, or the panoramic picture with reduced resolution. A video image processing device, which is improved in that the device includes: a collection module for collecting high-resolution video images; a first processing module for defining one or more senses in the video image Interest and need to further The screen area of the step processing is ROI (Region Of Interest), and the complex ROI area to be processed is identified; the second processing module is used to analyze and process the complex ROI area to obtain the key of each ROI area Information, and obtain further key ROI areas based on the key information; and a distribution module for sending the obtained ROI areas and analysis results to the corresponding display terminal, and display them on the screen of the display terminal; wherein The video image processing device further includes a judgment module and a down-resolution module. The judgment module is used to judge whether the computing resources are sufficient. When it is judged that the computing resources are insufficient, the down-resolution module is used to High-resolution video images are processed to convert them into low-resolution video images. The video image processing device according to claim 5, wherein the acquisition module is further used for preprocessing the video image. The video image processing device according to claim 5, wherein the video image processing device further includes a third processing module for analyzing and processing the key ROI area according to the acquired key ROI area to obtain The related information of the key ROI area, and further related ROI areas are obtained according to the related information. The video image processing device according to claim 5, wherein the ROI is a sub-picture of the panoramic picture, the panoramic picture itself, or the panoramic picture with reduced resolution. An electronic device, which is improved in that the electronic device executes the video image processing method according to any one of claims 1 to 4.

Images