CN117311921A

CN117311921A - Scheduling method and device for image data processing and vehicle

Info

Publication number: CN117311921A
Application number: CN202210725815.0A
Authority: CN
Inventors: 穆健; 肖如楼; 刘修安; 白玉; 孙晓奇
Original assignee: BYD Co Ltd
Current assignee: BYD Co Ltd
Priority date: 2022-06-24
Filing date: 2022-06-24
Publication date: 2023-12-29

Abstract

The invention provides a scheduling method and device for image data processing and a vehicle, wherein the scheduling method and device are applied to a central processing unit of a vehicle-mounted terminal, the vehicle-mounted terminal further comprises a graphics processor and a digital signal processor, and the method comprises the following steps: acquiring the load state of each processor and the format information of the image data to be processed; each processor comprises the graphic processor, the central processing unit and the digital signal processor; determining a target processor from the processors according to the load state and the format information; and outputting the image data to be processed to the target processor so as to process the image data to be processed by the target processor. The embodiment of the invention effectively relieves the condition of excessively high occupation of the CPU or the GPU in the concurrent scene, and avoids the problem that the system is blocked easily and inconvenience is brought to the user in the mode of carrying out image data processing in the multi-concurrent scene by the existing Android vehicle machine system.

Description

Scheduling method and device for image data processing and vehicle

Technical Field

The present invention relates to the field of intelligent automobiles, and in particular, to a scheduling method and apparatus for image data processing, and a vehicle.

Background

In recent years, with the popularization of Android systems of mobile phones, a vehicle-mounted intelligent Android system vehicle-mounted platform rapidly occupies the market in modern automobiles.

In the existing Android car machine system, the number of cameras (cameras) is far greater than that of Android mobile phone systems, and a large number of concurrent use scenes of the cameras exist. For example, as shown In fig. 1, it is often required to receive image data output by a vehicle recorder (Digital Video Recorder, DVR), an In-vehicle passenger monitoring system (Occupancy Monitoring System, OMS), a driver monitoring system (Driver Monitoring System, DMS), and a panoramic image system through a camera hardware abstraction layer (Hardware Abstraction Layer, HAL), and then process and output to the DVR APP, the camera APP, an In-vehicle intelligent vision monitoring system (In-cabin monitoring System, IMS) APP, and the panoramic APP, respectively.

Because the processing of the data of the Camera in the Android car machine system is mainly a central processing unit (Central Processing Unit, CPU) or a graphic processing unit (GPUGraphics Processing Unit, GPU), the CPU or the GPU is occupied too high easily in a concurrent scene, so that the overall performance of the Android car machine system is reduced, and the system is blocked and the frame rate of the Camera is reduced. Especially, in order to match an open platform, the image correctness of the open camera APP in the vehicle-mounted system is corrected, and the rotation processing of the image is added on the basis of the original data processing, so that the system performance is more intense, the vehicle-mounted system cannot respond to the operation of a user in time, and inconvenience is brought to the user easily.

Disclosure of Invention

The embodiment of the invention provides a scheduling method, a scheduling device and a scheduling vehicle for image data processing, which are used for solving the problem that the image data processing mode of the existing Android vehicle-mounted system is easy to cause system blocking under a multi-concurrence scene, so that inconvenience is brought to a user.

In a first aspect, an embodiment of the present invention provides a scheduling method for image data processing, which is applied to a central processor of a vehicle-mounted terminal, where the vehicle-mounted terminal further includes a graphics processor and a digital signal processor, and the method includes:

acquiring the load state of each processor and the format information of the image data to be processed; each processor comprises the graphic processor, the central processing unit and the digital signal processor;

determining a target processor from the processors according to the load state and the format information;

and outputting the image data to be processed to the target processor so as to process the image data to be processed by the target processor.

Optionally, in the scheduling method, determining a target processor from the processors according to the load status and the format information includes:

Determining the digital signal processor as the target processor in the case that the load of the digital signal processor is less than a first load threshold;

and under the condition that the load of the digital signal processor is larger than or equal to a first load threshold value and the load of the graphic processor is larger than or equal to a second load threshold value, if the load of the central processing unit is smaller than a third load threshold value and the resolution of the image data to be processed is larger than a resolution threshold value, or if the load of the central processing unit is larger than or equal to the third load threshold value, determining the digital signal processor as the target processor, and reducing the frame rate of an image acquisition system related to the digital signal processor.

Optionally, in the scheduling method, determining a target processor from the processors according to the load status and the format information, further includes:

determining the graphics processor as the target processor in the case that the load of the digital signal processor is greater than or equal to a first load threshold and in the case that the load of the graphics processor is less than a second load threshold;

and if the resolution of the image data to be processed is smaller than the resolution threshold value, determining the central processor as the target processor.

Optionally, in the scheduling method, before the step of determining the target processor from the processors according to the load status and the format information, the method further includes:

determining the digital signal processor as the target processor under the condition that the resolutions before and after the processing of the image data to be processed are the same and the vehicle is in an OFF gear;

and if the resolution of the image data to be processed is the same before and after the processing and the vehicle is not in the OFF gear, executing the step of determining a target processor from the processors according to the load state and the format information if the image data to be processed needs to be processed in a rotating way.

determining the graphics processor as the target processor in the case that the resolutions before and after the processing of the image data to be processed are different;

and under the condition that the resolutions before and after the processing of the image data to be processed are the same and the vehicle is not in an OFF gear, if the image data to be processed does not need to be processed in a rotating way, determining the central processing unit as the target processor.

Optionally, in the scheduling method, when the digital signal processor is determined to be the target processor, outputting the image data to be processed to the target processor, so that the target processor processes the image data to be processed, including:

the central processing unit sends the image data to be processed to the digital signal processor through the buffer, and sends processing parameters corresponding to the image data to be processed to the digital signal processor, so that the digital signal processor processes the image data to be processed according to the processing parameters, and the processed image data is fed back to the central processing unit through the buffer.

Optionally, in the scheduling method, when determining that the digital signal processor is the target processor, the digital signal processor acquires a plurality of Y component data at a time to process the image data to be processed.

Optionally, in the scheduling method, when determining that the digital signal processor is the target processor and determining that format conversion and rotation processing are required to be performed on the image data to be processed according to the processing parameter, the digital signal processor performs rotation processing while performing format conversion on the image data to be processed.

In a second aspect, an embodiment of the present invention provides a scheduling apparatus for image data processing, which is applied to a central processor of a vehicle-mounted terminal, where the vehicle-mounted terminal further includes a graphics processor and a digital signal processor, and the apparatus includes:

the acquisition module is used for acquiring the load state of each processor and the format information of the image data to be processed; each processor comprises the graphic processor, the central processing unit and the digital signal processor;

the first determining module is used for determining a target processor from the processors according to the load state and the format information;

and the scheduling module is used for outputting the image data to be processed to the target processor so as to process the image data to be processed by the target processor.

In a third aspect, an embodiment of the present invention provides a vehicle, including a vehicle-mounted terminal, where the vehicle-mounted terminal includes a central processor, a graphics processor, and a digital signal processor, and further includes a scheduling device for image data processing according to the second aspect.

In a fourth aspect, an embodiment of the present invention provides an electronic device, including: a processor, a communication interface, a memory, and a communication bus; the processor, the communication interface and the memory complete communication with each other through a communication bus;

A memory for storing a computer program;

and a processor, configured to implement the steps in the scheduling method for image data processing according to the first aspect when executing the program stored in the memory.

In a fifth aspect, an embodiment of the present invention provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the scheduling method for image data processing described in the first aspect.

Aiming at the prior art, the invention has the following advantages:

in the embodiment of the invention, the format information at least comprising the load state of each processor and the image data to be processed is determined; each processor comprises a graphic processor, a central processing unit and a digital signal processor; determining a target processor from the processors according to the load state and the format information; and outputting the image data to be processed to the target processor so as to process the image data to be processed by the target processor. In the embodiment of the invention, not only is the digital signal processor introduced for processing the image data, but also the target processor for matching and processing the image data to be processed is determined according to the load states of the graphic processor, the central processing unit and the digital signal processor and by combining the format information of the image data to be processed, and the target processor is used for processing the image data to be processed, so that the processing system performance and the system resources of each processor can be considered, the situation that the occupation of a CPU or a GPU is too high in a concurrent scene is effectively relieved, and the problem that the system is blocked easily in a mode of processing the image data in a multiple concurrent scene by the existing Android vehicle system is avoided, and inconvenience is brought to a user.

The foregoing description is only an overview of the present invention, and is intended to be implemented in accordance with the teachings of the present invention in order that the same may be more clearly understood and to make the same and other objects, features and advantages of the present invention more readily apparent.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments will be briefly described below.

Fig. 1 is a schematic view of an Android car machine system camera usage scenario provided by an embodiment of the present invention;

FIG. 2 is a schematic diagram of a process of processing image data in an Android vehicle system according to the prior art;

FIG. 3 is a schematic diagram of another process for processing image data in an Android vehicle system according to the prior art;

fig. 4 is a schematic diagram of a scheduling method for image data processing according to an embodiment of the present invention;

FIG. 5 is a diagram of an image data processing framework of a camera module according to an embodiment of the present invention;

FIG. 6 is a schematic diagram of the interaction of hardware and software for processing image data using a digital signal processor according to an embodiment of the present invention;

fig. 7 is a schematic diagram of a process of processing image data in an Android vehicle system according to an embodiment of the present invention;

FIG. 8 is a flow chart of a digital signal processor for processing image data in YUYV format according to an embodiment of the present invention;

FIG. 9 is a flow chart of a digital signal processor for processing image data in YUYV format according to another embodiment of the invention;

FIG. 10 is a diagram illustrating the conversion and flipping of a plurality of YUYVs to YUV420 data formats according to an embodiment of the present invention;

FIG. 11 is a schematic view of a scheduling policy for image data processing in one implementation of an embodiment of the present invention;

FIG. 12 is a schematic view of a scheduling policy for image data processing in another implementation of the embodiment of the present invention;

fig. 13 is a schematic diagram of a scheduling apparatus for image data processing according to an embodiment of the present invention;

fig. 14 is a block diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

The existing automobile machine system is used by cameras in various scenes, for example, a vehicle recorder DVR needs to record road condition information of a user in real time; the intelligent vision monitoring system IMS of the automobile cabin comprises various monitoring systems such as a driver monitoring system DMS, a passenger monitoring system OMS, a rear blind area detection system RMS and the like; a stitching algorithm of a panoramic camera, etc.; and the entertainment third-party app which is automatically installed by the user can possibly perform functions such as image overturning and the like for better user experience; the above scenarios all involve varying degrees of image data processing effort requirements.

In the existing Android vehicle-mounted system, a CPU or a GPU is used as a main component when image data processing is performed, so that the CPU or the GPU occupies too high in a camera concurrency scene. Specifically, as shown in fig. 2-3, after the image data is output from the camera, the image data is firstly processed by a VIDEO Front End (VFE), then output into YUV422 format data, then the CPU or GPU converts the data into YUV420 data, then the CPU algorithm is used to rotate the horizontal screen data into a vertical screen format, and then the YUV420 data is transmitted to a corresponding APP for display through VIDEO preprocessing (Video preprocessing, VPE).

However, the CPU needs to respond and schedule the tasks of the whole system in time, the GPU needs to perform image rendering on the display of the system, and the higher load of the CPU can cause the system to be blocked and slow to operate; the higher load of the GPU can lead to slow interface refreshing, gives people a feeling of unsmooth, and particularly has stronger feeling when playing games.

In view of the above problems, an embodiment of the present invention provides a scheduling method for image data processing. Referring to fig. 4, a flowchart of a scheduling method for image data processing according to an embodiment of the present invention is shown, where the method is applied to a central processor of a vehicle-mounted terminal, and the vehicle-mounted terminal further includes a graphics processor and a digital signal processor, and the method includes steps 101 to 103.

Step 101, acquiring the load state of each processor and the format information of the image data to be processed; each processor comprises the graphics processor, the central processing unit and the digital signal processor.

In this step, the central processing unit is in communication connection with each camera module, the graphics processor and the digital signal processor, so that the tasks of the whole vehicle-mounted system can be responded and scheduled in time, and the load state of each processor and the format information of the image data processor to be processed output from the camera module can be obtained. Wherein the format information may include a resolution of the data.

And 102, determining a target processor from the processors according to the load state and the format information.

In this step, since the load state reflects the processing capacity of the processor, the format information of the image data to be processed expresses the processing performance requirement of the image data to be processed, and thus the processing capacity of each processor and the processing performance requirement of the image data to be processed can be integrated by the above-mentioned target information, the processor more suitable for the image data to be processed can be determined from the graphics processor, the central processing unit and the digital signal processor as the above-mentioned target processor. That is, the target processor not only can meet the processing requirement of the image data to be processed, but also has more resource advantage compared with other processors.

In this step, the digital signal processor may be introduced as the above-mentioned target processor to perform image data processing in the case where the digital signal processor is required to satisfy the processing requirements of the image data to be processed. The digital signal processor (Digital Signal Processor, DSP) is a processor of a special processing algorithm, and the vehicle-mounted system module is not operated on the DSP, so that the occupation of the DSP does not influence the performance of the system, and the consumption of the CPU and the GPU is reduced when the DSP is used for replacing the GPU and the CPU, and the overall performance of the system is improved.

Wherein, because the CPU and the GPU are used for processing the camera image data, the single frame processing time is limited by the system performance, and the upper limit of the resolution is 1080P; when the DSP is utilized to process the camera image data, the processing capacity of the algorithm based on the DSP can further improve the single-frame processing algorithm performance of the vehicle-mounted system, the resolution ratio of the processable image data can be increased, and the user experience is improved. Illustratively, the processing of single frame 1080P is reduced from the original 20ms to 10ms.

Wherein, when processing camera image data with the DSP, the processing algorithm can be extended based on processing requirements without introducing additional system resource pressure.

Step 103, outputting the image data to be processed to the target processor, so that the target processor processes the image data to be processed.

In this step, the image data to be processed output from the camera module is sent to the target processor, and then the target processor processes the image data. The target processor is a processor which is determined by integrating the processing capacity of each processor and the processing performance requirement of the image data to be processed and is more suitable for the image data to be processed, so that the target processor processes the image data to be processed, and the processing of the image data to be processed can be completed more quickly on the premise of meeting the data processing requirement.

In the scheduling method provided by the embodiment of the invention, not only is the digital signal processor introduced for processing the image data, but also the target processor for matching and processing the image data to be processed is determined according to the load states of the graphic processor, the central processing unit and the digital signal processor and by combining the format information of the image data to be processed, and the target processor is used for processing the image data to be processed, so that the processing system performance and the system resources of each processor can be considered, the situation that the CPU or the GPU occupies too high in a concurrent scene is effectively relieved, and the problem that the system is blocked easily in the mode of processing the image data in the multiple concurrent scenes of the existing Android vehicle-mounted system is avoided, and inconvenience is brought to users.

In the embodiment of the invention, DSP nodes are required to be integrated in an image data processing frame of a camera module, and particularly as shown in fig. 5, in the image data processing frame, data are processed by combining CPU, DSP, GPU according to different requirements and different scenes, so that the balance of the overall system performance is achieved, and better experience can be provided for users; the framework replaces the processing flow of the CPU and the GPU when the digital signal processor is taken as the target processor, reserves the processing capacity of the GPU and the CPU, and can process image data by taking the GPU or the CPU as the target processor when the GPU or the CPU has the advantage of data processing.

Optionally, in an embodiment, in a case that the digital signal processor is determined to be the target processor, outputting the image data to be processed to the target processor, so that the target processor processes the image data to be processed, specifically including:

In the embodiment of the invention, the central processing unit sends the processing parameters to the digital signal processor, the central processing unit and the digital signal processor exchange image data through the random access memory (random access memory, RAM), namely, the central processing unit sends the image data to be processed to the digital signal processor through the RAM, then the digital signal processor processes the image data to be processed according to the processing parameters, and then the processed image data is fed back to the central processing unit through the RAM for subsequent transmission, display and other steps.

Referring to fig. 6, a schematic diagram of a software-hardware interaction for processing image data by using a digital signal processor according to an embodiment of the present invention is shown. As shown in fig. 6, when conversion of the camera image data format yuyv to nv12 is required, a first format conversion function representing the above processing operation, which may be represented as yuyv_to_nv12, is transmitted to the digital signal processor; when the conversion from the data format yuyv to sp420 is needed and the image turning of 180 degrees is completed, and the conversion and the turning are completed simultaneously, a second format conversion function representing the processing operation is sent to the digital signal processor, wherein the second format conversion function can be represented as a rotation422ToSP420_180_ver; when the conversion from the data format yuyv to sp420 is needed and the 90-degree image flip is completed and the conversion and the flip are completed simultaneously, sending a third format conversion function representing the above processing operation to the digital signal processor, wherein the third format conversion function can be represented as rotation422ToSP420_90; and when the conversion of the data format yuyv to sp420 is required and the image flip of 270 degrees is completed and the conversion and the flip are completed simultaneously, a fourth format conversion function representing the above processing operation is sent to the digital signal processor, which may be represented as rotation422tosp420_270.

As shown in fig. 6, the central processing unit exchanges image data with the digital signal processor through the random access memory during data processing.

In the embodiment of the invention, when the digital signal processor is determined to be the target processor, the processing capability of the digital signal processor is introduced into the processing flow of the camera image data to replace the processing link of the original central processor or graphics processor. In the above scenario, the processing procedure of the image data in the Android car machine system is shown in fig. 7, after the image data is output from the camera, the image data is firstly subjected to VFE processing, then is output as YUV422 format data, then is converted into YUV420 data by the DSP, is subjected to rotation processing, and is then retransmitted to the corresponding APP for display through the VPE.

Optionally, in an implementation manner, in the scheduling method provided by the embodiment of the present invention, when determining that the digital signal processor is the target processor, according to the processing parameter, when determining that format conversion and rotation processing are required to be performed on the image data to be processed, the digital signal processor performs rotation processing while performing format conversion on the image data to be processed.

In the embodiment, the capability of the DSP is utilized to optimize the existing data processing algorithm, and two links of data format conversion and data rotation are combined into one step for processing, so that the frequency of data handling can be reduced, and the data processing performance is further optimized.

Specifically, referring to fig. 8, a flow chart of processing YUYV formatted image data incorporating a digital signal processor in one embodiment is shown. As shown in fig. 8, for a pixel point in the image data to be processed, it is firstly determined whether the position identifier i of the Y component is less than twice the resolution, if yes, it is indicated that the data is not processed, so that the corresponding Y component and UV component are assigned, then shifted to the next Y component, and the position identifier i of the Y component is processed by adding 2, i.e., i=i+2, if i is less than twice the resolution of the data, the above steps are continuously performed until i is determined to be greater than or equal to twice the resolution of the data.

In the above embodiment, one Y component data is processed per cycle, and the number of times of data processing is proportional to the resolution.

Optionally, in another implementation manner, in the scheduling method provided by the embodiment of the present invention, when determining that the digital signal processor is the target processor, the digital signal processor acquires a plurality of Y component data at a time to process the image data to be processed.

In the above embodiment, the capability of the DSP is utilized to optimize the existing data processing algorithm, and the processing such as format conversion and overturn is performed by acquiring a plurality of Y component data at a time, so that the computing power performance of the DSP can be fully utilized, and the image data processing speed can be significantly improved. Wherein the plurality of Y components may be 4, 8, 16, or the like.

Specifically, referring to fig. 9, a flow chart of processing YUYV formatted image data incorporating a digital signal processor in another embodiment is shown.

As shown in fig. 9, when processing of one frame of image data is started, the processing width widthcount is the image data width, and the processing height count is the image data height;

firstly judging whether the processing height of the data is greater than 0, if so, indicating that the data in the row direction is not processed, thus loading 16Y component data and 16 UV component data at a time for processing, specifically judging whether widthcount is greater than 0, if so, indicating that partial column data in the current row is not processed, thus loading 16Y component data and 16 UV component data for processing, and then reducing the processing width by 16 by using a function widthcount- =16; and continuously judging whether the widthcount is greater than 0, continuously loading 16Y component data and 16 UV component data for processing if the widthcount is greater than 0, indicating that the processing of the column data in the current row is completed if the widthcount is less than or equal to 0, so that the reassigned processing width widthcount is the width of the image data, performing 1 subtracting processing on the processing height by using a function height-1, and continuously judging whether the processing height is greater than 0, and if so, re-executing the steps and processing the data in the next row direction until the height is less than or equal to 0.

Referring to fig. 10, a schematic diagram of data format conversion and flipping of a plurality of YUYVs to YUV420 according to an embodiment of the present invention is shown.

In fig. 10, taking rotation by 90 degrees as an example, column directions Y0, Y4, Y8, Y12 are converted into row directions Y0, Y4, Y8, Y12, column directions Y1, Y5, Y9, Y13 are converted into row directions Y1, Y5, Y9, Y13, column directions Y2, Y6, Y10, Y14 are converted into row directions Y2, Y6, Y10, Y14, column directions Y3, Y7, Y11, Y15 are converted into row directions Y3, Y7, Y11, Y15, and Y plane data of YUV420 is obtained; meanwhile, the U and V signal values are subjected to interlaced sampling once in the row direction, so that the U and V signal values are converted into Y0, Y4, Y1 and Y5 which share U0 and V0, are converted into Y8, Y12, Y9 and Y13 which share U4 and V4, are converted into Y2, Y6, Y3 and Y7 which share U1 and V1, and are converted into Y10, Y14, Y11 and Y15 which share U5 and V5, and UV plane data of YUV420 is obtained.

Optionally, in one embodiment, the step 102 includes steps 201 to 202:

step 201, determining the digital signal processor as the target processor in the case that the load of the digital signal processor is smaller than a first load threshold.

In the step, the first load threshold is a load value for judging whether the computing power resource of the digital signal processor is sufficient or not; the method further includes determining whether the digital signal processor has sufficient computational power to process the image data to be processed by determining whether the load of the digital signal processor is less than the first load threshold.

Under the condition that the load of the digital signal processor is smaller than the first load threshold, the digital signal processor is low in load and sufficient in calculation power, so that the digital signal processor can be directly used as a target processor for processing image data to be processed, the powerful algorithm processing capacity of the digital signal processor is utilized, consumption of a CPU and a GPU in the image data processing process is reduced, and the single-frame processing algorithm performance of a vehicle-mounted system is improved.

Step 202, when the load of the digital signal processor is greater than or equal to a first load threshold and the load of the graphics processor is greater than or equal to a second load threshold, determining the digital signal processor as the target processor and reducing the frame rate of the image acquisition system related to the digital signal processor if the load of the central processor is less than a third load threshold and the resolution of the image data to be processed is greater than a resolution threshold, or if the load of the central processor is greater than or equal to the third load threshold.

In the step, the second load threshold is a load value for judging whether the computing power of the graphic processor is sufficient, and the third load threshold is a load value for judging whether the computing power of the central processor is sufficient; when the load of the digital signal processor is greater than or equal to the first load threshold and the load of the graphics processor is greater than or equal to the second load threshold, if the load of the central processing unit is less than the third load threshold and the resolution of the image data to be processed is greater than the resolution threshold, it is indicated that each processor does not have enough computing power to process the graphics data, and in consideration of the load states of the graphics processor and the central processing unit and the user experience, in order to avoid the situation that the system is blocked and the operation is slow due to the too high load of the central processing unit and the situation that the interface is refreshed slowly and is not smooth due to the too high load of the graphics processor, the digital signal processor is still determined as the target processor. In addition, considering that the user is likely to use the camera to entertain the scene under the scene, the user experience is ensured preferentially, so that an image acquisition system such as an IMS (Internet protocol multimedia subsystem) which operates in the background and needs to use a digital signal processor to process images can be notified to perform frame dropping processing, more DSP (digital signal processor) resources are released to entertain the scene of the user at the foreground, and the processing requirement of the current graphic data is met.

Under the above scenario, although the DSP may be processing in-car monitoring by using the algorithm, because the monitoring class runs in the background without involving the UI, and the monitoring class algorithm is limited by the algorithm, and does not need full frame processing, the frame dropping will not bring a larger experience drop to the user, so that the Camera frame dropping processing of the corresponding in-car monitoring class can be performed to release more DSP resources to the foreground user for entertaining the scene.

In this step, when the load of the digital signal processor is greater than or equal to the first load threshold and the load of the graphics processor is greater than or equal to the second load threshold, if the load of the central processor is less than the third load threshold and the resolution of the image data to be processed is greater than the resolution threshold, it is indicated that the processing of the image data to be processed may result in the overall performance and the data transfer performance of the central processor being reduced, and each processor does not have enough computing power to process the image data. And considering that the user uses the camera to entertain the scene under the scene, the user experience is ensured preferentially, so that the IMS running in the background can be informed to carry out the frame dropping process, more DSP resources are released to entertain the scene for the user in the foreground, and the processing requirement of the current graphic data is met.

Optionally, in a specific embodiment, the step 102 further includes steps 203 to 204.

And 203, determining the graphics processor as the target processor when the load of the digital signal processor is greater than or equal to a first load threshold and the load of the graphics processor is smaller than a second load threshold.

In this step, when the load of the digital signal processor is greater than or equal to the first load threshold, it is indicated that the load of the digital signal processor is high, and the computing power is relatively intense.

Wherein, in the case that the load of the graphics processor is smaller than the second load threshold, it is indicated that the load of the graphics processor is low and the computing power is sufficient, so that the graphics processor can be determined as the target processor.

And 204, determining the central processor as the target processor if the resolution of the image data to be processed is smaller than the resolution threshold under the condition that the load of the digital signal processor is larger than or equal to a first load threshold, the load of the graphic processor is larger than or equal to a second load threshold and the load of the central processor is smaller than a third load threshold.

In the step, when the load of the digital signal processor is greater than or equal to the first load threshold, the digital signal processor is higher in load and the calculation force is tense; when the load of the graphics processor is equal to or greater than the second load threshold, it is indicated that the load of the graphics processor is high, and the computing power is also intense, and it is possible to continue to determine whether the cpu has sufficient computing power to process the data in order to ensure the processing performance as much as possible.

Wherein the resolution threshold is an upper limit value of the resolution of the graphic data suitable for the CPU, for example 1080P; in the case where the load of the cpu is smaller than the third load threshold and the resolution of the image data to be processed is smaller than the resolution threshold, it is indicated that the load of the cpu is low, the calculation power is sufficient, and the processing of the graphics data to be processed does not cause the degradation of the overall performance of the cpu and the data transfer performance, so that the cpu can be determined as the target processor.

As shown in fig. 11, the overall scheduling policy of the above embodiment monitors the load of DSP, GPU, CPU in real time, and when the load of each processor is low, the DSP is preferably selected to process graphics data; when the DSP load is higher and the GPU load is lower, the GPU is preferentially selected to process the graphic data; while the load of the DSP and the GPU is very high, when the load of the CPU is lower and the resolution of the image is not more than 1080p, the CPU is preferentially used for processing the graphic data; when the load of the DSP and the GPU is very high and the load of the CPU is lower but the resolution of the image is more than or equal to 1080p, the DSP is considered to process the image data, and the IMS running in the background is informed to perform frame dropping processing, so that the processing capacities of the CPU, the GPU and the DSP are combined, the user experience is improved, and the computational power resource of the system is balanced.

Optionally, in one implementation, the method provided by the embodiment of the present invention further includes step 211 to step 213 before step 102.

Step 211, acquiring a gear state of the vehicle and processing parameters corresponding to the image data to be processed.

In the step, the gear state is used for judging whether the vehicle is in the OFF gear of the whole vehicle power failure; the above-mentioned processing parameters are used to represent specific processing operations for the image data to be processed, and may specifically include resolution conversion, data format conversion, rotation, and the like.

And 212, determining the digital signal processor as the target processor under the condition that the resolutions before and after the processing of the image data to be processed are the same and the vehicle is in an OFF gear.

In the step, if the resolution conversion of the image data to be processed is not required according to the processing parameters, the resolutions before and after the processing of the image data to be processed are the same, which means that the interpolation or scaling of the data is not required, so that the image processor is not required to process the data; if the vehicle is in the OFF gear at this time, the OFF gear does not have other scenes using the digital signal processor, so that the digital signal processor is determined as the target processor, and the efficiency of the digital signal processor can be fully utilized to improve the performance.

Step 213, if the resolutions before and after the processing of the image data to be processed are the same and the vehicle is not in the OFF gear, executing step 102 if the image data to be processed needs to be processed in a rotating way.

In the step, if the resolution conversion of the image data to be processed is not required according to the processing parameters, the resolutions before and after the processing of the image data to be processed are the same, which means that the interpolation or scaling of the data is not required, so that the image processor is not required to process the data; if the vehicle is not in the OFF gear at this time, the digital signal processor cannot determine whether there are enough computing power resources to process the graphics data, and thus the process goes to step 201, the load of DSP, GPU, CPU is monitored in real time, and the priority of DSP, GPU, CPU serving as the target processor is set to be sequentially reduced on the premise of sufficient computing power, so that the system resources of each processor are effectively balanced, and the user experience is improved.

Optionally, in a specific implementation, the method provided by the embodiment of the present invention further includes, before step 102, step 214 to step 215.

Step 214, determining the graphics processor as the target processor in the case that the resolutions of the image data to be processed before and after processing are different.

In this step, because the camera APP will seek the resolution data different from the driving map, the difference or scaling process needs to be performed on the original image data at this time, and the processing efficiency of the graphics processor on this scene is the highest, and such scene is generally a third-party camera APP processing foreground, but the APP occupying the GPU such as panorama is not in the foreground, and the GPU is used at a lower level, that is, other APPs will not excessively use the graphics processor, so the graphics processor is used as the target processor, and the graphics processor is used as the target processor to process the data, that is, after the data is stretched or interpolated, the processing quality and processing efficiency of the image can be ensured.

Step 215, if the resolutions before and after the processing of the image data to be processed are the same and the vehicle is not in the OFF gear, determining the central processing unit as the target processor if the image data to be processed does not need to be processed in a rotating way.

In the step, if the image data to be processed is determined to be unnecessary to carry out resolution conversion according to the processing parameters, the data is not required to be interpolated or scaled; if the vehicle is not in the OFF gear at this time, the digital signal processor cannot determine whether enough computing power resources exist to process the graphic data; if the image data to be processed does not need to be subjected to rotation processing, the occupation of computational effort is small in the scene, so that the central processing unit can be directly used as a target processor, the frequency of data handling can be reduced, and the data processing performance is optimized.

Referring to fig. 12, a scheduling policy diagram for image data processing in another embodiment of the present invention is shown. As shown in fig. 11, after receiving an image data stream input by a camera, determining whether the corresponding APP resolution requirements are the same as the original data, if not, stretching or interpolating the data is required, and because the GPU processing efficiency is higher than that of other processors, the GPU can ensure the smoothness of the data, so that the GPU is selected to process the data;

if the APP resolution is judged to be the same as the original data, judging whether the vehicle enters an OFF gear, wherein the user's requirements are long-range for observing the situations inside and outside the vehicle when in the OFF gear, but no other APP uses the GPU and the DSP, so that the efficiency advantage of the DSP can be fully exerted, the fluency when in the OFF gear is satisfied, and the user experience is improved;

if the vehicle is not in the OFF gear, determining a target processor for processing the data according to different requirements of image format conversion and rotation and the load state of each processor; if the image data five is not allowed to be selected, the data is directly processed by the central processing unit and is output to the corresponding APP for display; when the data needs to be rotated, the DSP is preferably selected to process the graphic data if the load of each processor is lower; when the DSP load is higher and the GPU load is lower, the GPU is preferentially selected to process the graphic data; while the load of the DSP and the GPU is very high, when the load of the CPU is lower and the resolution of the image is not more than 1080p, the CPU is preferentially used for processing the graphic data; when the load of the DSP and the GPU is high and the load of the CPU is low but the resolution of the image is greater than or equal to 1080p, the DSP is considered to process the image data, and the IMS running in the background is informed to perform frame dropping processing.

Fig. 13 is a schematic diagram of a scheduling device for image data processing, which is applied to a central processor of a vehicle-mounted terminal, wherein the vehicle-mounted terminal further includes a graphics processor and a digital signal processor, and the device includes:

a first obtaining module 131, configured to determine target information, where the target information includes at least a load status of each processor and format information of image data to be processed; each processor comprises the graphic processor, the central processing unit and the digital signal processor;

a first determining module 132, configured to determine a target processor from the processors according to the target information;

the scheduling module 133 is configured to output the image data to be processed to the target processor, so that the target processor processes the image data to be processed.

Optionally, in the apparatus, the first determining module 132 includes:

a first determining unit configured to determine the digital signal processor as the target processor in a case where a load of the digital signal processor is smaller than a first load threshold;

and the second determining unit is used for determining the digital signal processor as the target processor and reducing the frame rate of an image acquisition system related to the digital signal processor if the load of the central processing unit is greater than or equal to a third load threshold or the load of the central processing unit is smaller than the third load threshold and the resolution of the image data to be processed is greater than the resolution threshold under the condition that the load of the digital signal processor is greater than or equal to a first load threshold and the load of the graphic processor is greater than or equal to a second load threshold.

Optionally, in the apparatus, the first determining module 132 further includes:

a third determining unit configured to determine the graphics processor as the target processor in a case where a load of the digital signal processor is equal to or greater than a first load threshold and a load of the graphics processor is less than a second load threshold;

a fourth determining unit, configured to determine, when the load of the digital signal processor is greater than or equal to a first load threshold, the load of the graphics processor is greater than or equal to a second load threshold, and the load of the central processor is less than a third load threshold, the central processor as the target processor if the resolution of the image data to be processed is less than a resolution threshold;

optionally, the apparatus further comprises:

the second acquisition module is used for acquiring the gear state of the vehicle and the processing parameters corresponding to the image data to be processed before the step of determining the target processor from the processors according to the load state and the format information;

a second determining module, configured to determine, before the step of determining whether the load of the digital signal processor is less than a first load threshold, the digital signal processor as the target processor when the resolution of the image data to be processed is the same before and after the processing, and the vehicle is in an OFF range;

And the execution module is used for executing the step of determining whether the load of the digital signal processor is smaller than a first load threshold value if the image data to be processed needs to be processed in a rotating way under the condition that the resolutions before and after the image data to be processed are the same and the vehicle is not in an OFF gear.

Optionally, the apparatus further comprises:

a third determining module configured to determine, before the step of determining a target processor from the processors according to the load status and the format information, the graphics processor as the target processor in a case where resolutions before and after processing of the image data to be processed are different;

and the fourth determining module is used for determining the central processing unit as the target processing unit if the image data to be processed does not need rotation processing under the condition that the resolutions before and after the processing of the image data to be processed are the same and the vehicle is not in an OFF gear.

Optionally, in the device, the scheduling module 133 is specifically configured to, when determining that the digital signal processor is the target processor, send, by the central processing unit, the image data to be processed to the digital signal processor via a buffer, and send a processing parameter corresponding to the image data to be processed to the digital signal processor, so that the digital signal processor processes the image data to be processed according to the processing parameter, and feed back the processed image data to the central processing unit via the buffer.

The embodiment of the invention also provides a vehicle, which comprises a vehicle-mounted terminal, and the vehicle-mounted terminal comprises a central processing unit, a graphic processor, a digital signal processor and the scheduling device for processing the image data.

For the above-described apparatus and vehicle embodiments, since they are substantially similar to the scheduling method embodiments for image data processing, reference should be made to the description of the method embodiments for the relevant points.

According to the scheduling device and the vehicle for image data processing, which are provided by the embodiment of the invention, not only is the digital signal processor introduced for image data processing, but also the target processor for matching processing the image data to be processed is determined according to the load states of the graphic processor, the central processing unit and the digital signal processor and by combining the format information of the image data to be processed, and the target processor is used for processing the image data to be processed, so that the processing system performance and the system resources of each processor can be considered, the situation that the CPU or the GPU occupies too high in a concurrent scene is effectively relieved, and the problem that the system is blocked easily in a mode of processing the image data in a multiple concurrent scene in the conventional Android vehicle system is solved, and inconvenience is brought to a user.

The embodiment of the invention also provides an electronic device, as shown in fig. 14, which comprises a processor 1401, a communication interface 1402, a memory 1403 and a communication bus 1404, wherein the processor 1401, the communication interface 1402 and the memory 1403 communicate with each other through the communication bus 1404.

A memory 1403 for storing a computer program.

The processor 1401 is configured to execute a program stored in the memory 1403, and to implement the following steps:

determining target information, wherein the target information at least comprises the load state of each processor and the format information of the image data to be processed; each processor comprises the graphic processor, the central processing unit and the digital signal processor;

determining a target processor from the processors according to the target information;

The processor 1401 may further implement other steps in the scheduling method for image data processing, which is not described herein.

The communication bus mentioned by the above electronic device may be a peripheral component interconnect standard (Peripheral Component Interconnect, abbreviated as PCI) bus or an extended industry standard architecture (Extended Industry Standard Architecture, abbreviated as EISA) bus, or the like. The communication bus may be classified as an address bus, a data bus, a control bus, or the like. For ease of illustration, the figures are shown with only one bold line, but not with only one bus or one type of bus.

The communication interface is used for communication between the electronic device and other devices.

The memory may include random access memory (Random Access Memory, RAM) or non-volatile memory (non-volatile memory), such as at least one disk memory. Optionally, the memory may also be at least one memory device located remotely from the aforementioned processor.

The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU for short), a network processor (Network Processor, NP for short), etc.; but also digital signal processors (Digital Signal Processing, DSP for short), application specific integrated circuits (Application Specific Integrated Circuit, ASIC for short), field-programmable gate arrays (Field-Programmable Gate Array, FPGA for short) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

In yet another embodiment of the present invention, there is also provided a computer-readable storage medium having stored therein instructions that, when executed on a computer, cause the computer to perform the scheduling method for image data processing described in the above embodiment.

In a further embodiment of the present invention, a computer program product comprising instructions which, when run on a computer, cause the computer to perform the scheduling method for image data processing described in the above embodiment is also provided.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present invention, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another, for example, by wired (e.g., coaxial cable, optical fiber, digital Subscriber Line (DSL)), or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), etc.

It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

In this specification, each embodiment is described in a related manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. For embodiments of an apparatus, an electronic device, a computer-readable storage medium, and a computer program product containing instructions, the description is relatively simple, as it is substantially similar to method embodiments, with reference to the section of the method embodiments being relevant.

The foregoing description is only of the preferred embodiments of the present invention and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. The scheduling method for image data processing is characterized by being applied to a central processor of a vehicle-mounted terminal, wherein the vehicle-mounted terminal further comprises a graphics processor and a digital signal processor, and the method comprises the following steps:

2. The scheduling method according to claim 1, wherein determining a target processor from among the processors based on the load status and the format information, comprises:

3. The scheduling method according to claim 2, wherein determining a target processor from among the processors based on the load status and the format information, further comprises:

4. The scheduling method of claim 1, wherein prior to the step of determining a target processor from each of the processors based on the load status and the format information, the method further comprises:

acquiring a gear state of a vehicle and processing parameters corresponding to the image data to be processed;

5. The scheduling method of claim 4, wherein prior to the step of determining a target processor from each of the processors based on the load status and the format information, the method further comprises:

6. The scheduling method according to claim 1, wherein in the case where the digital signal processor is determined to be the target processor, outputting the image data to be processed to the target processor to process the image data to be processed by the target processor, specifically comprising:

7. The scheduling method according to claim 1, wherein in the case where the digital signal processor is determined to be the target processor, the digital signal processor acquires a plurality of Y-component data at a time to process when processing the image data to be processed.

8. The scheduling method according to claim 1, wherein when it is determined that the digital signal processor is the target processor, the digital signal processor performs rotation processing while performing format conversion on the image data to be processed when it is determined that format conversion and rotation processing on the image data to be processed are required according to the processing parameters.

9. A scheduling device for image data processing, characterized by being applied to a central processor of a vehicle-mounted terminal, the vehicle-mounted terminal further comprising a graphics processor and a digital signal processor, the device comprising:

10. An electronic device, comprising: a processor, a communication interface, a memory, and a communication bus; the processor, the communication interface and the memory complete communication with each other through a communication bus;

a memory for storing a computer program;

a processor for implementing the steps in the scheduling method for image data processing according to any one of claims 1 to 8 when executing a program stored on a memory.

11. A computer-readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the steps of the scheduling method for image data processing according to any one of claims 1 to 8.

12. A vehicle comprising a vehicle-mounted terminal including a central processor, a graphics processor, and a digital signal processor, and further comprising the scheduling apparatus for image data processing according to claim 9.