WO2023036133A1 - 图像检测与渲染方法及装置、设备、存储介质及计算机程序产品 - Google Patents

图像检测与渲染方法及装置、设备、存储介质及计算机程序产品 Download PDF

Info

Publication number
WO2023036133A1
WO2023036133A1 PCT/CN2022/117294 CN2022117294W WO2023036133A1 WO 2023036133 A1 WO2023036133 A1 WO 2023036133A1 CN 2022117294 W CN2022117294 W CN 2022117294W WO 2023036133 A1 WO2023036133 A1 WO 2023036133A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
thread
frame
detection
frame image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2022/117294
Other languages
English (en)
French (fr)
Inventor
文一凡
欧华富
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Sensetime Intelligent Technology Co Ltd
Original Assignee
Shanghai Sensetime Intelligent Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Sensetime Intelligent Technology Co Ltd filed Critical Shanghai Sensetime Intelligent Technology Co Ltd
Publication of WO2023036133A1 publication Critical patent/WO2023036133A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/20—Processor architectures; Processor configuration, e.g. pipelining

Definitions

  • the embodiment of the present disclosure is based on the Chinese patent application with the application number 202111063569.9, the application date is September 10, 2021, and the application name is "Image detection and rendering method and device, equipment, storage medium", and requires the Chinese patent application Priority, the entire content of this Chinese patent application is hereby incorporated into this disclosure by reference.
  • the present disclosure relates to the technical field of computer vision, and relates to but not limited to image detection and rendering methods and devices, equipment, storage media and computer program products.
  • Embodiments of the present disclosure provide an image detection and rendering method and device, device, storage medium and computer program product.
  • an embodiment of the present disclosure provides an image detection and rendering method, the method comprising:
  • the frame image to be rendered is determined by the first thread from the image after the image detection task is executed by the second thread, and an image rendering task is performed on the frame image to be rendered; wherein, the image rendering task is used to perform the image rendering task based on the The detection result of the frame image to be rendered is described, and the frame image to be rendered is rendered to obtain a special effect image of the frame image to be rendered.
  • an embodiment of the present disclosure provides an image detection and rendering device, the device includes a first acquisition module, a first execution module, and a second execution module, wherein:
  • the first acquiring module is configured to acquire frame images to be detected from the image sequence through the first thread
  • the first execution module is configured to execute an image detection task on the frame image to be detected through the second thread; wherein the image detection task is used to obtain a detection result of the frame image to be detected;
  • the second execution module is configured to use the first thread to determine the frame image to be rendered from the image after the image detection task is executed by the second thread, and execute an image rendering task on the frame image to be rendered; wherein, The image rendering task is used to render the frame image to be rendered based on the detection result of the frame image to be rendered to obtain a special effect image of the frame image to be rendered.
  • an embodiment of the present disclosure provides an electronic device, including a memory and a processor, the memory stores a computer program that can run on the processor, and the processor implements the above-mentioned image detection and rendering when executing the program steps in the method.
  • an embodiment of the present disclosure provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above image detection and rendering method are implemented.
  • an embodiment of the present disclosure provides a computer program product, including a computer-readable storage medium storing program code, and when instructions included in the program code are run by a processor of a computer device, the above-mentioned image detection and rendering are realized. method steps.
  • the frame image to be detected is acquired from the image sequence through the first thread; then the image detection task is performed on the frame image to be detected through the second thread; finally, the frame image to be detected is obtained through the first thread
  • FIG. 1 is a schematic flow diagram of an image detection and rendering method provided in the related art
  • FIG. 2 is a schematic flowchart of an image detection and rendering method provided by an embodiment of the present disclosure
  • FIG. 3A is a schematic flowchart of an image detection and rendering method provided by an embodiment of the present disclosure
  • FIG. 3B is a logic flow diagram of the task execution process provided by the embodiment of the present disclosure.
  • FIG. 4A is a schematic flowchart of an image detection and rendering method provided by an embodiment of the present disclosure
  • FIG. 4B is a logic flow diagram of the task execution process provided by the embodiment of the present disclosure.
  • FIG. 5A is a schematic flowchart of an image detection and rendering method provided by an embodiment of the present disclosure
  • FIG. 5B is a logic flow diagram of the task execution process provided by the embodiment of the present disclosure.
  • FIG. 6 is a schematic flowchart of an image detection and rendering method provided by an embodiment of the present disclosure.
  • FIG. 7 is a schematic flowchart of an image detection and rendering method provided by an embodiment of the present disclosure.
  • FIG. 8 is a schematic diagram of the composition and structure of an image detection and rendering device provided by an embodiment of the present disclosure.
  • FIG. 9 is a schematic diagram of a hardware entity of an electronic device provided by an embodiment of the present disclosure.
  • first ⁇ second ⁇ third involved in the embodiments of the present disclosure is only to distinguish similar objects, and does not represent a specific ordering of objects. Understandably, “first ⁇ second ⁇ third Where permitted, the specific order or sequence may be interchanged such that the embodiments of the disclosure described herein can be practiced in sequences other than those illustrated or described herein.
  • FIG. 1 An image detection and rendering method provided in the related art, as shown in FIG. 1 , this method is a face detection and rendering process based on a single JS thread 101, and the dotted line box 102 and the dotted line box 103 represent the current frame image and the next frame image respectively.
  • the JS thread 101 first executes the acquisition of the current frame image 11, wherein the current frame image can be the current frame data collected in real time from the terminal camera; then executes the face detection 12, that is, the current frame image is sent to The face detection module performs recognition to obtain the face detection result of the current frame image, and then the JS thread 101 executes image rendering 13 to complete the rendering of the current frame image based on the face detection result of the current frame image.
  • the JS thread 101 first executes to obtain the next frame of image 14, and then executes the face detection 15 to obtain the face detection result of the next frame of image, and then the JS thread 101 based on the following For the face detection result of one frame of image, perform image rendering 16 to complete the rendering of the next frame of image.
  • the above-mentioned JS thread detection and rendering task scheme performs detection and rendering logic for a single frame in sequence, that is, first detects the Nth frame, and then superimposes the detection results of the Nth frame image for rendering to obtain the Nth frame rendering result.
  • the face detection algorithm and the rendering process of special effects are carried out in sequence, resulting in a long time to process a single frame image, and the preview is laggy, which cannot meet the frame rate index of user experience.
  • all tasks will occupy the same thread resources, thereby reducing the display frame rate and experience.
  • the image detection and rendering methods provided by the embodiments of the present disclosure are described below.
  • the image detection module and the image rendering module in the embodiments of the present disclosure are called through different threads respectively, wherein the image rendering module is called by the first thread and used in the image detection algorithm
  • the second thread executes the WebAssembly solution.
  • the rendering engine By combining with the rendering engine, it provides users with special effect screen display based on image detection and special effect rendering, such as augmented reality (Augmented Reality, AR) experience.
  • augmented reality Augmented Reality, AR
  • World Wide Web also known as the World Wide Web
  • World Wide Web is a global, dynamically interactive, cross-platform distributed network based on hypertext and Hyper Text Transfer Protocol (Hyper Text Transfer Protocol, HTTP).
  • Graphical Information System The World Wide Web is a network service built on the Internet. It provides a graphical, easy-to-access and intuitive interface for viewers to search and browse information on the Internet. The information nodes of the network are organized into an interrelated network structure.
  • the Web Worker specification is part of the Hyper Text Markup Language (Hyper Text Markup Language 5, HTML5) standard, which defines a set of application programming interfaces (Application Programming Interface, API), which allows a JavaScript program to run on the main thread in another thread.
  • Hyper Text Markup Language Hyper Text Markup Language 5, HTML5
  • API Application Programming Interface
  • WebAssembly is a technical solution that can use JS language to write code and run on the browser. Due to its fast, efficient, and high-performance features, WebAssembly technology is widely used in the transplantation of various algorithms to the front end, greatly expanding the application scenarios of the algorithm. For example, WebAssembly technology is also used in the image detection algorithm. By combining with the rendering engine, it provides users with an augmented reality experience based on image detection.
  • An embodiment of the present disclosure provides an image detection and rendering method, which is applied to an electronic device.
  • the electronic devices include, but are not limited to, mobile phones, notebook computers, tablet computers and palm Internet devices, multimedia devices, streaming media devices, mobile Internet devices, wearable devices or other types of electronic devices.
  • the functions realized by the method can be realized by calling the program codes by the processor in the electronic device, and of course the program codes can be stored in the computer storage medium.
  • the electronic device at least includes a processor and a storage medium.
  • the processor can be used for processing of image detection and rendering, and the memory can be used for storing data required and generated during the process of image detection and rendering.
  • FIG. 2 is a schematic flowchart of an image detection and rendering method provided by an embodiment of the present disclosure. As shown in FIG. 2, the method at least includes the following steps:
  • Step S210 acquire the frame image to be detected from the image sequence through the first thread
  • the image sequence includes at least two frames of images.
  • the image sequence can be an image acquisition device set on the electronic device, such as an image captured by a camera module in real time; in other implementations, the image sequence can be transmitted to the The image that needs to be detected and rendered by the electronic device; in some implementations, the image sequence may also be the original captured image that the electronic device calls the local album and obtains from it in response to a task processing instruction, which is not discussed in this embodiment of the present disclosure limit.
  • the frame image to be detected is an image obtained by shooting the target object or a pre-stored image of the target object, so as to facilitate subsequent execution of an image detection task on the image to be detected.
  • the image detection task detects and recognizes the face area in the face image.
  • medical images detect human organs or parts in medical images through image detection tasks.
  • Step S220 performing an image detection task on the frame image to be detected through the second thread
  • the image detection task is used to obtain the detection result of the frame image to be detected.
  • the first thread is a Web-side JS thread
  • the second thread is a Worker thread
  • the JS thread is the main thread
  • the Worker thread is a sub-thread, and the same main thread can include At least one child thread.
  • Step S230 using the first thread to determine a frame image to be rendered from the image after the image detection task is performed by the second thread, and perform an image rendering task on the frame image to be rendered.
  • the image rendering task is used to render the frame image to be rendered based on the detection result of the frame image to be rendered to obtain a special effect image of the frame image to be rendered.
  • the execution result of the image detection task is to obtain the face recognition result in the selfie; the execution result of the image rendering task is to generate a rendered image of the face.
  • the above-mentioned image acquisition device provided on the electronic device can capture images within the field of view, and transmit multiple consecutive frames of images as video streams to the processor of the electronic device.
  • An image detection module including a detection algorithm and a recognition algorithm, and an image rendering module including a rendering algorithm are set in the processor of the electronic device.
  • the electronic device sequentially calls each frame of image for image detection and rendering, generates and displays a rendered image of the corresponding frame of image.
  • the image acquisition device may include, but is not limited to, a Red Green Blue (Red Green Blue, RGB) camera, or a Time Of Flight (Time Of Flight, TOF) camera, and the like.
  • Red Green Blue Red Green Blue
  • TOF Time Of Flight
  • the image detection module can determine the target object in each frame of image by detecting and recognizing each frame of image.
  • the target object can be set according to actual needs, for example, it can include but is not limited to: setting all objects in the image as the target object; or setting the object in the middle area of the image as the target object; or can be set to identify After the object in the image is displayed, the user selects the target object by himself, which is not limited in this embodiment of the present disclosure.
  • the rendered image is an AR special effect image
  • data packets corresponding to different AR special effects may be stored in the electronic device; data corresponding to different AR special effects may also be transmitted by other devices (for example, cloud servers) package to the electronic device; in the image rendering module, by using different data packages, different AR special effects can be displayed on the front-end interface of the electronic device.
  • the image detection module and the image rendering module in the embodiment of the present disclosure are called through different threads respectively.
  • the image rendering module can be performed on the first thread due to the limitation of the system browser. Therefore, the image detection module can run on the second thread through the WebAssembly scheme to realize the detection and recognition of each frame of image. In this way, the image detection task part no longer occupies the performance of the first thread, which can improve the display frame rate and experience effect.
  • the frame image to be detected is obtained from the image sequence through the first thread; then the image detection task is performed on the frame image to be detected through the second thread; finally, the frame image to be detected is obtained through the first thread through the Determine the frame image to be rendered in the image after the second thread performs the image detection task, and execute the image rendering task on the frame image to be rendered; thus, for the acquired image, the front end can use at least two threads to perform real-time detection and render special effects , so that all tasks will not occupy the same thread resources, which can improve the rendering frame rate and satisfy the user experience.
  • FIG. 3A is a schematic flowchart of an image detection and rendering method provided by an embodiment of the present disclosure. As shown in FIG. 3A, the method at least includes the following steps:
  • Step S310 acquiring an image sequence to be detected
  • the image sequence includes at least two frames of images.
  • Step S320 sequentially acquire the current frame image in the image sequence through the first thread
  • the first thread acquires the current frame image in the image sequence in real time.
  • the current frame image is the first frame image with the earliest time stamp in the image sequence.
  • Step S330 performing image detection on the current frame image acquired by the first thread through the second thread, to obtain a detection result of the current frame image
  • performing image detection on the current frame image may be to use any known image detection and recognition technology to detect objects in the current frame image to determine the target object in the current frame image.
  • the target object may be a human body, a human face, a human hand, other animals, and the like.
  • the detection and recognition technology includes, but is not limited to, technologies such as image segmentation and key point detection, for example. It can be understood that those skilled in the art may select one or more detection and recognition technologies to realize object recognition in the current frame image, which is not limited in this embodiment of the present disclosure.
  • the second thread is called and executed by the first thread, and every time the first thread acquires a frame of image, it sends the frame of image to the second thread, and performs image detection on the frame of image in the second thread to obtain the detection result After the code flow of the second thread is completed, the detection result of the current frame image is returned to the first thread.
  • step S340 the first thread renders the current frame image based on the detection result of the current frame image, and generates a rendered special effect image of the current frame image.
  • the first thread superimposes the pre-designed rendering logic on the detection result of the current frame image returned by the second thread, and obtains and displays the rendering result of the current frame image.
  • the first thread and the second thread respectively execute the image rendering task and the image detection task, so that the image detection part no longer occupies the performance of the first thread, and a better rendering frame rate can be obtained.
  • another thread is always idle.
  • the process of image acquisition and image rendering is performed on the first thread 301
  • the process of image detection is performed on the second thread 302 .
  • the dotted line box 303 represents the task processing flow of the current frame image, that is, the first thread 301 acquires the current frame image 31 and transmits the current frame image to the second thread 302, and then the second thread 302 performs image detection 32 on the current frame image. Then the second thread 302 returns the detection result of the current frame image to the first thread 301, and the first thread performs image rendering 33 to complete the rendering of the current frame image.
  • the dotted line box 304 represents the task processing flow of the next frame image, that is, the first thread 301 executes to acquire the next frame image 34, the second thread 302 performs image detection 35 for the next frame image, and the first thread executes image detection 35.
  • Rendering 36 completes the overall flow of the next frame of images.
  • the first thread 301 performs image rendering 33 on the current frame image based on the detection result of the current frame image returned by the second thread 302
  • the first thread 301 continues to obtain the next frame image 34, that is to say, through double
  • the threads (the first thread and the second thread) sequentially perform the detection task and the rendering task for each frame of image in the image sequence, and then perform the detection and rendering of the next frame of image.
  • the image detection task is first executed by the second thread to obtain the detection result, and then the image rendering task of the current frame image is executed by the first thread based on the detection result, thus
  • the detection process no longer occupies the performance of the first thread, and the display frame rate and experience effect can be improved.
  • FIG. 4A is a schematic flowchart of an image detection and rendering method provided by an embodiment of the present disclosure. As shown in FIG. 4A, the method can be implemented through the following steps:
  • Step S410 acquire the frame image to be detected from the image sequence through the first thread
  • the image sequence includes at least two frames of images.
  • Step S420 during the process of the first thread obtaining the current frame image from the image sequence as the frame image to be detected, the second thread executes the image processing on the previous frame image of the current frame image detection task;
  • the first thread acquires the previous frame image of the current frame image from the image sequence and transmits it to the second thread, and then the second thread performs image detection on the previous frame image. While the second thread performs image detection on the previous frame image, the first thread prepares the current frame image at the same time, so that after the second thread returns the detection result of the previous frame image, there is the current frame image, that is, new picture data to be detected .
  • the first thread has not yet obtained the detection result of the previous frame image returned by the second thread, that is to say, the first thread has no task to be rendered temporarily, and can Use the idle period to acquire the current frame image.
  • Step S430 during the process of the second thread performing the image detection task on the current frame image, using the first thread to use the previous frame image after the image detection task as the frame image to be rendered, Execute the image rendering task on the to-be-rendered frame image.
  • the second thread when the second thread renders the current frame image, it means that the second thread has completed the image detection task of the previous frame image and obtained the detection result of the previous frame image.
  • the first thread can The detection result of one frame of image is rendered to the previous frame of image to obtain the rendered image of the previous frame of image.
  • the detection result of the previous frame image is returned to the first thread by the second thread; that is, the second thread passes the first thread Called to return the detection result of the previous frame image to the first thread after the execution of the code flow of the second thread is completed.
  • the second thread After the second thread returns the detection result of the previous frame image to the first thread, the second thread is in an idle state. At this time, the second thread can directly perform image detection on the current frame image prepared by the first thread.
  • the process of image acquisition and image rendering is performed on the first thread 401
  • the process of image detection is performed on the second thread 402 .
  • the dotted line box 403 represents the task processing flow of the current frame image, that is, the first thread 401 obtains the current frame image 41 and transmits it to the second thread 402, and then the second thread 402 performs image detection 42, and then the second thread 402 returns to the current frame
  • the detection result of the image is sent to the first thread 401, and the first thread executes the image rendering 43 to complete the rendering of the current frame image.
  • the dotted line box 404 represents the task processing flow of the next frame image, that is, the first thread 401 executes to acquire the next frame image 44, the second thread 402 executes image detection 45, and the first thread executes image rendering 46 to complete the following The overall process of a frame of image.
  • the first thread 401 executes to obtain the next frame image 44; in the process of the first thread 401 performing image rendering 43, the second thread is also for the next frame image.
  • a frame of image executes the process of image detection 45 .
  • the current frame image and the next frame image are used as two adjacent frames of images to perform image detection tasks and image rendering tasks in parallel.
  • frame image 47 and sequentially perform image detection and image rendering to complete the detection and rendering of each frame image in the image sequence.
  • the next frame image is acquired by the first thread while the image detection task is performed by the second thread, and then the current frame image is processed by the first thread based on the detection result While the image is being rendered, the second thread performs the detection process of the next frame of the image.
  • the first thread and the second thread can perform their respective tasks at the same time by using frame interpolation to alternately perform image detection and image rendering, saving waiting time and improving Display frame rate and experience effect.
  • FIG. 5A is a schematic flowchart of an image detection and rendering method provided by an embodiment of the present disclosure. As shown in FIG. 5A, the method can be implemented through the following steps:
  • Step S510 acquiring the frame image to be detected from the image sequence through the first thread and putting it into the image detection queue
  • the image detection queue is used to buffer the frame images to be detected.
  • Step S520 executing the image detection task on the first image obtained from the image detection queue through the second thread
  • the first image is the first frame image in the image detection queue. That is to say, there are pictures that can be detected directly in the image detection queue, and when the second thread is idle, it can obtain a detection task from the image detection queue, without waiting for the first thread to render the last frame of image, and then pass the second thread One thread obtains the current frame image from the image sequence.
  • Step S530 using the first thread to obtain a second image from the image rendering queue as the frame image to be rendered, and execute the image rendering task on the frame image to be rendered;
  • the image rendering queue is used to cache images that have completed the image detection task
  • the second image in the image rendering queue is the first image or the first image that has completed the image detection task
  • the image of the previous frame; that is to say, the image rendering queue stores images and image detection results that can be rendered directly. To obtain a rendering task, there is no need to wait for the second thread to complete the detection process to obtain the detection result.
  • the rendering task cached in the image rendering queue is the previous frame image of the first image and its detection result; After the detection, the first image and its detection result are transmitted to the image rendering queue. At this time, the rendering tasks cached in the image rendering queue are the first image and its detection result.
  • the image rendering queue and the image detection queue are the same task queue, and at the same moment, at least one of the following images is stored in the task queue: at most one frame of the frame to be detected An image; at most one frame of the image on which the image detection task has been executed. Since the first thread has completed the rendering task of the previous frame image of the first image and removed the previous frame image, the image detection queue and the image rendering queue are optimized to be the same task queue, so that the queue includes at most one The detection task and at most one rendering task can meet the needs of detection and rendering at the same time, while reducing resource waste and improving the computing power of the device.
  • the process of image acquisition and image rendering is performed on the first thread 501
  • the process of image detection is performed on the second thread 502 .
  • the dotted line box 503 and the dotted line box 504 represent the code flow of loop execution, wherein in the first loop, in the process of image rendering 54 of the first frame image by the first thread 501, the second frame image is rendered by the second thread 502 Perform image detection 55 ; meanwhile, the first thread 501 can directly acquire the third frame image 56 after completing the image rendering 54 .
  • the second thread 502 performs image detection 58 on the third frame image; simultaneously, after the first thread 501 completes the image rendering 57
  • the fourth frame image 59 can be acquired directly.
  • obtaining the first frame image 51 is the starting point of the code flow at the initial time, representing that the first thread 501 obtains the picture task for the first time and transmits it to the image detection queue (not shown in the figure) as a frame image to be detected.
  • the second thread 502 When the second thread 502 is in an idle state, it directly transmits the first frame image to the second thread 502 for image detection 53 .
  • the second frame image 52 is acquired and transmitted to the image detection queue as a new frame image to be detected.
  • Image detection 53 execution ends in the second thread afterwards, and the detection result of the first frame image is returned by the second thread and transmitted to the image rendering queue (not shown in the figure) as the frame image to be rendered; at this moment, the second thread is in an idle state,
  • the second frame image can be obtained from the image detection queue for image detection 55, and at the same time, the frame image to be rendered is obtained from the image rendering queue through the first thread 501 and image rendering 54 is performed to complete the display of the special effect screen of the first frame image.
  • the detection and rendering tasks can be executed in parallel to improve the rendering frame rate and satisfy the user experience.
  • FIG. 6 is a schematic flowchart of an image detection and rendering method provided by an embodiment of the present disclosure. As shown in FIG. 6, the method at least includes the following steps:
  • Step S610 acquiring the frame image to be detected from the image sequence through the first thread and putting it into the image detection queue
  • the image detection queue is used to buffer the frame images to be detected.
  • Step S620 when it is determined that the second thread is in an idle state, the first frame image in the image detection queue is acquired by the first thread as the first image and sent to the second thread;
  • the first image is the first frame image acquired by the first thread from the image sequence and transmitted to the image detection queue.
  • the second thread when the second thread completes the image detection task on the first image and transfers the first image to the image rendering queue, it is determined that the second The thread is in the idle state; in some other possible implementations, the current frame image is the first frame image acquired by the first thread from the image sequence, and the first thread will obtain the first frame image from the image sequence for the first time.
  • the image acquired in the image sequence is transmitted to the image detection queue as the frame image to be detected, it is determined that the second thread is in the idle state. In this way, the idle state of the second thread in different situations is determined, so that the frame images to be detected in the image detection queue are passed to the second thread in time for image detection, thereby improving the overall detection and rendering efficiency of the program.
  • Step S630 performing image detection on the first image through the second thread to obtain a detection result of the first image
  • the detection result of the frame image to be detected is transmitted as a new rendering task to the image rendering queue through the second thread.
  • Step S640 using the first thread to obtain a second image from the image rendering queue as the frame image to be rendered, and execute the image rendering task on the frame image to be rendered.
  • the image rendering queue is used to cache images that have completed the image detection task
  • the second image in the image rendering queue is the first image or the first image that has completed the image detection task previous frame image.
  • the first thread renders the frame image to be rendered based on the detection result of the frame image to be rendered (the first image or the previous frame image).
  • the frame image to be detected in the image detection queue is passed to the second thread for image detection, thereby improving the overall detection and rendering of the program efficiency.
  • FIG. 7 is a schematic flowchart of an image detection and rendering method provided by an embodiment of the present disclosure. As shown in FIG. 7, the method at least includes the following steps:
  • Step S710 acquiring the frame image to be detected from the image sequence through the first thread and putting it into the image detection queue
  • the image detection queue is used to buffer the frame images to be detected.
  • Step S720 executing the image detection task on the first image acquired from the image detection queue through the second thread
  • the first image is the first frame image in the image detection queue.
  • the second thread performs image detection on the first image, obtains a detection result of the first image, and transmits the first image after the detection task is executed to the image rendering queue as a rendering task.
  • Step S730 using the first thread to obtain a second image from the image rendering queue as the frame image to be rendered, and execute the image rendering task on the frame image to be rendered;
  • the image rendering queue is used to cache images that have completed the image detection task
  • the second image in the image rendering queue is the first image or the first image that has completed the image detection task previous frame image.
  • Step S740 in the case that the frame image to be detected does not exist in the image detection queue and the first thread is in an idle state, obtain a frame image from the image sequence through the first thread as a new frame image The frame image to be detected is transmitted to the image detection queue.
  • the first thread finishes rendering the frame image to be rendered, it is determined that the first thread is in an idle state; in some other possible implementation manners, in the second During the process of the thread performing image detection on the image acquired by the first thread from the image sequence for the first time, it is determined that the first thread is in an idle state. In this way, the idle state of the first thread in different situations is determined, so as to better control the flow of picture tasks, so that the detection and rendering process of the overall solution can be executed efficiently and orderly.
  • the first thread obtains the next frame image from the image sequence, that is, the new image detection task is stored in the image
  • the detection queue is reserved, which can increase the intensity of task occupation as much as possible, thereby effectively improving the efficiency of program detection and rendering.
  • each embodiment in this specification is described in a progressive manner, each embodiment focuses on the differences from other embodiments, and the same and similar parts of each embodiment can be referred to each other.
  • the embodiment of the present disclosure takes the face detection and rendering process as an example, and the image rendering queue and the image detection queue are the same task queue, and proposes a two-threaded asynchronous face detection and rendering method based on Web Worker, by introducing dual threads and delayed rendering
  • the solution can detect and render in parallel, improve the rendering frame rate, and satisfy the user experience.
  • Step S801 determining a task queue for saving frame images to be detected and frame images to be rendered
  • the frame image to be rendered is an image that has been detected and obtained a face detection result.
  • Step S802 in the cycle of each frame, first detect whether the face detection module is idle; if idle, then obtain a frame image to be detected in the task queue and send it to the second thread for detection;
  • Step S803 the first thread checks whether there is a frame image to be rendered in the task queue for rendering
  • the frame image to be rendered is removed from the task queue.
  • Step S804 after the rendering is finished, if there is no frame image to be detected in the task queue, a new frame is acquired from the terminal camera as the frame image to be detected, and put into the task queue for standby.
  • the human face effects can be cyclically detected and rendered.
  • step S802 to step S804 there should be at most 2 tasks in the task queue (a frame image to be rendered is a task that has been detected and rendered, and a frame image to be detected is a task that has not been detected or is being detected), Therefore, while the first thread no longer needs the task of face detection, the intensity of task occupation can be increased as much as possible, which can effectively improve the efficiency of program detection and rendering.
  • the overall detection and rendering efficiency is improved by introducing a second thread to perform face detection tasks.
  • the first thread can simultaneously render the detection result of the N-1th frame image to generate a rendered image of the N-1th frame image.
  • This scheme of delaying rendering by one frame can detect and render in parallel, improve the rendering frame rate, and satisfy the user experience.
  • an embodiment of the present disclosure further provides an image detection and rendering device, the device includes each module included, and each unit included in each module, which can be implemented by a processor in an electronic device; of course It can also be realized by a specific logic circuit; in the process of implementation, the processor can be a central processing unit (Central Processing Unit, CPU), a microprocessor (Micro Processing Unit, MPU), a digital signal processor (Digital Signal Processor, DSP) or Field Programmable Gate Array (Field Programmable Gate Array, FPGA), etc.
  • CPU Central Processing Unit
  • MPU Micro Processing Unit
  • DSP Digital Signal Processor
  • FPGA Field Programmable Gate Array
  • Fig. 8 is a schematic diagram of the composition and structure of an image detection and rendering device provided by an embodiment of the present disclosure.
  • the device 800 includes a first acquisition module 810, a first execution module 820, and a second execution module 830, wherein:
  • the first acquiring module 810 is configured to acquire frame images to be detected from the image sequence through the first thread
  • the first execution module 820 is configured to execute an image detection task on the frame image to be detected through a second thread; wherein the image detection task is used to obtain a detection result of the frame image to be detected;
  • the second execution module 830 is configured to determine the frame image to be rendered from the image after the image detection task is executed by the second thread through the first thread, and execute an image rendering task on the frame image to be rendered; wherein The image rendering task is used to render the frame image to be rendered based on the detection result of the frame image to be rendered to obtain a special effect image of the frame image to be rendered.
  • the first acquisition module, the first execution module and the second execution module are executed on the current frame image in the image sequence
  • the first acquisition module, the first execution module and the second execution module further Execute on the next frame image; wherein, the frame image to be detected and the frame image to be rendered are both the current frame image.
  • the first executing module 820 is further configured to use the second A thread executes the image detection task on the previous frame image of the current frame image;
  • the second execution module 830 is further configured to perform the image detection task on the current frame image by the second thread Using the first thread to use the previous frame image after the image detection task as the to-be-rendered frame image, and execute the image rendering task on the to-be-rendered frame image.
  • the first acquisition module 810 is further configured to acquire the frame image to be detected from the image sequence through the first thread and put it into the image detection queue; wherein, the image detection The queue is used to cache the frame image to be detected; the first execution module 820 is further configured to execute the image detection task on the first image acquired from the image detection queue through the second thread; the The first image is the first frame image in the image detection queue; the second execution module 830 is further configured to obtain a second image from the image rendering queue through the first thread as the frame image to be rendered, and The frame image to be rendered executes the image rendering task; the image rendering queue is used to cache images that have completed the image detection task, and the second image in the image rendering queue is the image that has completed the image detection task The first image or the previous frame image of the first image.
  • the image detection queue and the image rendering queue are the same task queue, and at the same moment, at least one of the following images is stored in the task queue: at most one frame of the frame to be detected An image; at most one frame of the image on which the image detection task has been executed.
  • the device further includes a second acquisition module configured to, when the frame image to be detected does not exist in the image detection queue and the first thread is in an idle state, through the The first thread acquires a frame of image from the image sequence and transmits it to the image detection queue as a new frame image to be detected.
  • a second acquisition module configured to, when the frame image to be detected does not exist in the image detection queue and the first thread is in an idle state, through the The first thread acquires a frame of image from the image sequence and transmits it to the image detection queue as a new frame image to be detected.
  • the device further includes a first determining module configured to determine that the first thread is in an idle state after the first thread finishes rendering the frame image to be rendered; or, During the process of the second thread acquiring the first frame image from the image detection queue to perform image detection, it is determined that the first thread is in an idle state.
  • the first acquisition module includes a sending unit and a detection unit, wherein: the sending unit is configured to, when it is determined that the second thread is in an idle state, Obtaining the first frame image in the image detection queue as the first image and sending it to the second thread; the detection unit is configured to perform image detection on the first image through the second thread to obtain the Describe the detection results of the first image.
  • the device further includes a second determination module configured to complete the image detection task on the first image in the second thread and transmit the first image to the In the case of an image rendering queue, it is determined that the second thread is in the idle state; or, when the first thread first acquires an image from the image sequence as the frame image to be detected and transmits it to the image In the case of detecting the queue, it is determined that the second thread is in the idle state.
  • a second determination module configured to complete the image detection task on the first image in the second thread and transmit the first image to the In the case of an image rendering queue, it is determined that the second thread is in the idle state; or, when the first thread first acquires an image from the image sequence as the frame image to be detected and transmits it to the image In the case of detecting the queue, it is determined that the second thread is in the idle state.
  • the description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment.
  • the description of the method embodiments of the present disclosure please refer to the description of the method embodiments of the present disclosure for understanding.
  • the above image detection and rendering method is realized in the form of software function modules and sold or used as an independent product, it can also be stored in a computer-readable storage medium.
  • the computer software products are stored in a storage medium, and include several instructions to make An electronic device (which may be a smart phone with a camera, a tablet computer, etc.) executes all or part of the methods described in various embodiments of the present disclosure.
  • the aforementioned storage medium includes: various media that can store program codes such as U disk, mobile hard disk, read-only memory (Read Only Memory, ROM), magnetic disk or optical disk.
  • embodiments of the present disclosure are not limited to any specific combination of hardware and software.
  • an embodiment of the present disclosure provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps in the image detection and rendering method described in any of the above-mentioned embodiments are implemented.
  • a chip is also provided, and the chip includes at least one of the following: a programmable logic circuit; program instructions, which are used to implement any of the above-mentioned embodiments when the chip is running. The steps in the image detection and rendering method are described.
  • an embodiment of the present disclosure also provides a computer program product, which is used to implement the image detection and rendering method described in any of the above embodiments when the computer program product is executed by the processor of the electronic device. A step of.
  • FIG. 9 is a schematic diagram of hardware entities of an electronic device provided by an embodiment of the present disclosure.
  • a program when the processor 920 executes the program, implements the steps in any one of the image detection and rendering methods in the embodiments of the present disclosure.
  • the memory 910 is configured to store instructions and applications executable by the processor 920, and can also cache data to be processed or processed by the processor 920 and various modules in the electronic device (for example, image data, audio data, voice communication data and video data) Communication data), which can be realized by flash memory (FLASH) or random access memory (Random Access Memory, RAM).
  • FLASH flash memory
  • RAM Random Access Memory
  • the processor 920 executes the program, the steps of any one of the above-mentioned image detection and rendering methods are realized.
  • the processor 920 generally controls the overall operations of the electronic device 900 .
  • the above-mentioned processor can be an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a digital signal processor (Digital Signal Processor, DSP), a digital signal processing device (Digital Signal Processing Device, DSPD), a programmable logic device (Programmable Logic At least one of Device, PLD), Field Programmable Gate Array (Field Programmable Gate Array, FPGA), Central Processing Unit (Central Processing Unit, CPU), controller, microcontroller, microprocessor. Understandably, the electronic device that implements the above processor function may also be other, which is not specifically limited in this embodiment of the present disclosure.
  • the above-mentioned computer storage medium/memory can be read-only memory (Read Only Memory, ROM), programmable read-only memory (Programmable Read-Only Memory, PROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, EPROM), Electrically Erasable Programmable Read-Only Memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), Magnetic Random Access Memory (Ferromagnetic Random Access Memory, FRAM), Flash Memory (Flash Memory), Magnetic Surface Memory, CD-ROM, or CD-ROM (Compact Disc Read-Only Memory, CD-ROM) and other memories; it can also be various electronic devices including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants wait.
  • the disclosed devices and methods may be implemented in other ways.
  • the device embodiments described above are schematic.
  • the division of the units is a logical function division.
  • the coupling, or direct coupling, or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical or other forms of.
  • the units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed to multiple network units; Part or all of the units can be selected according to actual needs to achieve the purpose of the solutions of the embodiments of the present disclosure.
  • each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may be used as a single unit, or two or more units may be integrated into one unit; the above-mentioned integration
  • the unit can be realized in the form of hardware or in the form of hardware plus software functional unit.
  • the above-mentioned integrated units of the present disclosure are realized in the form of software function modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.
  • the computer software products are stored in a storage medium, and include several instructions to make The equipment automatic test line executes all or part of the methods described in various embodiments of the present disclosure.
  • the aforementioned storage medium includes various media capable of storing program codes such as removable storage devices, ROMs, magnetic disks or optical disks.
  • the products applying the disclosed technical solution have clearly notified the personal information processing rules and obtained the individual's independent consent before processing personal information.
  • the disclosed technical solution involves sensitive personal information the products applying the disclosed technical solution have obtained individual consent before processing sensitive personal information, and at the same time meet the requirement of "express consent". For example, at a personal information collection device such as a camera, a clear and prominent sign is set up to inform that it has entered the scope of personal information collection, and personal information will be collected.
  • the personal information processing rules may include Information processor, purpose of personal information processing, processing method, type of personal information processed and other information.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Quality & Reliability (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Processing Or Creating Images (AREA)

Abstract

本公开实施例公开了一种图像检测与渲染方法、装置、设备、存储介质及计算机程序产品，所述方法包括：通过第一线程从图像序列中获取待检测帧图像；通过第二线程对所述待检测帧图像执行图像检测任务；其中，所述图像检测任务用于获得所述待检测帧图像的检测结果；通过所述第一线程从经过所述第二线程执行图像检测任务后的图像中确定待渲染帧图像，对所述待渲染帧图像执行图像渲染任务；其中，所述图像渲染任务用于基于所述待渲染帧图像的检测结果，对所述待渲染帧图像进行渲染，得到所述待渲染帧图像的特效画面。

Description

图像检测与渲染方法及装置、设备、存储介质及计算机程序产品
相关申请的交叉引用
本公开实施例基于申请号为202111063569.9、申请日为2021年09月10日、申请名称为“图像检测与渲染方法及装置、设备、存储介质”的中国专利申请提出，并要求该中国专利申请的优先权，该中国专利申请的全部内容在此以引入方式并入本公开。
技术领域
本公开涉及计算机视觉技术领域，涉及但不限定于图像检测与渲染方法及装置、设备、存储介质及计算机程序产品。
背景技术
随着摄像技术的普及和应用，利用设备进行图像拍摄或视频拍摄已经成为人们日常生活中用来记录生活的重要手段之一。为了呈现的视觉效果更加丰富多彩，往往需要在图像检测的基础上再做特效渲染，将最终渲染后的图像呈现给用户。
相关技术中由于前端(Web)的Java描述语言(JavaScript，JS)单线程工作机制，基于深度学习的图像检测和特效渲染按序进行，导致单帧时间过长，预览时较为卡顿，不能满足用户体验的帧率指标。
发明内容
本公开实施例提供一种图像检测与渲染方法及装置、设备、存储介质及计算机程序产品。
本公开实施例的技术方案是这样实现的：
第一方面，本公开实施例提供一种图像检测与渲染方法，所述方法包括：
通过第一线程从图像序列中获取待检测帧图像；
通过第二线程对所述待检测帧图像执行图像检测任务；其中，所述图像检测任务用于获得所述待检测帧图像的检测结果；
通过所述第一线程从经过所述第二线程执行图像检测任务后的图像中确定待渲染帧图像，对所述待渲染帧图像执行图像渲染任务；其中，所述图像渲染任务用于基于所述待渲染帧图像的检测结果，对所述待渲染帧图像进行渲染，得到所述待渲染帧图像的特效画面。
第二方面，本公开实施例提供一种图像检测与渲染装置，所述装置包括第一获取模块、第一执行模块和第二执行模块，其中：
所述第一获取模块，配置为通过第一线程从图像序列中获取待检测帧图像；
所述第一执行模块，配置为通过所述第二线程对所述待检测帧图像执行图像检测任务；其中，所述图像检测任务用于获得所述待检测帧图像的检测结果；
所述第二执行模块，配置为通过所述第一线程从经过所述第二线程执行图像检测任务后的图像中确定待渲染帧图像，对所述待渲染帧图像执行图像渲染任务；其中，所述图像渲染任务用于基于所述待渲染帧图像的检测结果，对所述待渲染帧图像进行渲染，得到所述待渲染帧图像的特效画面。
第三方面，本公开实施例提供一种电子设备，包括存储器和处理器，所述存储器存储有可在处理器上运行的计算机程序，所述处理器执行所述程序时实现上述图像检测与渲染方法中的步骤。
第四方面，本公开实施例提供一种计算机可读存储介质，其上存储有计算机程序，该计算机程序被处理器执行时实现上述图像检测与渲染方法中的步骤。
第五方面，本公开实施例提供了一种计算机程序产品，包括存储了程序代码的计算机可读存储介质，所述程序代码包括的指令被计算机设备的处理器运行时，实现上述图像检测与渲染方法的步骤。
本公开实施例提供的技术方案带来的有益效果至少包括：
在本公开实施例中，首先，通过第一线程从图像序列中获取待检测帧图像；然后通过所述第二线程对所述待检测帧图像执行图像检测任务；最后通过所述第一线程从经过所述第二 线程执行图像检测任务后的图像中确定待渲染帧图像，对所述待渲染帧图像执行图像渲染任务；如此，针对获取的图像，前端可以采用至少两个线程进行实时检测并渲染特效，这样不会存在所有任务都挤占同一个线程资源，能够提升渲染帧率，满足用户体验。
应当理解的是，以上的一般描述和后文的细节描述是示例性和解释性的，而非限制本公开。
附图说明
为了清楚地说明本公开实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，下面描述中的附图是本公开的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其它的附图，其中：
图1为相关技术中提供的图像检测与渲染方法的流程示意图；
图2为本公开实施例提供的图像检测与渲染方法的流程示意图；
图3A为本公开实施例提供的图像检测与渲染方法的流程示意图；
图3B为本公开实施例提供的任务执行过程的逻辑流程图；
图4A为本公开实施例提供的图像检测与渲染方法的流程示意图；
图4B为本公开实施例提供的任务执行过程的逻辑流程图；
图5A为本公开实施例提供的图像检测与渲染方法的流程示意图；
图5B为本公开实施例提供的任务执行过程的逻辑流程图；
图6为本公开实施例提供的图像检测与渲染方法的流程示意图；
图7为本公开实施例提供的图像检测与渲染方法的流程示意图；
图8为本公开实施例提供的图像检测与渲染装置的组成结构示意图；
图9为本公开实施例提供的电子设备的硬件实体示意图。
具体实施方式
为使本公开实施例的目的、技术方案和优点更加清楚，下面将结合本公开实施例中的附图，对本公开实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本公开一部分实施例，而不是全部的实施例。以下实施例用于说明本公开，但不用来限制本公开的范围。基于本公开中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本公开保护的范围。
在以下的描述中，涉及到“一些实施例”，其描述了所有可能实施例的子集，但是可以理解，“一些实施例”可以是所有可能实施例的相同子集或不同子集，并且可以在不冲突的情况下相互结合。
需要指出，本公开实施例所涉及的术语“第一\第二\第三”仅仅是是区别类似的对象，不代表针对对象的特定排序，可以理解地，“第一\第二\第三”在允许的情况下可以互换特定的顺序或先后次序，以使这里描述的本公开实施例能够以除了在这里图示或描述的以外的顺序实施。
本技术领域技术人员可以理解，除非另外定义，这里使用的所有术语(包括技术术语和科学术语)，具有与本公开实施例所属领域中的普通技术人员的一般理解相同的意义。还应该理解的是，诸如通用字典中定义的那些术语，应该被理解为具有与现有技术的上下文中的意义一致的意义，并且除非像这里一样被特定定义，否则不会用理想化或过于正式的含义来解释。
相关技术中提供的一种图像检测与渲染方法，如图1所示，该方法为基于单个JS线程101的人脸检测渲染流程，虚线框102和虚线框103分别表示针对当前帧图像和针对下一帧图像的任务流程。在针对当前帧图像的任务流程中，JS线程101先执行获取当前帧图像11，其中当前帧图像可以是从终端摄像头实时采集的当前画面数据；然后执行人脸检测12，即将当前帧图像送至人脸检测模块进行识别，得到当前帧图像的人脸检测结果，之后JS线程101基于当前帧图像的人脸检测结果，执行图像渲染13完成对当前帧图像的渲染。同样地，在针对下一帧图像的任务流程中，JS线程101先执行获取下一帧图像14，然后执行人脸检测 15，得到下一帧图像的人脸检测结果，之后JS线程101基于下一帧图像的人脸检测结果，执行图像渲染16完成对下一帧图像的渲染。
上述的JS线程进行人脸检测和渲染任务的方案，针对单帧按序进行检测和渲染逻辑，即先进行第N帧检测，然后叠加第N帧图像的检测结果进行渲染，得到第N帧渲染结果。但由于Web端JS单线程工作机制，人脸检测算法和特效渲染过程按序进行，导致单帧图像处理时间过长，预览时较为卡顿，不能满足用户体验的帧率指标。例如当在Web端使用摄像头进行实时人脸检测并渲染特效时，所有任务会挤占同一个线程资源，从而降低显示帧率及体验效果。
以下说明本公开实施例提供的图像检测与渲染方法，本公开实施例中的图像检测模块和图像渲染模块分别通过不同线程调用，其中，图像渲染模块使用第一线程调用，在图像检测算法中使用第二线程执行WebAssembly方案，通过与渲染引擎结合，为用户提供基于图像检测和特效渲染的特效画面展示，例如增强现实(Augmented Reality，AR)体验。
其中，全球广域网(World Wide Web，Web)也称为万维网，是一种基于超文本和超文本传输协议(Hyper Text Transfer Prototol，HTTP)的、全球性的、动态交互的、跨平台的分布式图形信息系统。万维网是建立在英特网(Internet)上的一种网络服务，为浏览者在英特网上查找和浏览信息提供了图形化的、易于访问的直观界面，其中的文档及超级链接将英特网上的信息节点组织成一个互为关联的网状结构。
Web Worker规范是超文本标记语言(Hyper Text Markup Language 5，HTML5)标准的一部分，这一规范定义了一套应用程序编程接口(Application Programming Interface，API)，该规范允许一段JavaScript程序运行在主线程之外的另外一个线程中。
Web Worker规范中定义了两类工作线程，分别是专用线程(Dedicated Worker)和共享线程(Shared Worker)，其中，专用线程只能为一个页面所使用，而共享线程则可以被多个页面所共享。
WebAssembly是一种可以使用JS语言编写代码并且能在浏览器上运行的技术方案。WebAssembly技术由于其快速、高效、高性能等特点，广泛应用于各类算法到前端的移植，极大地拓展了算法的应用场景。例如图像检测算法中也使用了WebAssembly技术，通过与渲染引擎结合，为用户提供基于图像检测的增强现实体验。
本公开实施例提供一种图像检测与渲染方法，应用于电子设备。所述电子设备包括但不限于手机、笔记本电脑、平板电脑和掌上上网设备、多媒体设备、流媒体设备、移动互联网设备、可穿戴设备或其他类型的电子设备。该方法所实现的功能可以通过电子设备中的处理器调用程序代码来实现，当然程序代码可以保存在计算机存储介质中，可见，该电子设备至少包括处理器和存储介质。处理器可以用于进行图像检测与渲染的处理，存储器可以用于存储进行图像检测与渲染过程中需要的数据以及产生的数据。
图2为本公开实施例提供的图像检测与渲染方法的流程示意图，如图2所示，该方法至少包括以下步骤：
步骤S210，通过第一线程从图像序列中获取待检测帧图像；
这里，所述图像序列包括至少两帧图像。在一些实现方式中，该图像序列可以为电子设备上设置的图像采集装置，如摄像头模组实时采集的图像；在另一些实现方式中，该图像序列可以为其他设备通过即时通信的方式传输给电子设备的需要进行检测渲染展示的图像；在一些实现方式中，该图像序列也可以是电子设备响应于任务处理指令，调用本地相册并从中获取的原始采集图像，对此本公开实施例不做限制。
这里，所述待检测帧图像为对目标对象进行拍摄得到的图像或者预先存储的目标对象的图像，从而便于后续对该待检测图像执行图像检测任务。例如目标对象为人脸时，图像检测任务检测识别人脸图像中的人脸区域。或者针对医学影像，通过图像检测任务检测医学影像中的人体器官或部位。
步骤S220，通过所述第二线程对所述待检测帧图像执行图像检测任务；
这里，所述图像检测任务用于获得所述待检测帧图像的检测结果。
在一些实现方式中，所述第一线程为Web端JS线程，所述第二线程为Worker线程；在一些实现方式中，JS线程为主线程，Worker线程为子线程，同一主线程下可以包括至少一个子线程。
步骤S230，通过所述第一线程从经过所述第二线程执行图像检测任务后的图像中确定待渲染帧图像，对所述待渲染帧图像执行图像渲染任务。
这里，所述图像渲染任务用于基于所述待渲染帧图像的检测结果，对所述待渲染帧图像进行渲染，得到所述待渲染帧图像的特效画面。示例地，针对用户的自拍照，图像检测任务的执行结果则是获得该自拍照中的人脸识别结果；图像渲染任务的执行结果则是生成人脸的渲染图像。
在一种可能的实现方式中，如上所述的电子设备上设置的图像采集装置能够实施捕获视野范围内的图像，并将连续多帧图像作为视频流传输给电子设备的处理器。电子设备的处理器中设置了包含检测算法、识别算法的图像检测模块，以及包含渲染算法的图像渲染模块。在需要对采集的图像进行检测并渲染的场景下，电子设备依次调用每一帧图像以进行图像检测和渲染，生成相应帧图像的渲染图像并展示。
在一种可能的实现方式中，图像采集装置，可以包括但不限于为红绿蓝(Red Green Blue，RGB)摄像头，或飞行时间(Time Of Flight，TOF)摄像头等。
在一种可能的实现方式中，图像检测模块通过对每一帧图像进行检测识别，可以确定出每一帧图像中的目标对象。可以理解的是，目标对象可以根据实际需要进行设置，例如可以包括但不限于：设置为图像中所有对象均为目标对象；或设置图像中处于中间区域的对象为目标对象；或者可以设置为识别出图像中的对象后，用户自行选择目标对象等，对此本公开实施例不做限制。
在一种可能的实现方式中，所述渲染图像为AR特效图像，电子设备中可以存储有不同AR特效对应的数据包；还可以由其他设备(例如，云端服务器)传输不同AR特效对应的数据包到该电子设备；在图像渲染模块，通过使用不同的数据包，可以将不同的AR特效展示在电子设备的前端界面上。
值得注意的是，为了解决所有图像的获取任务、图像检测任务和图像渲染任务均挤占同一第一线程资源导致单帧处理时间过长，造成预览卡顿不能满足帧率(即单位时间内的画面帧数)指标的问题，本公开实施例中的图像检测模块和图像渲染模块分别通过不同线程调用。其中，图像渲染模块因为系统浏览器的限制可以在第一线程进行，因此，图像检测模块可以通过WebAssembly方案运行在第二线程以实现对每一帧图像的检测识别。这样，图像检测任务部分不再占用第一线程性能，能够提高显示帧率及体验效果。
在本公开实施例中，首先，通过第一线程从图像序列中获取待检测帧图像；然后通过第二线程对所述待检测帧图像执行图像检测任务；最后通过所述第一线程从经过所述第二线程执行图像检测任务后的图像中确定待渲染帧图像，对所述待渲染帧图像执行图像渲染任务；如此，针对获取的图像，前端可以采用至少两个线程进行实时检测并渲染特效，这样不会存在所有任务都挤占同一个线程资源，能够提升渲染帧率，满足用户体验。
在一些实施例中，按照所述图像序列中图像的先后顺序，在针对所述图像序列中当前帧图像执行完图像检测任务和图像渲染任务后，再继续执行下一帧图像；其中，所述待检测帧图像和待渲染帧图像均为所述当前帧图像。图3A为本公开实施例提供的图像检测与渲染方法的流程示意图，如图3A所示，该方法至少包括以下步骤：
步骤S310，获取待检测的图像序列；
这里，所述图像序列包括至少两帧图像。
步骤S320，通过所述第一线程依次获取所述图像序列中的当前帧图像；
这里，第一线程实时获取图像序列中的当前帧图像。在每一次循环中，当前帧图像均为图像序列中时间戳最早的第一帧图像。
步骤S330，通过所述第二线程对所述第一线程获取的所述当前帧图像进行图像检测，得到所述当前帧图像的检测结果；
这里，对当前帧图像进行图像检测，可以是采用任何已知图像检测识别技术对当前帧图像中的对象进行检测，确定出当前帧图像中的目标对象。该目标对象可以是人体、人脸、人手、其他动物等。
其中，检测识别技术例如包括但不限于：图像分割、关键点检测等技术。可以理解的是，本领域技术人员可以选择一种或多种检测识别技术，实现对当前帧图像中对象的识别，对此本公开实施例不做限制。
示例性的，第二线程通过第一线程调用执行，每当第一线程获取一帧图像后就将该帧图像发送给第二线程，在第二线程中对该帧图像进行图像检测得到检测结果，第二线程代码流程结束后再向第一线程返回当前帧图像的检测结果。
步骤S340，通过所述第一线程基于当前帧图像的检测结果，对当前帧图像进行渲染，生成所述当前帧图像的渲染特效画面。
这里，第一线程将预先设计的渲染逻辑叠加在第二线程返回的当前帧图像的检测结果上，得到当前帧图像的渲染结果并展示。
上述实施例中，对于同一帧图像，第一线程和第二线程分别执行图像渲染任务和图像检测任务，使得图像检测部分不再占用第一线程性能，可以得到较好的渲染帧率。但是，在各个线程执行每一步任务的过程中，另一个线程一直处于空闲状态。
如图3B所示，由于系统浏览器的限制，在第一线程301进行获取图像和图像渲染的过程，在第二线程302进行图像检测的过程。虚线框303中表示当前帧图像的任务处理流程，即第一线程301获取当前帧图像31并将当前帧图像传输给第二线程302，然后由第二线程302针对当前帧图像执行图像检测32。之后第二线程302返回当前帧图像的检测结果给第一线程301，由第一线程执行图像渲染33完成对当前帧图像的渲染。同样地，虚线框304表示下一帧图像的任务处理流程，即依次由第一线程301执行获取下一帧图像34、第二线程302针对下一帧图像执行图像检测35、第一线程执行图像渲染36完成对下一帧图像的整体流程。
应注意，在第一线程301基于第二线程302返回的当前帧图像的检测结果，对当前帧图像执行图像渲染33完成之后，第一线程301继续获取下一帧图像34，也就是说通过双线程(第一线程和第二线程)对于图像序列中的每一帧图像依次执行检测任务和渲染任务之后再进行下一帧图像的检测和渲染。
在本公开实施例中，对第一线程实时获取的当前帧图像，先通过第二线程执行图像检测任务，得到检测结果，再由第一线程基于检测结果执行当前帧图像的图像渲染任务，如此使得检测过程不再占用第一线程的性能，可以提高显示帧率及体验效果。
图4A为本公开实施例提供的图像检测与渲染方法的流程示意图，如图4A所示，所述方法可以通过以下步骤实现：
步骤S410，通过第一线程从图像序列中获取待检测帧图像；
这里，所述图像序列包括至少两帧图像。
步骤S420，在所述第一线程从所述图像序列中获取当前帧图像作为所述待检测帧图像的过程中，所述第二线程对所述当前帧图像的前一帧图像执行所述图像检测任务；
这里，首先第一线程从图像序列中获取当前帧图像的前一帧图像并传输给第二线程，然后由第二线程对该前一帧图像进行图像检测。在第二线程对前一帧图像进行图像检测的同时，第一线程同时准备好当前帧图像，以便第二线程返回前一帧图像的检测结果之后又有当前帧图像即新的画面数据待检测。
其中，在第二线程执行前一帧图像的检测任务的过程中，第一线程尚未得到第二线程返回的前一帧图像的检测结果，也就是说第一线程暂时没有待渲染的任务，可以利用空闲时段获取当前帧图像。
步骤S430，在所述第二线程对所述当前帧图像执行所述图像检测任务的过程中，通过所述第一线程将经过所述图像检测任务后的前一帧图像作为待渲染帧图像，对所述待渲染帧图像执行所述图像渲染任务。
这里，所述第二线程对当前帧图像进行渲染的过程中，说明第二线程已经完成对前一帧图像的图像检测任务，得到前一帧图像的检测结果，此时第一线程可以基于前一帧图像的检测结果，对前一帧图像进行渲染，得到前一帧图像的渲染图像。
在所述第二线程得到前一帧图像的检测结果之后，通过所述第二线程向所述第一线程返回所述前一帧图像的检测结果；也就是说，第二线程通过第一线程调用，在第二线程的代码流程执行完成后向第一线程返回前一帧图像的检测结果。
这里，第二线程向第一线程返回前一帧图像的检测结果之后，第二线程便处于空闲状态，此时第二线程可以直接对第一线程准备好的当前帧图像进行图像检测。
如图4B所示，由于系统浏览器的限制，在第一线程401进行获取图像和图像渲染的过程，在第二线程402进行图像检测的过程。虚线框403中表示当前帧图像的任务处理流程，即第一线程401获取当前帧图像41并传输给第二线程402，然后由第二线程402执行图像检测42，之后第二线程402返回当前帧图像的检测结果给第一线程401，由第一线程执行图像渲染43完成对当前帧图像的渲染。同样地，虚线框404表示下一帧图像的任务处理流程，即依次由第一线程401执行获取下一帧图像44、第二线程402执行图像检测45、第一线程执行图像渲染46完成对下一帧图像的整体流程。
其中，在第二线程402针对当前帧图像执行图像检测42的同时由第一线程401执行获取下一帧图像44；在第一线程401执行图像渲染43的过程中，第二线程也在针对下一帧图像执行图像检测45过程。这样，当前帧图像和下一帧图像作为相邻的两帧图像交叉并行执行图像检测任务和图像渲染任务，也就是说每两帧图像相互插帧处理，构成一组循环代码，直至获取第N帧图像47，并依次执行图像检测、图像渲染以完成对图像序列中每一帧图像的检测并渲染。
在本公开实施例中，对第一线程实时获取的当前帧图像，在通过第二线程执行图像检测任务的同时由第一线程获取下一帧图像，然后在第一线程基于检测结果对当前帧图像进行渲染的同时由第二线程执行下一帧图像的检测过程，如此通过利用插帧交替进行图像检测和图像渲染，第一线程和第二线程可以同时执行各自任务，节省等待时间，可以提高显示帧率及体验效果。
图5A为本公开实施例提供的图像检测与渲染方法的流程示意图，如图5A所示，所述方法可以通过以下步骤实现：
步骤S510，通过所述第一线程从所述图像序列中获取所述待检测帧图像放入图像检测队列中；
这里，所述图像检测队列用于缓存所述待检测帧图像。
步骤S520，通过所述第二线程对从所述图像检测队列中获取的第一图像执行所述图像检测任务；
这里，所述第一图像为所述图像检测队列中的首帧图像。也就是说，图像检测队列中保存有可以直接检测的画面，第二线程空闲时就可以从图像检测队列中获取一个检测任务，不需要等待第一线程渲染完上一帧图像后，再通过第一线程从图像序列中获取当前帧图像。
步骤S530，通过所述第一线程从图像渲染队列中获取第二图像作为所述待渲染帧图像，对所述待渲染帧图像执行所述图像渲染任务；
这里，所述图像渲染队列用于缓存已执行完所述图像检测任务的图像，所述图像渲染队列中的第二图像为完成所述图像检测任务的所述第一图像或所述第一图像的前一帧图像；也就是说，图像渲染队列中保存有可以直接渲染的画面和图像检测结果，第一线程空闲时例如将当前帧图像传输至图像检测队列之后就可以直接从图像渲染队列中获取一个渲染任务，不需要等待第二线程执行完检测过程得到检测结果。
其中，在第二线程正在对第一图像执行图像检测任务的过程中，图像渲染队列中缓存的渲染任务为第一图像的前一帧图像及其检测结果；在第二线程完成对第一图像的检测后将第一图像及其检测结果传输至图像渲染队列中，此时图像渲染队列中缓存的渲染任务为第一图像及其检测结果。
在一些可能的实施例中，所述图像渲染队列与所述图像检测队列为同一任务队列，且在同一时刻所述任务队列中保存有以下至少之一的图像：至多一帧所述待检测帧图像；至多一帧所述已执行完所述图像检测任务的图像。由于第一线程已经执行完第一图像的前一帧图像的渲染任务并移除了该前一帧图像，这样，将图像检测队列和图像渲染队列优化为同一任务队列，使得队列中包括最多一个检测任务和最多一个渲染任务，可以同时满足检测和渲染的需要，同时减少资源浪费，提高设备的运算能力。
如图5B所示，由于系统浏览器的限制，在第一线程501进行获取图像和图像渲染的过程，在第二线程502进行图像检测的过程。虚线框503和虚线框504表示循环执行的代码流程，其中在第一个循环中，在第一线程501对第一帧图像进行图像渲染54的过程中，由第二线程502对第二帧图像进行图像检测55；同时第一线程501完成图像渲染54后可直接获取第三帧图像56。在第二个循环中，在第一线程501对第二帧图像进行图像渲染57的过程中，由第二线程502对第三帧图像进行图像检测58；同时第一线程501完成图像渲染57后可直接获取第四帧图像59。
在实施中，获取首帧图像51是初始时候的代码流程起点，代表第一线程501第一次获取画面任务并作为待检测帧图像传输至图像检测队列(图中未示出)，此时第二线程502处于空闲状态，就直接把这个首帧图像传输给第二线程502进行图像检测53。在上述过程中由于第一线程501没有需要渲染的图像而处于空闲状态，因此执行获取第二帧图像52并将第二帧图像作为新的待检测帧图像传输至图像检测队列。之后第二线程中图像检测53执行结束，第二线程返回第一帧图像的检测结果并作为待渲染帧图像传输至图像渲染队列(图中未示出)；此时第二线程处于空闲状态，又可以从图像检测队列中获取第二帧图像进行图像检测55，同时通过第一线程501从图像渲染队列中获取待渲染帧图像并执行图像渲染54，完成第一帧图像的特效画面的显示。
在本公开实施例中，通过引入任务队列保存待检测帧图像和待渲染帧图像，可以使得在第二线程对当前帧图像进行图像检测的过程中，同时第一线程可以对检测完成的上一帧图像进行渲染。也就是通过引入第二线程和延迟一帧渲染的方式，能够并行执行检测和渲染任务，提升渲染帧率，满足用户体验。
图6为本公开实施例为提供的图像检测与渲染方法的流程示意图，如图6所示，所述方法至少包括以下步骤：
步骤S610，通过所述第一线程从所述图像序列中获取所述待检测帧图像放入图像检测队列中；
这里，所述图像检测队列用于缓存所述待检测帧图像。
步骤S620，在确定所述第二线程处于空闲状态的情况下，通过所述第一线程获取所述图像检测队列中首帧图像作为所述第一图像并发送给所述第二线程；
这里，所述第一图像为所述第一线程从所述图像序列中获取并传输至所述图像检测队列的首帧图像。
在一些可能的实现方式中，在所述第二线程完成对所述第一图像的所述图像检测任务并将所述第一图像传输到所述图像渲染队列的情况下，确定所述第二线程处于所述空闲状态；在另一些可能的实现方式中，所述当前帧图像为所述第一线程从所述图像序列中获取的首帧图像，在所述第一线程将首次从所述图像序列中获取的图像作为所述待检测帧图像传输至所述图像检测队列的情况下，确定所述第二线程处于所述空闲状态。这样，确定第二线程在不同情况下的空闲状态，以便及时将图像检测队列中的待检测帧图像传给第二线程进行图像检测，从而提高程序整体的检测渲染效率。
步骤S630，通过所述第二线程对所述第一图像进行图像检测，得到所述第一图像的检测结果；
这里，通过所述第二线程将所述待检测帧图像的检测结果作为新的渲染任务传输至所述图像渲染队列中。
步骤S640，通过所述第一线程从图像渲染队列中获取第二图像作为所述待渲染帧图像， 对所述待渲染帧图像执行所述图像渲染任务。
这里，所述图像渲染队列用于缓存已执行完所述图像检测任务的图像，所述图像渲染队列中的第二图像为完成所述图像检测任务的所述第一图像或所述第一图像的前一帧图像。
在实施中，第一线程基于待渲染帧图像(第一图像或其前一帧图像)的检测结果，对待渲染帧图像进行渲染。
本公开实施例在任务流程处理中，在检测到第二线程处于空闲状态的情况下，就将图像检测队列中的待检测帧图像传给第二线程进行图像检测，从而提高程序整体的检测渲染效率。
图7为本公开实施例为提供的图像检测与渲染方法的流程示意图，如图7所示，所述方法至少包括以下步骤：
步骤S710，通过所述第一线程从所述图像序列中获取所述待检测帧图像放入图像检测队列中；
这里，所述图像检测队列用于缓存所述待检测帧图像。
步骤S720，通过所述第二线程对从所述图像检测队列中获取的第一图像执行所述图像检测任务；
这里，所述第一图像为所述图像检测队列中的首帧图像。
其中，第二线程对所述第一图像进行图像检测，得到第一图像的检测结果，并将执行完检测任务的第一图像作为渲染任务传输到图像渲染队列中。
步骤S730，通过所述第一线程从图像渲染队列中获取第二图像作为所述待渲染帧图像，对所述待渲染帧图像执行所述图像渲染任务；
这里，所述图像渲染队列用于缓存已执行完所述图像检测任务的图像，所述图像渲染队列中的第二图像为完成所述图像检测任务的所述第一图像或所述第一图像的前一帧图像。
步骤S740，在所述图像检测队列中不存在所述待检测帧图像且所述第一线程为空闲状态的情况下，通过所述第一线程从所述图像序列获取一帧图像作为新的所述待检测帧图像传输至所述图像检测队列中。
这里，可以先确定图像检测队列不存在待检测帧图像再确定第一线程为空闲状态，也可以先确定第一线程为空闲状态再确定图像检测队列不存在待检测帧图像，本公开实施例对此不做限定。
在一些可能的实现方式中，在所述第一线程完成对所述待渲染帧图像的渲染之后，确定所述第一线程为空闲状态；在另一些可能的实现方式中，在所述第二线程对所述第一线程首次从所述图像序列中获取的图像进行图像检测的过程中，确定所述第一线程为空闲状态。这样，确定第一线程在不同情况下的空闲状态，以便更好地控制画面任务流向，使整体方案的检测渲染过程高效有序地执行。
在本公开实施例中，如果图像检测队列中没有待检测帧图像即图像检测任务不足且第一线程空闲，则第一线程从图像序列中获取下一帧图像即新的图像检测任务存入图像检测队列备用，如此能尽可能提高任务占用的密集程度，从而有效提高程序检测渲染效率。
应注意，本说明书中的各个实施例均采用递进的方式描述，每个实施例重点说明的都是与其他实施例的不同之处，各个实施例之间相同相似的部分可互相参见。
下面结合一个具体实施例对上述图像检测与渲染方法进行说明，然而值得注意的是，该具体实施例是为了更好地说明本公开，并不构成对本公开的不当限定。
本公开实施例以人脸检测渲染流程，且图像渲染队列与图像检测队列为同一任务队列为例，提出一种基于Web Worker的双线程异步人脸检测与渲染方法，通过引入双线程和延迟渲染方案，能够并行检测和渲染，提升渲染帧率，满足用户体验。
步骤S801，确定任务队列，用来保存待检测帧图像和待渲染帧图像；
这里，待渲染帧图像为已经检测完成并获得人脸检测结果的图像。
步骤S802，在每一帧的循环中，先检测人脸检测模块是否闲置；如果闲置，则在任务队列中获取一帧待检测帧图像送去第二线程进行检测；
步骤S803，第一线程查看任务队列中是否有待渲染帧图像进行渲染；
这里，在第一线程完成对待渲染帧图像的渲染过程之后，从任务队列中移除该待渲染帧图像。
步骤S804，渲染结束后，如果任务队列中无待检测帧图像，则从终端摄像头中获取一帧新的画面作为待检测帧图像，并入任务队列中备用。
重复上述步骤S802至步骤S804即可循环检测渲染人脸特效。
按如上步骤S802至步骤S804的逻辑，任务队列中应该保持至多2个任务(一帧待渲染帧图像即检测完成待渲染的任务、一帧待检测帧图像即尚未检测或正在检测的任务)，从而第一线程不再需要人脸检测任务的同时尽可能提高任务占用的密集程度，可以有效提高程序检测渲染效率。
本公开实施例通过引入第二线程进行人脸检测任务，提高整体检测渲染效率。同时通过任务队列的使用，使得第二线程在针对第N帧图像进行检测时，第一线程可以同时针对第N-1帧图像的检测结果进行渲染，生成第N-1帧图像的渲染图像。这样延迟一帧渲染的方案，能够并行检测和渲染，提升渲染帧率，满足用户体验。
需要说明的是，上文对各个实施例的描述倾向于强调各个实施例之间的不同之处，其相同或相似之处可以互相参考。
基于前述的实施例，本公开实施例再提供一种图像检测与渲染装置，所述装置包括所包括的各模块、以及各模块所包括各单元，可以通过电子设备中的处理器来实现；当然也可通过具体的逻辑电路实现；在实施的过程中，处理器可以为中央处理器(Central Processing Unit，CPU)、微处理器(Micro Processing Unit，MPU)、数字信号处理器(Digital Signal Processor，DSP)或现场可编程门阵列(Field Programmable Gate Array，FPGA)等。
图8为本公开实施例提供的图像检测与渲染装置的组成结构示意图，如图8所示，所述装置800包括第一获取模块810、第一执行模块820和第二执行模块830，其中：
所述第一获取模块810，配置为通过第一线程从图像序列中获取待检测帧图像；
所述第一执行模块820，配置为通过第二线程对所述待检测帧图像执行图像检测任务；其中，所述图像检测任务用于获得所述待检测帧图像的检测结果；
所述第二执行模块830，配置为通过所述第一线程从经过所述第二线程执行图像检测任务后的图像中确定待渲染帧图像，对所述待渲染帧图像执行图像渲染任务；其中，所述图像渲染任务用于基于所述待渲染帧图像的检测结果，对所述待渲染帧图像进行渲染，得到所述待渲染帧图像的特效画面。
在一些可能的实施例中，第一获取模块、第一执行模块和第二执行模块在对所述图像序列中当前帧图像执行后，第一获取模块、第一执行模块和第二执行模块再对下一帧图像执行；其中，所述待检测帧图像和待渲染帧图像均为所述当前帧图像。
在一些可能的实施例中，所述第一执行模块820还配置为在所述第一线程从所述图像序列中获取当前帧图像作为所述待检测帧图像的过程中，通过所述第二线程对所述当前帧图像的前一帧图像执行所述图像检测任务；所述第二执行模块830还配置为在所述第二线程对所述当前帧图像执行所述图像检测任务的过程中，通过所述第一线程将经过所述图像检测任务后的前一帧图像作为所述待渲染帧图像，对所述待渲染帧图像执行所述图像渲染任务。
在一些可能的实施例中，所述第一获取模块810还配置为通过所述第一线程从所述图像序列中获取所述待检测帧图像放入图像检测队列中；其中，所述图像检测队列用于缓存所述待检测帧图像；所述第一执行模块820，还配置为通过所述第二线程对从所述图像检测队列中获取的第一图像执行所述图像检测任务；所述第一图像为所述图像检测队列中的首帧图像；所述第二执行模块830，还配置为通过所述第一线程从图像渲染队列中获取第二图像作为所述待渲染帧图像，对所述待渲染帧图像执行所述图像渲染任务；所述图像渲染队列用于缓存已执行完所述图像检测任务的图像，所述图像渲染队列中的第二图像为完成所述图像检测任务的所述第一图像或所述第一图像的前一帧图像。
在一些可能的实施例中，所述图像检测队列与所述图像渲染队列为同一任务队列，且在 同一时刻所述任务队列中保存有以下至少之一的图像：至多一帧所述待检测帧图像；至多一帧所述已执行完所述图像检测任务的图像。
在一些可能的实施例中，所述装置还包括第二获取模块，配置为在所述图像检测队列中不存在所述待检测帧图像且所述第一线程为空闲状态的情况下，通过所述第一线程从所述图像序列获取一帧图像作为新的待检测帧图像传输至所述图像检测队列中。
在一些可能的实施例中，所述装置还包括第一确定模块，配置为在所述第一线程完成对所述待渲染帧图像的渲染之后，确定所述第一线程为空闲状态；或者，在所述第二线程从所述图像检测队列中获取所述首帧图像进行图像检测的过程中，确定所述第一线程为空闲状态。
在一些可能的实施例中，所述第一获取模块包括发送单元和检测单元，其中：所述发送单元，配置为在确定所述第二线程处于空闲状态的情况下，通过所述第一线程获取所述图像检测队列中首帧图像作为所述第一图像并发送给所述第二线程；所述检测单元，配置为通过所述第二线程对所述第一图像进行图像检测，得到所述第一图像的检测结果。
在一些可能的实施例中，所述装置还包括第二确定模块，配置为在所述第二线程完成对所述第一图像的所述图像检测任务并将所述第一图像传输到所述图像渲染队列的情况下，确定所述第二线程处于所述空闲状态；或者，在所述第一线程将首次从所述图像序列中获取的图像作为所述待检测帧图像传输至所述图像检测队列的情况下，确定所述第二线程处于所述空闲状态。
这里需要指出的是：以上装置实施例的描述，与上述方法实施例的描述是类似的，具有同方法实施例相似的有益效果。对于本公开装置实施例中未披露的技术细节，请参照本公开方法实施例的描述而理解。
需要说明的是，本公开实施例中，如果以软件功能模块的形式实现上述图像检测与渲染方法，并作为独立的产品销售或使用时，也可以存储在一个计算机可读取存储介质中。基于这样的理解，本公开实施例的技术方案本质上或者说对相关技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得电子设备(可以是具有摄像头的智能手机、平板电脑等)执行本公开各个实施例所述方法的全部或部分。而前述的存储介质包括：U盘、移动硬盘、只读存储器(Read Only Memory，ROM)、磁碟或者光盘等各种可以存储程序代码的介质。这样，本公开实施例不限制于任何特定的硬件和软件结合。
对应地，本公开实施例提供一种计算机可读存储介质，其上存储有计算机程序，该计算机程序被处理器执行时实现上述任一实施例中所述图像检测与渲染方法中的步骤。
对应地，本公开实施例中，还提供了一种芯片，所述芯片包括以下至少之一：可编程逻辑电路；程序指令，当所述芯片运行时，用于实现上述任一实施例中所述图像检测与渲染方法中的步骤。
对应地，本公开实施例中，还提供了一种计算机程序产品，当该计算机程序产品被电子设备的处理器执行时，其用于实现上述任一实施例中所述图像检测与渲染方法中的步骤。
基于同一技术构思，本公开实施例提供一种电子设备，用于实施上述方法实施例记载的图像检测与渲染方法。图9为本公开实施例提供的电子设备的硬件实体示意图，如图9所示，所述电子设备900包括存储器910和处理器920，所述存储器910存储有可在处理器920上运行的计算机程序，所述处理器920执行所述程序时实现本公开实施例任一所述图像检测与渲染方法中的步骤。
存储器910配置为存储由处理器920可执行的指令和应用，还可以缓存待处理器920以及电子设备中各模块待处理或已经处理的数据(例如，图像数据、音频数据、语音通信数据和视频通信数据)，可以通过闪存(FLASH)或随机访问存储器(Random Access Memory，RAM)实现。
处理器920执行程序时实现上述任一项的图像检测与渲染方法的步骤。处理器920通常控制电子设备900的总体操作。
上述处理器可以为特定用途集成电路(Application Specific Integrated Circuit，ASIC)、数字信号处理器(Digital Signal Processor，DSP)、数字信号处理装置(Digital Signal Processing Device，DSPD)、可编程逻辑装置(Programmable Logic Device，PLD)、现场可编程门阵列(Field Programmable Gate Array，FPGA)、中央处理器(Central Processing Unit，CPU)、控制器、微控制器、微处理器中的至少一种。可以理解地，实现上述处理器功能的电子器件还可以为其它，本公开实施例不作具体限定。
上述计算机存储介质/存储器可以是只读存储器(Read Only Memory，ROM)、可编程只读存储器(Programmable Read-Only Memory，PROM)、可擦除可编程只读存储器(Erasable Programmable Read-Only Memory，EPROM)、电可擦除可编程只读存储器(Electrically Erasable Programmable Read-Only Memory，EEPROM)、磁性随机存取存储器(Ferromagnetic Random Access Memory，FRAM)、快闪存储器(Flash Memory)、磁表面存储器、光盘、或只读光盘(Compact Disc Read-Only Memory，CD-ROM)等存储器；也可以是包括上述存储器之一或任意组合的各种电子设备，如移动电话、计算机、平板设备、个人数字助理等。
这里需要指出的是：以上存储介质和设备实施例的描述，与上述方法实施例的描述是类似的，具有同方法实施例相似的有益效果。对于本公开存储介质和设备实施例中未披露的技术细节，请参照本公开方法实施例的描述而理解。
应理解，说明书通篇中提到的“一个实施例”或“一实施例”意味着与实施例有关的特定特征、结构或特性包括在本公开的至少一个实施例中。因此，在整个说明书各处出现的“在一个实施例中”或“在一实施例中”未必一定指相同的实施例。此外，这些特定的特征、结构或特性可以任意适合的方式结合在一个或多个实施例中。应理解，在本公开的各种实施例中，上述各过程的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本公开实施例的实施过程构成任何限定。上述本公开实施例序号仅仅为了描述，不代表实施例的优劣。
需要说明的是，在本文中，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者装置不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者装置所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括该要素的过程、方法、物品或者装置中还存在另外的相同要素。
在本公开所提供的几个实施例中，应该理解到，所揭露的设备和方法，可以通过其它的方式实现。以上所描述的设备实施例是示意性的，例如，所述单元的划分，为一种逻辑功能划分，实际实现时可以有另外的划分方式，如：多个单元或组件可以结合，或可以集成到另一个系统，或一些特征可以忽略，或不执行。另外，所显示或讨论的各组成部分相互之间的耦合、或直接耦合、或通信连接可以是通过一些接口，设备或单元的间接耦合或通信连接，可以是电性的、机械的或其它形式的。
上述作为分离部件说明的单元可以是、或也可以不是物理上分开的，作为单元显示的部件可以是、或也可以不是物理单元；既可以位于一个地方，也可以分布到多个网络单元上；可以根据实际的需要选择其中的部分或全部单元来实现本公开实施例方案的目的。
另外，在本公开各实施例中的各功能单元可以全部集成在一个处理单元中，也可以是各单元分别单独作为一个单元，也可以两个或两个以上单元集成在一个单元中；上述集成的单元既可以采用硬件的形式实现，也可以采用硬件加软件功能单元的形式实现。
或者，本公开上述集成的单元如果以软件功能模块的形式实现并作为独立的产品销售或使用时，也可以存储在一个计算机可读取存储介质中。基于这样的理解，本公开实施例的技术方案本质上或者说对相关技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得设备自动测试线执行本公开各个实施例所述方法的全部或部分。而前述的存储介质包括：移动存储设备、ROM、磁碟或者光盘等各种可以存储程序代码的介质。
本公开所提供的几个方法实施例中所揭露的方法，在不冲突的情况下可以任意组合，得 到新的方法实施例。
若本公开技术方案涉及个人信息，应用本公开技术方案的产品在处理个人信息前，已明确告知个人信息处理规则，并取得个人自主同意。若本公开技术方案涉及敏感个人信息，应用本公开技术方案的产品在处理敏感个人信息前，已取得个人单独同意，并且同时满足“明示同意”的要求。例如，在摄像头等个人信息采集装置处，设置明确显著的标识告知已进入个人信息采集范围，将会对个人信息进行采集，若个人自愿进入采集范围即视为同意对其个人信息进行采集；或者在个人信息处理的装置上，利用明显的标识/信息告知个人信息处理规则的情况下，通过弹窗信息或请个人自行上传其个人信息等方式获得个人授权；其中，个人信息处理规则可包括个人信息处理者、个人信息处理目的、处理方式、处理的个人信息种类等信息。
本公开所提供的几个方法或设备实施例中所揭露的特征，在不冲突的情况下可以任意组合，得到新的方法实施例或设备实施例。
以上所述，仅为本公开的实施方式，但本公开的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本公开揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本公开的保护范围之内。因此，本公开的保护范围应以所述权利要求的保护范围为准。

Claims (13)

  1. 一种图像检测与渲染方法，包括以下步骤：
    通过第一线程从图像序列中获取待检测帧图像；
    通过第二线程对所述待检测帧图像执行图像检测任务；其中，所述图像检测任务用于获得所述待检测帧图像的检测结果；
    通过所述第一线程从经过所述第二线程执行图像检测任务后的图像中确定待渲染帧图像，对所述待渲染帧图像执行图像渲染任务；其中，所述图像渲染任务用于基于所述待渲染帧图像的检测结果，对所述待渲染帧图像进行渲染，得到所述待渲染帧图像的特效画面。
  2. 如权利要求1所述的方法，其中，按照所述图像序列中图像的先后顺序，在针对所述图像序列中当前帧图像执行完所述步骤后，再按照所述步骤执行下一帧图像；
    其中，所述待检测帧图像和待渲染帧图像均为所述当前帧图像。
  3. 如权利要求1所述的方法，其中，所述通过第二线程对所述待检测帧图像执行图像检测任务，包括：在所述第一线程从所述图像序列中获取当前帧图像作为所述待检测帧图像的过程中，通过所述第二线程对所述当前帧图像的前一帧图像执行所述图像检测任务；
    所述通过所述第一线程从经过所述第二线程执行图像检测任务后的图像中确定待渲染帧图像，对所述待渲染帧图像执行图像渲染任务，包括：在所述第二线程对所述当前帧图像执行所述图像检测任务的过程中，通过所述第一线程将经过所述图像检测任务后的前一帧图像作为所述待渲染帧图像，对所述待渲染帧图像执行所述图像渲染任务。
  4. 如权利要求1所述的方法，其中，所述通过第一线程从图像序列中获取待检测帧图像，包括：通过所述第一线程从所述图像序列中获取所述待检测帧图像放入图像检测队列中；其中，所述图像检测队列用于缓存所述待检测帧图像；
    所述通过第二线程对所述待检测帧图像执行图像检测任务，包括：通过所述第二线程对从所述图像检测队列中获取的第一图像执行所述图像检测任务；所述第一图像为所述图像检测队列中的首帧图像；
    所述通过所述第一线程从经过所述第二线程执行图像检测任务后的图像中确定待渲染帧图像，对所述待渲染帧图像执行图像渲染任务，包括：通过所述第一线程从图像渲染队列中获取第二图像作为所述待渲染帧图像，对所述待渲染帧图像执行所述图像渲染任务；所述图像渲染队列用于缓存已执行完所述图像检测任务的图像，所述图像渲染队列中的第二图像为完成所述图像检测任务的所述第一图像或所述第一图像的前一帧图像。
  5. 如权利要求4所述的方法，其中，所述图像检测队列与所述图像渲染队列为同一任务队列，且在同一时刻所述任务队列中保存有以下至少之一的图像：至多一帧所述待检测帧图像；至多一帧所述已执行完所述图像检测任务的图像。
  6. 如权利要求4所述的方法，其中，所述方法还包括：
    在所述图像检测队列中不存在所述待检测帧图像且所述第一线程为空闲状态的情况下，通过所述第一线程从所述图像序列获取一帧图像作为新的所述待检测帧图像传输至所述图像检测队列中。
  7. 如权利要求6所述的方法，其中，所述方法还包括：
    在所述第一线程完成对所述待渲染帧图像的渲染之后，确定所述第一线程为空闲状态；或者，
    在所述第二线程对所述第一线程首次从所述图像序列中获取的图像进行图像检测的过程中，确定所述第一线程为空闲状态。
  8. 如权利要求4至7任一项所述的方法，其中，所述通过所述第二线程对从所述图像检测队列中获取的第一图像执行图像检测任务，包括：
    在确定所述第二线程处于空闲状态的情况下，通过所述第一线程获取所述图像检测队列中首帧图像作为所述第一图像并发送给所述第二线程；
    通过所述第二线程对所述第一图像进行图像检测，得到所述第一图像的检测结果。
  9. 如权利要求8所述的方法，其中，所述方法还包括：
    在所述第二线程完成对所述第一图像的所述图像检测任务并将所述第一图像传输到所述图像渲染队列的情况下，确定所述第二线程处于所述空闲状态；或者，
    在所述第一线程将首次从所述图像序列中获取的图像作为所述待检测帧图像传输至所述图像检测队列的情况下，确定所述第二线程处于所述空闲状态。
  10. 一种图像检测与渲染装置，所述装置包括第一获取模块、第一执行模块和第二执行模块，其中：
    所述第一获取模块，配置为通过第一线程从图像序列中获取待检测帧图像；
    所述第一执行模块，配置为通过第二线程对所述待检测帧图像执行图像检测任务；其中，所述图像检测任务用于获得所述待检测帧图像的检测结果；
    所述第二执行模块，配置为通过所述第一线程从经过所述第二线程执行图像检测任务后的图像中确定待渲染帧图像，对所述待渲染帧图像执行图像渲染任务；其中，所述图像渲染任务用于基于所述待渲染帧图像的检测结果，对所述待渲染帧图像进行渲染，得到所述待渲染帧图像的特效画面。
  11. 一种电子设备，包括存储器和处理器，所述存储器存储有可在处理器上运行的计算机程序，所述处理器执行所述程序时实现权利要求1至9任一项所述方法中的步骤。
  12. 一种计算机可读存储介质，其上存储有计算机程序，该计算机程序被处理器执行时实现权利要求1至9中任一项所述方法中的步骤。
  13. 一种计算机程序产品，包括存储了程序代码的计算机可读存储介质，所述程序代码包括的指令被计算机设备的处理器运行时，实现权利要求1至9中任一项所述方法中的步骤。
PCT/CN2022/117294 2021-09-10 2022-09-06 图像检测与渲染方法及装置、设备、存储介质及计算机程序产品 Ceased WO2023036133A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111063569.9 2021-09-10
CN202111063569.9A CN113506298A (zh) 2021-09-10 2021-09-10 图像检测与渲染方法及装置、设备、存储介质

Publications (1)

Publication Number Publication Date
WO2023036133A1 true WO2023036133A1 (zh) 2023-03-16

Family

ID=78016760

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/117294 Ceased WO2023036133A1 (zh) 2021-09-10 2022-09-06 图像检测与渲染方法及装置、设备、存储介质及计算机程序产品

Country Status (2)

Country Link
CN (1) CN113506298A (zh)
WO (1) WO2023036133A1 (zh)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113506298A (zh) * 2021-09-10 2021-10-15 北京市商汤科技开发有限公司 图像检测与渲染方法及装置、设备、存储介质
CN114237739B (zh) * 2021-12-08 2024-02-02 广州讯飞易听说网络科技有限公司 应用程序的图像加载方法、计算机设备及存储介质
CN116459511A (zh) * 2022-01-11 2023-07-21 华为技术有限公司 特效优化方法、装置、存储介质及程序产品
CN115018970B (zh) * 2022-03-09 2025-08-12 奥比中光科技集团股份有限公司 一种图像渲染方法、装置、设备及计算机可读存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103399729A (zh) * 2013-06-28 2013-11-20 广州市动景计算机科技有限公司 一种HTML5 Canvas应用处理方法、装置及处理器
CN107993183A (zh) * 2017-11-24 2018-05-04 暴风集团股份有限公司 图像处理装置、方法、终端和服务器
US20180307454A1 (en) * 2017-04-19 2018-10-25 International Business Machines Corporation Automatic real-time configuration of a multi-head display system
CN112887510A (zh) * 2021-01-19 2021-06-01 三一重工股份有限公司 基于视频检测的视频播放方法及系统
CN113506298A (zh) * 2021-09-10 2021-10-15 北京市商汤科技开发有限公司 图像检测与渲染方法及装置、设备、存储介质

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105389159A (zh) * 2014-09-03 2016-03-09 优视科技有限公司 图片渲染方法、装置和移动终端
CN107203960B (zh) * 2016-06-30 2021-03-09 北京新媒传信科技有限公司 图像渲染方法及装置
US10957020B2 (en) * 2017-12-04 2021-03-23 Nvidia Corporation Systems and methods for frame time smoothing based on modified animation advancement and use of post render queues
CN110443880B (zh) * 2019-08-08 2023-05-16 Oppo广东移动通信有限公司 图像渲染方法、装置、存储介质及电子设备
CN113347488B (zh) * 2021-08-04 2021-11-19 腾讯科技（深圳）有限公司 视频渲染方法、装置、设备及存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103399729A (zh) * 2013-06-28 2013-11-20 广州市动景计算机科技有限公司 一种HTML5 Canvas应用处理方法、装置及处理器
US20180307454A1 (en) * 2017-04-19 2018-10-25 International Business Machines Corporation Automatic real-time configuration of a multi-head display system
CN107993183A (zh) * 2017-11-24 2018-05-04 暴风集团股份有限公司 图像处理装置、方法、终端和服务器
CN112887510A (zh) * 2021-01-19 2021-06-01 三一重工股份有限公司 基于视频检测的视频播放方法及系统
CN113506298A (zh) * 2021-09-10 2021-10-15 北京市商汤科技开发有限公司 图像检测与渲染方法及装置、设备、存储介质

Also Published As

Publication number Publication date
CN113506298A (zh) 2021-10-15

Similar Documents

Publication Publication Date Title
CN113453040B (zh) 短视频的生成方法、装置、相关设备及介质
WO2021031850A1 (zh) 图像处理的方法、装置、电子设备及存储介质
US20250088732A1 (en) Method for capturing image in video recording and electronic device
CN111881862B (zh) 手势识别方法及相关装置
CN113506298A (zh) 图像检测与渲染方法及装置、设备、存储介质
WO2021254502A1 (zh) 目标对象显示方法、装置及电子设备
CN113961278B (zh) 一种页面显示方法及相关设备
CN118555444B (zh) 视频生成方法及电子设备
WO2019128568A1 (zh) 内容推送方法、装置及设备
WO2014187108A1 (zh) 服务器、客户端及视频处理方法
CN116916151B (zh) 拍摄方法、电子设备和存储介质
US12292835B2 (en) Photographing method and related apparatus
WO2024179101A9 (zh) 一种拍摄方法
CN108965981A (zh) 视频播放方法、装置、存储介质及电子设备
WO2024179100A1 (zh) 一种拍摄方法
CN108513069A (zh) 图像处理方法、装置、存储介质及电子设备
CN110392306A (zh) 一种数据处理方法以及设备
CN115811638A (zh) 一种信息展示方法、装置、设备及存储介质
CN108259767A (zh) 图像处理方法、装置、存储介质及电子设备
CN115601842B (zh) 一种自动抓拍的方法、电子设备及存储介质
CN117156293B (zh) 拍照方法和相关装置
CN114926351B (zh) 图像处理方法、电子设备以及计算机存储介质
CN118555470A (zh) 一种拍照方法及电子设备
CN118555467A (zh) 一种拍照方法及电子设备
CN108495038B (zh) 图像处理方法、装置、存储介质及电子设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22866596

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22866596

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 22866596

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 23-09-2024)

122 Ep: pct application non-entry in european phase

Ref document number: 22866596

Country of ref document: EP

Kind code of ref document: A1