TWI796075B - Image processing system and processing method of video stream - Google Patents

Abstract

An image processing system and a processing method of video stream are provided. The first image is obtained according to a parameter. The deformation adjustment procedure is performed on the first image, and the second image is generated. The identifying detection procedure is performed on the second image, and a detected result is generated. Control information is generated according to the detected result. The parameter is adjusted according to the control information, and a third image is generated. The second image and the third image are outputted. Therefore, the subsequent application could be enhanced through the dual image outputs.

Description

Image processing system and video stream processing method

The present invention relates to an image processing technology, and in particular to an image processing system and a video stream processing method.

In the conventional technology, although a camera equipped with a wide-angle lens or a fisheye lens can capture an image with a wider field of view (FoV), the edge of the image may be curved and form an unnatural appearance. Distortion in wide-angle or fisheye images can make content difficult to read and is more likely to cause discomfort to the user's eyes.

The known fisheye or wide-angle camera (camera) will send the panoramic image (panoramic image) to a back-end (back-end) system with strong computing power (such as a host or computer) for processing together Display of de-warping, detection and dewarped image. However, the complexity of the back-end device is high, and the application scenarios of the camera are limited.

In view of this, an embodiment of the present invention provides an image processing system and a video stream processing method, which can generate dual video streams for detection and display applications respectively.

The video stream processing method of the embodiment of the present invention includes (but not limited to) the following steps: obtaining the first image according to parameters. The deformation correction process is performed on the first image, and the second image is generated. The recognition and detection process is performed on the second image, and a detection result is generated. Generate control information according to the detection result. Adjust parameters according to the control information, and generate a third image. Outputting the second image and the third image.

The image processing system of the embodiment of the present invention includes (but not limited to) a controller. The controller is configured to obtain the first image according to the parameters, perform a deformation correction procedure on the first image to generate a second image, adjust the parameters according to the control information to generate a third image, and output the second image and the third image. The control information is generated according to the detection result generated by the recognition and detection process on the second image.

Based on the above, according to the image processing system and the video stream processing method of the embodiments of the present invention, the distortion correction program can generate two kinds of images for the two video streams respectively, and use them for different back-end applications. In this way, labor can be effectively divided and the flexibility of the structure can be improved.

In order to make the above-mentioned features and advantages of the present invention more comprehensible, the following specific embodiments are described in detail together with the accompanying drawings.

FIG. 1 is a block diagram of components of an image processing system 1 according to an embodiment of the present invention. Please refer to FIG. 1 , the image processing system 1 includes (but not limited to) an image capture device 10 , a controller 30 , a backend device 50 and a display device 60 .

The image capturing device 10 may be a camera, a video camera, a monitor or a device with similar functions. The image capture device 10 may include (but not limited to) a lens 11 and an image sensor 15 (for example, a Charge Coupled Device (CCD) or a Complementary Metal-Oxide-Semiconductor (CMOS), etc. ). In one embodiment, the image can be captured through the lens 11 and the image sensor 15 . For example, the light is imaged on the image sensor 15 through the lens 11 .

In some embodiments, the specifications of the image capture device 10 (for example, image capture aperture, magnification, focal length, image capture viewing angle, size of the image sensor 15 , etc.) and its quantity can be adjusted according to actual needs. For example, the lens 11 is a fisheye or wide-angle lens, and is used to generate a fisheye image or a wide-angle image.

The controller 30 can be coupled to the image capture device 10 through a camera interface, I2C and/or other transmission interfaces. The controller 30 includes (but not limited to) a memory 31 , transmission interfaces 32 , 33 , 34 and a computing unit 39 .

Memory 31 can be any type of fixed or removable random access memory (Radom Access Memory, RAM), read only memory (Read Only Memory, ROM), flash memory (flash memory), traditional hard disk (Hard Disk Drive, HDD), Solid-State Drive (Solid-State Drive, SSD) or similar components. In one embodiment, the memory 31 is used to store program codes, software modules, configurations, data or files.

The transmission interfaces 32, 33, 34 may be camera interfaces, I2C or other transmission interfaces. For example, the transmission interface 32 is an I2C interface, the transmission interface 33 is a Mobile Industry Processor Interface (MIPI), and the transmission interface 34 is a Universal Serial Bus (USB) interface. For another example, the transmission interface 33 is a Digital Parallel Bus (DPS) interface, a Low-Voltage Differential Signaling (LVDS) interface, and a High-Speed Serial Pixel (HiSPi) interface. Or V-by-One interface. In one embodiment, the transmission interfaces 32, 33, 34 are used to connect external devices. For example, the transmission interfaces 32 and 33 are connected to the backend device 50 , and the transmission interface 34 is connected to the display device 60 .

The computing unit 39 is coupled to the memory 31 and the transmission interfaces 32 , 33 , 34 . The computing unit 39 can be an image processor or a graphics processing unit (Graphic Processing unit, GPU), or other programmable general purpose or special purpose microprocessor (Microprocessor), digital signal processor (Digital Signal Processor, DSP) ), programmable controller, Field Programmable Gate Array (Field Programmable Gate Array, FPGA), application-specific integrated circuit (Application-Specific Integrated Circuit, ASIC) or other similar components or a combination of the above components. In one embodiment, the computing unit 39 is used to execute all or part of the operations of the controller 30 , and can load and execute various program codes, software modules, files and data stored in the memory 31 .

The backend device 50 and the display device 60 may be a desktop computer, a notebook computer, a server, a smart phone or a tablet computer. In one embodiment, the backend device 50 is used for image calculation and/or detection. In one embodiment, the display device 60 is used for displaying images. In some embodiments, the functions of the backend device 50 and/or the display device 60 may be implemented on a microcontroller, chip, SoC or system.

FIG. 2 is a block diagram of components of an image processing system 2 according to another embodiment of the present invention. Please refer to FIG. 2 , the difference between the controller 30' of this embodiment and the controller 30 of FIG. The transmission interface 35 can be a USB interface and provides endpoints 36 , 37 , 38 . It is worth mentioning that the endpoints 36, 37, and 38 of this embodiment are used to connect to the backend device 70, wherein the endpoints 36, 37 are also used to connect to the processor 71 of the backend device 70, and the endpoint The point 38 is used to connect the video unit 72 of the backend device 70 .

The backend device 70 can be a desktop computer, a notebook computer, a server, a smart phone or a tablet computer. The processor 71 can be an image processor or a GPU, or other programmable general-purpose or special-purpose microprocessors, digital signal processors (Digital, programmable controllers, field programmable logic gate arrays, special Using an integrated circuit or other similar components or a combination of the above components, the image unit 72 may be a processing circuit or a control circuit for image output/display/play.

In the following, the methods described in the embodiments of the present invention will be described in combination with various devices, components and modules in the image processing system 1 or the image processing system 2 . Each process of the method can be adjusted accordingly according to the implementation situation, and is not limited thereto.

FIG. 3 is a flowchart of a method for processing a video stream according to an embodiment of the invention. Referring to FIG. 3, the controllers 30, 30' obtain the first image according to the parameters (step S310). Parameters include setting parameters for de-wrapping, pan, tilt, zoom, rotation, shift and/or Field of View (FoV) adjustment . In one embodiment, the parameters are setting parameters used for the image capture operation of the image capture device 10 or other external image capture devices. For example, the controllers 30, 30' can send commands or messages to the image capture device 10 or other external image capture devices, and the commands or messages are related to setting parameters. For example, change the image capture range. The image capture device 10 or other external image capture devices can perform image capture operations according to the set parameters to obtain images.

In one embodiment, the controllers 30, 30' obtain the first image from the image capture device 10. Specifically, the first image is an image obtained by shooting one or more targets through the image capture device 10 or other external image capture devices. In one embodiment, the target is a human body. In some embodiments, the first image is for the upper body of the human (eg, waist, shoulders, or above the chest). In other embodiments, the targets may be various types of organisms or non-organisms. The controllers 30, 30' can obtain the first image captured by the image capture device 10 through the camera interface and/or I2C.

The controllers 30, 30' perform a deformation correction process on the first image and generate a second image (step S330). The distortion correction procedure is used to adjust the distortion in the first image. In one embodiment, the deformation correction procedure may be dewarping, panning, tilting, zooming, rotating, translating and/or viewing angle adjustment.

In one embodiment, the image sensor 15 can output raw data of the first image (for example, sensing intensities of a plurality of primary colors) to the controller 30, 30'. The controllers 30, 30' can process the original data according to the Image Signal Processing (ISP) program, and output the visible full-color image to the distortion correction program.

In another embodiment, the image sensor 15 can output lightness-chroma-density (YUV) or other color-coded data of the first image to the controller 30, 30'. The controllers 30, 30' can ignore or disable the processing of the ISP program on the color-coded data (ie, skip the ISP program for the color-coded data), and directly perform the deformation correction process on the color-coded data.

In one embodiment, the first image is a wide-angle image or a fisheye image, and the controllers 30, 30' can generate a dewarped panoramic image (ie, the second image) through a deformation correction procedure. This panorama can be used for subsequent detection of one or more targets. For example, panoramas can be used to detect faces, gestures or body parts. In another embodiment, the controllers 30, 30' can turn the angle of view of the first image on any axis, zoom in all or part of the area, translate in all or part of the area and/or adjust the size of the angle of view to generate the second image. Two images.

In one embodiment, the controller 30 can transmit the second image to the back-end device 50 through the transmission interface 33, and the controller 30′ can transmit the second image to the processor 71 of the back-end device 70 through the endpoint 37. For example, the controllers 30, 30' can output the second image through the first video stream VS1.

The back-end device 50 or the processor 71 performs a recognition detection program (or an object detection program) on the second image, and generates a detection result (step S330 ). The recognition detection procedure is, for example, determining one or more regions of interest (Region of Interest, RoI) (or fixed bounding box (bounding box), or rectangular frame (bounding rectangle)) or a representative point (pinot) (which may be located on the outline, center, or anywhere on the target) to identify the type of target (for example, human, male, or woman, dog or cat, table or chair, etc.).

In one embodiment, the detection results are related to one or more Regions of Interest (RoI) (or bounding boxes or bounding rectangles) in the second image. For example, the recognition detection program can determine the ROI corresponding to the object in the second image, and the ROI frames all or part of the object.

The backend device 50 or the processor 71 generates control information CI according to the detection result (step S340). The control information is related to the detection result of the second image. The detection result is, for example, identifying one or more human faces, gestures, body parts or other objects in the second image (ie, object detection), and for example, judging the position/movement of one or more objects ( i.e. Object Tracking). In one embodiment, the control information is based on the detection results and adjusted for the parameters of the deformation correction procedure. For example, the position of a human face in an image is used to set the angle of view steering in a warp correction procedure. For another example, the size of the human face in the image is used to set the area magnification in the deformation correction process.

In one embodiment, the backend device 50 can transmit the control information CI to the controller 30 through the transmission interface 32, and the processor 71 can transmit the control information CI to the controller 30′ through the endpoint 36.

The controllers 30, 30' adjust parameters according to the control information, and generate a third image (step S350). For example, the controllers 30, 30' generate commands or messages related to the setting parameters of the image capture device 10 according to the control information, and capture the first image again through the image capture device 10 accordingly. In one embodiment, the controllers 30, 30' obtain the re-captured first image, adjust the first image through a deformation correction procedure according to the control information CI, and generate a third image.

In one embodiment, the third image includes one or more windows. That is, the third image is divided into one or more windows. The controller 30, 30' can program one or more regions of interest into these windows. For example, the controller 30, 30' scales the region of interest in the panorama to fit the size of the specified window, and displaces the zoomed region of interest (by means such as panning, tilting, rotating, panning, and/or viewing angle adjustment) to this window.

For example, FIG. 4 is an example illustrating the multi-window preview. Referring to FIG. 4, the controllers 30, 30' divide the third image TIM (shown at the bottom of the figure) into three windows TW1, TW2, TW3. Wherein, window TW3 is captured from four characters in the first image SIM (shown at the top of the figure), and windows TW1 and TW2 are respectively captured from only one character in the first image SIM.

In one embodiment, the controllers 30, 30' can output the second image and the third image simultaneously or time-divisionally through the deformation correction process. For example, the controllers 30, 30' can generate the first third image according to the detection result of the first second image. Next, the controllers 30, 30' can output the second image and the third image (step S360). For example, the controllers 30, 30' sequentially generate the second image and the third image through time-division multiplexing. Furthermore, after the controller 30, 30' generates the first second image, it generates a third image based on the detection result of the second image. In addition, the controllers 30, 30' of the present invention can output the second image and the third image at the same time or time-divisionally, and there is no limitation on the output timing herein.

In one embodiment, the controllers 30, 30' can output the second image through the first video stream VS1, and output the third image through the second video stream VS2. Specifically, the second video stream is different from the first video stream. The first video stream VS1 carries the second image, and the second video stream VS2 carries the third image. That is to say, the controllers 30, 30' can provide outputs of dual video streams. In addition, depending on the computing speed, the two video streams can be output to the external device at the same time or not.

In one embodiment, taking the architecture of FIG. 1 as an example, the computing unit 39 outputs the first video stream VS1 through the transmission interface 33, outputs the second video stream VS2 through the transmission interface 34, and inputs the control information CI through the transmission interface 32. . For example, the MIPI outputs the first video stream VS1, an endpoint of the USB interface outputs the second video stream VS2, and the I2C interface inputs the control information CI.

In another embodiment, taking the structure of FIG. 2 as an example, the computing unit 39 outputs the first video stream VS1 through the terminal 37 of the transmission interface 35, outputs the second video stream VS2 through the terminal 38, and transmits the second video stream VS2 through the terminal 35. 36 Input control information CI. For example, two endpoints of the USB interface output the first video stream VS1 and the second video stream VS2, and the other endpoint inputs the control information CI.

The applications of the first video stream VS1 and the second video stream VS2 may be different. FIG. 5 is a flow chart of the application of dual video streams according to an embodiment of the present invention. Please refer to FIG. 1 and FIG. 5 together. In one embodiment, the first video stream VS1 is output to the back-end device 50, and the back-end device 50 performs an identification detection procedure on the first video stream VS1 (ie, for second image calculation and detection) (step S510) to generate a detection result. The recognition detection process may be the aforementioned object detection and/or tracking. In addition, the backend device 50 can generate the control information CI according to the detection result. It should be noted that, for the detection result and the control information CI, reference may be made to the description of steps S330 and S340 , which will not be repeated here. The backend device 50 can further feed back the control information CI to the controllers 30, 30', so that the controllers 30, 30' can generate the third image according to the control information CI. Similarly, in the implementation structure of FIG. 2, the processor 71 of the backend device 70 further feeds back the control information CI to the controller 30', so that the controller 30' can generate the third image according to the control information CI.

In one embodiment, the second video stream VS2 is output to the display device 60 , and the display device 60 displays the third image (ie, image preview) carried by the second video stream VS2 through a display program (step S550 ). Taking FIG. 4 as an example, the third image TIM is used for multi-window preview and display. It should be noted that the isochronous mode and the bulk mode provided by UVC (USB video device class or UVC) of the USB can be used for the display device 60 to display the third image in real time. However, the embodiment of the present invention can still adopt other protocols related to real-time video output.

To sum up, in the image processing system and video stream processing method of the embodiments of the present invention, dual video streaming is provided. A video stream can be used to detect faces, gestures or body parts, and control another video stream to provide dewarped multiple regions of interest image display .

Embodiments of the present invention provide a flexible framework and are widely applicable (for example, applicable to different types of video-related products). The system of the embodiment of the present invention can effectively divide labor. For example, a backend or computer system may be responsible for detection of faces, gestures or body parts, and a display device or system may be responsible for dewarping the image display of multiple regions of interest. In this way, general camera manufacturers can easily upgrade their products to fisheye or wide-angle cameras, and provide functions such as multi-target detection and tracking of human faces, gestures, or human body parts.

Although the present invention has been disclosed above with the embodiments, it is not intended to limit the present invention. Anyone with ordinary knowledge in the technical field may make some changes and modifications without departing from the spirit and scope of the present invention. The scope of protection of the present invention should be defined by the scope of the appended patent application.

1, 2: Image processing system 10: Image capture device 11: Lens 15: Image sensor 30, 30': Controller 31: memory 39: Processor 32, 33, 34, 35: transmission interface 36, 37, 38: endpoints 50, 70: back-end device 60: Display device 71: Processor 72: Image unit VS1: The first video stream VS2: Second video stream CI: Control Information S310~S360, S510~S550: steps TW1~TW3: Windows SIM: First Imaging TIM: Third Imaging

FIG. 1 is a block diagram of components of an image processing system according to an embodiment of the invention. FIG. 2 is a block diagram of components of an image processing system according to another embodiment of the invention. FIG. 3 is a flowchart of a method for processing a video stream according to an embodiment of the invention. FIG. 4 is an example illustrating multi-window preview. FIG. 5 is a flowchart of an application of dual video streams according to an embodiment of the present invention.

S310~S360: Steps

Claims (20)

A method for processing a video stream, comprising: obtaining a first image according to a parameter; performing a deformation correction procedure on the first image, and generating a second image; performing an identification and detection procedure on the second image, and generating a detection result; generating control information according to the detection result; adjusting the parameter according to the control information; obtaining the first image again according to the adjusted parameter; adjusting the first image through the deformation correction procedure according to the control information, And generate a third image; and output the second image and the third image. The video stream processing method as described in Claim 1, comprising: obtaining the first image from an image capture device; performing the deformation correction on the first image according to an image signal processing (Image Signal Processing, ISP) program program to generate the second image, wherein the deformation correction program is used to adjust the deformation in the first image; generate the third image through the deformation correction program according to the control information; and simultaneously output the second image and the first image Three images, wherein the second image is output through a first video stream, and the third image is output through a second video stream, wherein the second video stream is different from the first video stream. The video stream processing method as described in claim 2 further includes: simultaneously generating the second image and the third image through the distortion correction procedure. The video stream processing method as described in claim 2, further comprising: performing the identification detection procedure on the first video stream through a backend device to generate the detection result, wherein the detection result is related to At least one Region of Interest (RoI) in the second image. The video stream processing method as described in claim 4, further comprising: arranging the at least one region of interest into at least one window, wherein the third image includes the at least one window; and displaying the third image through a display device . The video stream processing method as described in Claim 2, wherein outputting the second image through the first video stream and outputting the third image through the second video stream include: through a mobile industry processor interface ( Mobile Industry Processor Interface (MIPI) outputs the first video stream, and outputs the second video stream through a Universal Serial Bus (USB). The video stream processing method as described in claim 2, wherein outputting the second image through the first video stream and outputting the third image through the second video stream include: through two terminals of a USB interface (endpoints) respectively output the first video stream and the second video stream. The video stream processing method as described in Claim 7 further includes: inputting the control information through another endpoint of the USB interface. The processing method of the video stream as described in Claim 2, wherein the deformation correction procedure includes de-wrapping, panning, tilting, zooming, rotation, translation At least one of (shift) and viewing angle (Field of View, FoV) adjustment, and the parameter includes setting parameters of at least one of anti-distortion, left and right rotation, tilt, scaling, rotation, translation, and viewing angle adjustment. The video stream processing method as described in claim 1, wherein the first image is a wide-angle image or a fisheye image, and the second image is a dewarped panorama. An image processing system, comprising: a controller configured to: obtain a first image according to a parameter; perform a deformation correction procedure on the first image and generate a second image; adjust the parameter according to a control information , wherein the control information is generated according to a detection result generated by performing a recognition and detection procedure on the second image; the first image is obtained again according to the adjusted parameters; and adjusted through the deformation correction procedure according to the control information the first image, and generate a third image; and output the second image and the third image. The image processing system as described in claim 11 further includes: an image capture device coupled to the controller, including a lens and an image sensor, and used to capture an image through the lens and the image sensor The first image, wherein the controller is further used to: Obtain the first image from an image capture device; perform a deformation correction process on the first image according to an image signal processing program to generate the second image, wherein the deformation correction process is used to adjust the image in the first image deformation; generate the third image through the deformation correction program according to the control information; and simultaneously output the second image and the third image, wherein the second image is output through a first video stream, and the second image is output through a second video stream and outputting the third image stream, wherein the second video stream is different from the first video stream. The image processing system as claimed in claim 12, wherein the controller is further configured to: simultaneously generate the second image and the third image through the deformation correction procedure. The image processing system as described in claim 12, further comprising: a back-end device, coupled to the controller, and used for performing the identification detection procedure on the first video stream to generate the detection result, wherein the The detection result is related to at least one ROI in the second image. The image processing system as claimed in claim 14, wherein the detection result is related to at least one region of interest in the second image, and the image processing system further includes: a display device coupled to the controller, wherein the The controller is further used to: arrange the at least one ROI into at least one window, wherein the third image includes the at least one window; and The third image is displayed through the display device. The image processing system as described in claim 12, wherein the controller includes: a mobile industry processor interface for outputting the first video stream; and a universal serial bus (USB) for outputting the second video streaming. The image processing system as claimed in claim 12, wherein the controller includes: a USB interface including two endpoints and used to output the first video stream and the second video stream respectively. The image processing system as claimed in claim 17, wherein the USB interface further includes: another endpoint for inputting the control information. The image processing system according to claim 12, wherein the deformation correction program includes at least one of anti-distortion, left-right rotation, tilt, scaling, rotation, translation, and viewing angle adjustment, and the parameters include anti-distortion, left-right rotation, and tilt , scaling, rotation, translation and viewing angle adjustment at least one of the setting parameters. The image processing system as claimed in claim 11, wherein the first image is a wide-angle image or a fisheye image, and the second image is a dewarped panorama.

Images