JP2019086725A - Picture processor - Google Patents

Abstract

To prevent generation of a missing display frame when pictures input from a plurality of picture iput interfaces are switched and are displayed.SOLUTION: The present invention includes: picture processing units 601, 603, and 605 for three picture inputs, respectively; interfaces 602, 604, and 606; a picture copying unit 613 for copying and displaying a picture to be input, in a display unit; five frame buffers 607, 608, 609, 610, and 611 for storing picture data of input pictures to use the picture data for display; a picture input Vsync monitoring unit for monitoring a Vsync output from the picture input unit; a display Vsync monitoring unit for monitoring the Vsync output from the display unit 612 for display; an input/output switching unit 614 for receiving the interface specification of an input picture from the picture copying unit 613; and a display picture timing adjusting unit. The frame buffer for picture input is switched when the Vsync for picture input is in a disable state and the Vsync for display is in an enable state.SELECTED DRAWING: Figure 6

Description

  The present invention relates to a video processing apparatus.

  Recently, a large number of devices and apparatuses having video input and output functions such as smartphones and drone have been proposed.

  These devices / devices are realized by software operating on a general-purpose non-real-time OS that anyone can use, utilizing the video input and output functions of the application processor used.

  An advantage of software configuration is that since video input devices connectable to a plurality of different interfaces possessed by the application processor can be used, various video input devices using these interfaces, for example, video input devices using an optical lens and the like By connecting an infrared video input device and a video input device for distance measurement using a laser, it becomes possible to input and output more various video images than a video using only a single video input device. There is. The specific configuration of these will be described.

  The mechanism for displaying the input video in the non-real time OS is generally configured as shown in FIG. That is, the image processing unit 102 processes the image data input from the imaging unit 101 into data that can be processed by a computer. In order to fetch the processed data, the video copying unit 105 operating on the CPU 106 receives the data from the video processing unit, and is stored as video data 104 on the memory 103 managed by the video copying unit. The video copying unit reads the stored video data, and passes 110 the video data to the video display control unit 107. The video display control unit causes the display device 108 to display the received video data.

  In contrast to such a configuration, it is general to adopt the configuration shown in FIG. 2 in a dedicated imaging device using a real time OS such as a camera. That is, video data input from the imaging unit 201 is processed by the video processing unit 202 into data that can be processed by a computer. The video copying unit 205 operating on the CPU 206 issues a video output instruction 209 to the video processing unit, and the video processing unit directly writes the video data 204 in the memory 203. The image copying unit issues an image input instruction 210 so as to read the image data output from the image processing unit to the image display control unit 207 while instructing the output. The video display control unit having received the video input instruction reads the video data in the memory and displays the video on the display unit 208.

  As described above, a dedicated imaging device such as a camera is configured to minimize the time until the input image is output. The method of configuring with a real-time OS is effective for a single video input device, but in the case of multiple inputs, it is necessary to have the same configuration for each video input device. On the other hand, the non-real-time OS has a configuration in which the video copying unit mediates video input and video output, so that a plurality of video input units can be easily provided. The configuration is shown in FIG.

  That is, the video of the video input 1 is processed by the video processing unit 301 into data processable by the computer, the video of the video input 2 is processed by the video processing unit 303 into data processable by the computer, and the video of the video input 3 is The video processing unit 305 processes the data into computer-processable data. A video copying unit 309 operating on the CPU 310 receives the video data of the video input 1 via the interface 302 of the video input 1 and stores the video data 307 in a memory 308 secured and managed by the video copying unit.

  Similarly, the video data of the video input 2 is received via the interface 304 of the video input 2 and the video data is stored in a memory secured and managed by the video copying unit, and the video data of the video input 3 is The video data is stored in a memory which is received via the network and secured and managed by the video copying unit.

  In order to display a video, the video copying unit passes video data 314 to be displayed to the video display control unit 311. The video display control unit 311 displays the received video on the display device 312. As described above, the non-real-time OS receives the different video inputs via different interfaces and displays them, regardless of the number and types of the interfaces, by using the same path from receiving the video data 313 to storing it in the memory 313. Makes it easy to receive various video inputs.

JP 2012-89158 A JP 2004-253922 A

  However, when trying to display input images from different interfaces individually from software on a non-real-time OS, it is difficult to output an image without missing frames to the input image using the same path. there were.

  FIG. 4 shows a memory transmission of general video input / output processing in the non-real time OS, and FIG. 5 shows a timing chart of Vsync of each video input and video display. In FIG. 4, when the video copying unit 417 tries to display video data processed into data that can be processed by a computer by the video processing unit 401 of the video input 1, the video copying unit starts displaying the video input 1 505. To receive the video data from the driver 403 of the interface 1. The video output from the video input 1 is continuously output by the Vsync 501 of the video input 1. Therefore, the video copying unit needs to store the received input video in the frame buffer 404 of the video input 1 secured in the memory 416 every time it is received.

  The video copy unit stores the video input 1 and allows the display driver 413 to display video on the display control and display unit 415, so that the frame buffer 414 for the display unit on the memory can be used as the video input 1 frame buffer. Copy the saved video data. The copied video data is read by the display driver so that it can be displayed at the timing of the display Vsync 504, and passed to the display control and display unit. By continuously performing such copying operation, the video of the video input 1 is displayed on the display device. At this time, the display Vsync is enabled to match the timing after copying the video data of the video input 1 Vsync, thereby maintaining the mutual timing relationship between the input and the output.

  When the video of the video input 2 and the video input 3 operates simultaneously with the start of the display of the video input 1, each video input is output at the timing of the Vsync 502 of the video input 2 and the Vsync 503 of the video input 3. At this time, Vsync of each input device will be different for each video input. In order to switch to the video input 2 when the video input 1 is displayed, the display switch 506 of the video input 2 is issued. Since it is necessary to perform writing to the display frame buffer after issuance, a timing is required not to rewrite the video data while the display frame buffer is being used. This is issued when the display Vsync is disabled. Do.

  In order to rewrite the frame buffer for the display unit to the video data of the video input 2, the display Vsync is disabled until the rewriting is completed, the rewriting is completed, and the display Vsync is operated to match the video input 2 Vsync. And the video of the video input 2 can be displayed. At this time, the video data of Scene 3 of the video input 1 and the video input 2 can not be displayed, and a continuous video display is not made, resulting in a missing video.

  A plurality of video input units, a display unit, a CPU, a plurality of interfaces for connecting the input device, a video copying unit for copying and displaying input video on the display unit, and video data of the input video, A memory capable of securing at least four or more frame buffers used for display, a video input Vsync monitor for monitoring Vsync output by the video input unit, and a display for monitoring Vsync output by the display for display Vsync monitoring unit, an input / output switching unit including an input request receiving unit that receives an interface specification of an input video from the video copying unit, a display video timing arbitration unit, and an input video Vsync to be switched and displayed is displayed without being displayed. And an input / output switching unit that changes the frame buffer for storing input video when Vsync is enabled. Provided.

  Even though different interfaces are used, sharing of frame buffers enables resource reduction. Since the operation of software can be minimized from the video input to the output, the output delay to the input can be minimized. By sharing, switching and using the input frame buffer and the display frame buffer, it is possible to minimize the delay from input to output similar to that of the real-time OS even when using a non-real-time OS.

Video display on non-real time OS. Video display of a dedicated imaging device using a real-time OS. Video display by multiple video input in non real time OS. Video transmission of video display in non real time OS. Video input and display timing chart. Video transmission of video display. Configuration of input / output video switching control unit. Vsync timing chart.

  FIG. 6 shows a video processing apparatus to which the present invention is applied. FIG. 7 shows an input / output video switching control unit. FIG. 8 shows a timing chart of video input and display Vsync according to the present invention. The invention will be described according to FIGS.

  In FIG. 6, 601 is a video processing unit for video input 1, 602 is an interface 1 for video input 1, 603 is a video processing unit for video input 2, 604 is an interface 2 for video input 2, 605 is a video input 3 Video processing unit 606: interface 3 of video input 3 607: video input / output combined frame buffer 1 608: video input / output combined frame buffer 2 609: video input / output combined frame buffer 3 610: video input / output combined The frame buffer 4 611 is a video input / output combined frame buffer 5 612 is an output control and display unit, 613 is a video copy unit, 614 is an input / output switching unit, 615 is a video input instruction, 616 is a frame buffer specification instruction, 617 is Video input Vsync monitoring, 618 is a display Vsync monitoring, and 619 is an input switching instruction.

  In FIG. 7, 701 is an input request reception unit, 702 is an output destination setting unit, 703 is a video input Vsync monitoring unit, 704 is a display frame buffer setting unit, 705 is a display Vsync monitoring unit, and 706 is a display video timing arbitration unit. It is. In FIG. 8, 801 is Vsync for video input 1, 802 is Vsync for video input 2, 803 is Vsync for video input 3, 804 is Vsync for display, 805 is a switching instruction for video input 1, and 806 is switching for video input 2. An instruction 807 is an instruction to switch the video input 3.

  The video copying unit 613 issues input switching instructions 619 and 805 to the input / output switching unit 614 so as to switch to the input from the corresponding interface when selecting the interface of the video input to be displayed. When the display is not displayed, the display Vsync 804 is always disabled, and therefore, an instruction is issued when both of the video input 1 Vsync 801 and the video input 1 are disabled.

  In the input / output switching unit 614 that has received the instruction, the input request receiving unit 701 receives the instruction, and the output destination setting unit 702 issues a video input instruction 615 to the video input 1, the video input 2 and the video input 3. The instructed video input 1, video input 2 and video input 3 output the video data output from each video processing unit to the input / output switching unit. The video switching unit stores the received video in a frame buffer according to the output.

  In order to display the video input 1, the video data of the video input 1 includes the video data of Scene 1 as a video input / output combined frame buffer 609, Scene 2 as a video input combined use frame buffer 610, and Scene 3 as a video input / output combined frame buffer 611 Control storage of video data input to be stored.

  The video data of the video input 2 is stored in the video input / output combined frame buffer 608 regardless of the scene, and the video data of the video input 3 is saved in the video input / output combined frame buffer 608 regardless of the scene. After the storage of the video data of the video input 1 is completed, the display frame buffer setting unit 704 of the input / output switching unit causes the output control / display unit 612 to perform the input / output combined frame buffer 609, the input / output combined frame buffer 610, and the input / output. A frame buffer designation instruction 616 is issued to sequentially read and display the dual purpose frame buffer 611 cyclically.

  The output control and display unit having received the instruction sequentially reads out and displays the input / output combined frame buffer 609, the input / output combined frame buffer 610, and the input / output combined frame buffer 611 according to the display Vsync 804.

  Next, the video copy unit issues a switching instruction 806 of the video input 2 to the input / output switching unit so as to display the video input 2. In the input / output switching unit that has received the instruction, the video input Vsync 802 of the video input 2 and the display Vsync are monitored by the video input Vsync monitoring unit of the input / output switching unit and the display Vsync monitoring unit. When Vsync 2 is disabled and Vsync for display is enabled, the video data of video input 1 is stored in the video input / output combined frame buffer 607, and the video data of video input 2 is video data of Scene 4 The storage of video data input is controlled so as to store the video input / output combined frame buffer 609, the Scene 5 as the video input / output combined frame buffer 610, and the scene 6 into the video input / output combined frame buffer 611.

  The display at this time is when Scene 3 is read out from the video input / output combined frame buffer 611. Even if Scene 4 of the video input 2 is stored in the video input / output combined frame buffer 609, the display is not affected. Next, when the display Vsync changes from disable to enable, and the video data is changed from the video input 1 to the video input 2 when the video input / output combined frame buffer 609 is read out, the display does not lose any frames. Display the video data of

  The same control is performed when switching to the video input 3. The video copying unit issues a switching instruction 807 of the video input 3 to the input / output switching unit. In the input / output switching unit that has received the instruction, the video input Vsync 803 of the video input 3 and the display Vsync are monitored by the video input Vsync monitoring unit and the display Vsync monitoring unit of the input / output switching unit. When Vsync is disabled and Vsync for display is enabled, the video data of video input 2 is stored in the video input / output combined frame buffer 608, and the video data of video input 3 is video data of Scene 6 The storage of video data input is controlled so that the video input / output combined frame buffer 611 and the Scene 7 are saved to the video input / output combined frame buffer 609 and the scene 8 to the video input / output combined frame buffer 610.

  The display at this time is when Scene 5 is read out from the video input / output combined frame buffer 610. Even if the scene 6 of the video input 3 is stored in the video input / output combined frame buffer 611, the display is not affected. Next, when the display Vsync changes from disable to enable, and the video data is changed from the video input 2 to the video input 3 when the video input / output combined frame buffer 610 is read out, the display is displayed without video loss 3 Display the video data of

  As described above, when displaying a plurality of video inputs using the video input / output mechanism of the non-real-time OS, a frame for saving the input video when the Vsync of the input video is disabled and the Vsync for display is enabled. By switching the buffer, it is possible to switch and display the displayed frame without omission.

  It should be noted that, in this embodiment based on the standard specification of the non-real-time OS, no special interface is necessary for the interface for performing such video input, so standard interfaces such as USB, PCIe, MIPI, etc. Can be used.

101 imaging unit 102 video processing unit 103 memory

Claims (9)

Multiple video input units,
A display unit,
CPU and
Multiple interfaces connecting input devices,
A video copying unit for copying and displaying an input video on a display unit;
A memory capable of securing at least four or more frame buffers used to store video data of input video and to use for display;
A video input Vsync monitoring unit that monitors Vsync output from the video input unit;
A display Vsync monitoring unit that monitors Vsync output by the display unit for display;
An input / output switching unit including an input request receiving unit for receiving an interface specification of an input video from the video copying unit;
Display video timing arbitration unit,
An input / output switching unit which changes the frame buffer for storing the input video when the input video Vsync to be switched and displayed is disabled and the display Vsync is enabled.   The video processing apparatus according to claim 1, wherein three of the frame buffers used for display are sequentially read out.   2. The video processing apparatus according to claim 1, wherein video data of an input video used for display sequentially overwrites and stores three of the frame buffers.   2. The video processing apparatus according to claim 1, wherein video data of an input video not used for display is stored in a frame buffer not used for display of the frame buffer.   The video processing apparatus according to claim 1, wherein the input video Vsync is in a disabled state before the video input is performed.   The image processing according to claim 1, wherein when the input video Vsync before the video input is made is in the disable state, the input / output switching unit performs the setting of the frame buffer even if the output Vsync is disabled. apparatus.   The image processing apparatus according to claim 1, wherein at least one of the interfaces is a USB.   The image processing apparatus according to claim 1, wherein at least one of the interfaces is PCIe.   The image processing apparatus according to claim 1, wherein at least one of the interfaces is MIPI.