JP6214367B2 - Image composition apparatus and image composition program - Google Patents

Description

  The present invention relates to an image composition apparatus and an image composition program for performing image composition, display, and screen capture processing at high speed and efficiency.

  2. Description of the Related Art Conventionally, there is known an image composition display device that stores a plurality of input video data on a memory, reads the video data stored on the memory in accordance with display timing, and displays the data on a screen. For example, in the video synthesizing arithmetic processing apparatus described in Patent Document 1 below, a frame memory is prepared for each video signal to be synthesized, video data for one line is written into the frame memory by a write line buffer, and one line from the frame memory is written. The minute data is read out and image composition calculation processing is performed. Therefore, compared to the conventional case of synthesizing an image after one frame, there is less time delay in creating a synthesized video, and a video synthesis screen can be obtained in real time.

JP 2001-148806 A

  Since the conventional video composition processing apparatus is configured as described above, the video composition screen is not stored in the frame memory, and the video composition screen cannot be captured and stored.

  In addition, when a high-speed memory such as DDR2 SDRAM or DDR3 SDRAM is used for the frame memory, data read from the frame memory is delayed when there is a conflict between writing to the frame memory and a read request, etc., creating a composite image There is a problem that it is difficult to perform in accordance with the timing of the video composition screen display.

  An object of the present invention is to reduce a delay time from video image input to display by controlling a memory access timing to a frame buffer according to a processing load of a video composition process.

The image composition apparatus of the present invention
An image input unit that captures a plurality of input images and stores the plurality of input images in a storage area of a frame buffer;
An external image input unit that inputs an external image different from the plurality of input images captured by the image input unit;
An image composition unit for outputting load information indicating a load of image composition processing for compositing the plurality of input images stored in the frame buffer and the external image input by the external image input unit;
The load information output from the image composition unit is input, and the storage area of the frame buffer for reading the plurality of input images and the reading order of the input images are determined based on the input load information. The plurality of input images stored in the frame buffer are read out and stored in the first line buffer according to the determined storage area and the reading order, and are stored in the first line buffer. The plurality of input images stored in the buffer and the external image input by the external image input unit are combined by the image combining unit, and a combined image that is a combination result is sent to the image combining unit to the second line buffer. And a memory access control unit to be stored.

  According to this invention, the memory access control unit determines the storage area of the frame buffer from which a plurality of input images are read and the order in which the input images are read based on the load information. Therefore, it is possible to reduce the delay time from video image input to display.

FIG. 3 is a diagram of the first embodiment and is a configuration diagram of the image composition display device 101. FIG. 3 is a flowchart of processing performed by the image composition display device 101 in the first embodiment. FIG. 3 is a sequence diagram illustrating processing contents of the image composition display device 101 in the first embodiment. FIG. 3 is a diagram of the first embodiment, showing the display content of the final screen. FIG. 4 is a diagram of the first embodiment, and shows the image data reading range when the final screen is displayed. FIG. 3 is a diagram illustrating the method of using the frame buffer in the first embodiment. FIG. 5 is a diagram of the second embodiment and is a configuration diagram of the image composition display device 102. FIG. 10 is a diagram illustrating the hardware configuration of the image composition display devices 101 and 102 according to the third embodiment.

Embodiment 1 FIG.
FIG. 1 is a configuration diagram showing an image composition display device 101 (image composition device) according to the first embodiment. In the following embodiments, the terms “video” and “image” are used, but both have the same meaning. The image (video) includes both a still image and a moving image. The image (video) targeted by the image composition display device 101 may be a still image, a moving image, or both a still image and a moving image. The same applies to the image composition display device 102 according to the second embodiment. As shown in FIG. 1, the image composition display device 101 includes a video input unit 10, a synchronization signal generation unit 20, a video composition display unit 30, and a memory access control unit 40. The video composition display unit 30 includes a temporary storage line buffer 31 and a display line buffer 32.

(Video input unit 10)
The video input unit 10 (image input unit) starts operation in response to an image synthesis processing start instruction e from a CPU (not shown in FIG. 1), and takes in a plurality of input videos (input images) from a video camera or the like. Then, writing to the frame buffer 200 is performed. The video input unit 10 may hold a temporary buffer for several lines for temporarily storing image data.

(Synchronization signal generator 20)
The synchronization signal generation unit 20 (external image input unit) inputs an external video (external image) different from the video input to the video input unit 10, and generates a synchronization signal and video data in accordance with the timing of this input. And output to the video composition display unit 30. When no external video is input, a dot clock for display, a synchronization signal, and the like are generated inside the synchronization signal generation unit 20. In addition, the synchronization signal generation unit 20 outputs information related to display timing such as horizontal / vertical synchronization signals to the memory access control unit 40.

(Video composition display unit 30)
The video composition display unit 30 reads a plurality of image data stored in the frame buffer 200 by the video input unit 10 and temporarily stores the line buffer 31 for temporary storage, and a display for temporarily storing and outputting the composite screen. A line buffer 32. The video composition display unit 30 performs processing such as enlargement, reduction, rotation, and color conversion on a plurality of image data (external video) stored in the frame buffer 200, and the video input data from the synchronization signal generation unit 20. The display is performed by combining the two. In addition, the video composition display unit 30 has a function of writing back the data of the temporary storage line buffer 31 and the display line buffer 32 to the frame buffer 200 in accordance with a write instruction from the memory access control unit 40. In addition, the video composition display unit 30 outputs window display position information (load information c) to the memory access control unit 40.

(Memory access control unit 40)
Based on the display timing information output from the synchronization signal generation unit 20 and the window display position information (load information c) output from the video composition display unit 30, the memory access control unit 40 outputs the information to the frame buffer 200. The access area and access timing, and the writing timing from the video composition display unit 30 to the frame buffer 200 are controlled.

(1) For example, the memory access control unit 40 can use the display line buffer 32 for writing when the final display screen is captured. Thereby, compared with a system that stores the final display screen on the frame buffer 200 every time, it is possible to reduce the required memory bandwidth and the memory cost.
(2) Further, when screen capture is unnecessary, the memory access control unit 40 writes the video input unit 10, writes the video image stored in the frame buffer 200, and the drawing window of the frame buffer 200 (implementation). The necessary part of the form 2) is read out and displayed in combination. The final display screen is not stored in the frame buffer 200.
(3) Further, when the processing load on the video composition display unit 30 is small, such as when the input video is desired to be displayed as it is, the memory access control unit 40 reads the data read from the frame buffer 200. By transmitting to the display line buffer 32, the number of accesses to the frame buffer 200 can be reduced and the delay time from video input to display can be reduced. 2 and 3, the memory access control unit 40 causes the video composition display unit 30 to read a plurality of input images stored in the frame buffer 200 in S3 of FIG. Sometimes (S7-1 in FIG. 3), load information (FIG. 3) indicates whether a plurality of input images to be read by the video composition display unit 30 should be stored in the display line buffer 32 instead of the temporary storage line buffer 31. Based on the load information c). If the memory access control unit 40 determines from the load information that the load of the image composition processing of the video composition display unit 30 is small, the memory access control unit 40 determines that it should be stored in the display line buffer 32 and reads it to the video composition display unit 30. The displayed plurality of input images are stored in the display line buffer 32. That is, in S7-1 of FIG. 3, the data read from the frame buffer 200 is stored in the temporary storage line buffer 31, but is stored in the display line buffer 32 when the processing load is small as described above.
(4) On the other hand, when the processing load on the video composition display unit 30 is large, such as rotating the input video, the memory access control unit 40 displays a “temporary storage area” (see FIG. 6) on the frame buffer 200. (Described later) is secured, and the result of the video composition is written back to the frame buffer 200 only when necessary, thereby preventing the screen during the composition from being displayed. 2 and 3, the memory access control unit 40 performs temporary storage different from the storage area of the frame buffer 200 stored in the video input unit 10 in S3 of FIG. Whether or not the temporary storage area is reserved in the frame buffer 200 is determined based on the load information c. If the memory access control unit 40 determines from the load information c that the load of the image composition processing of the video composition display unit 30 is large, the memory access control unit 40 determines to secure a temporary storage area and is used by the video composition display unit 30. A temporary storage area (FIG. 6) is secured in the frame buffer 200. The memory access control unit 40 determines whether the load of the image composition processing performed by the video composition display unit 30 is large or small based on the load information c, but the memory access control unit 40 has a determination rule for load determination. The load information c output from the video composition display unit 30 is checked against this determination rule to determine the magnitude of the load.
(5) The “temporary storage screen area” on the frame buffer 200 is controlled so that it can be used preferentially from the display area where the processing load of the video composition display unit 30 is high in the final display screen. That is, the video composition display unit 30 causes the “temporary storage screen area” on the frame buffer 200 to be used in preference to the video composition by the video composition display unit 30. As a result, when a high-speed memory such as DDR2 SDRAM or DDR3 SDRAM is used for the frame buffer 200, even if a conflict occurs between a write to the frame buffer 200 and a read request, the “temporary storage screen area” is utilized. By (use), it is possible to prevent disturbance of the screen display due to the delay of the image composition process.

  In FIG. 1, the frame buffer 200 is external to the image composition display apparatus 101, but may be incorporated in the image composition display apparatus 101.

Next, the operation will be described.
FIG. 2 is a flowchart showing the processing contents of the image composition display device 101 according to the first embodiment. FIG. 3 is a sequence diagram of FIG.

  The image composition display device 101 starts operation by receiving an image composition processing start instruction e from a CPU (not shown) of a system in which the image composition display device 101 is mounted.

  Upon receiving an image synthesis processing start instruction e from the CPU, the video input unit 10 selects image data to be captured from input images (images include both still images and moving images), and stores them in the frame buffer 200. Store (step S1). Step S1 is repeated until all the images to be combined are captured (step S2).

Next, when the capturing of all the images to be synthesized is completed, the video input unit 10 notifies the memory access control unit 40 of the completion. On the other hand, the video composition display unit 30 outputs “information on the display position of the window” and “information on operation modes such as enlargement, reduction, rotation, and color conversion” to the memory access control unit 40. These “information on the display position of the window” and “information on the operation mode such as enlargement, reduction, rotation, and color conversion” output from the memory access control unit 40 are load information c indicating the processing load of the image composition processing. It is. The memory access control unit 40 receives load information c input (notified) from the video composition display unit 30, and uses the load information c to read out image data from the frame buffer 200 and a storage area of the frame buffer 200 The reading order of image data to be read is determined (step S3).
FIG. 4 shows an example of the display content of the final screen. The broken horizontal line in FIG. 4 is a convenient line indicating the display area and is not related to display.
FIG. 5 shows the image data reading range when the final screen is displayed. When it is desired to display three video windows as shown in FIG. 4, the display area is divided into display areas 1 to 7 in the vertical direction of the screen, and image data in the range shown in FIG. Read from the corresponding storage area of the frame buffer 200. As for the display area 1 and the display area 7, it is not necessary to read out image data. The video data to be displayed in each video window can be configured to display different video in the input video, or to display the same video data in a plurality of windows.

  Next, the memory access control unit 40 waits until the timing (display timing information b) of the display vertical synchronization signal output from the synchronization signal generation unit 20 in order to synchronize the video synthesis and display timing (step S4).

  Next, the memory access control unit 40 determines whether it is not necessary to read out image data from the frame buffer 200 in order from the first line of the display screen (step S5). Move (step S6) and repeat the process of step S5.

  When it is necessary to read out image data from the frame buffer 200, the memory access control unit 40 displays the video composition display from the storage area of the frame buffer 200 only in the display area necessary for display according to the reading order determined in step S3. The image data is read by the unit 30 and stored in the temporary storage line buffer 31 by the video composition display unit 30 (step S7).

  Next, the memory access control unit 40 causes the video composition display unit 30 to synthesize the data stored in the temporary storage line buffer 31 with the video data from the synchronization signal generation unit 20, and to produce a composite image that is a composite result. Is stored in the display line buffer 32 (step S8).

  Next, the memory access control unit 40 determines whether or not the final screen needs to be captured (step S9). If it is determined that the final screen is necessary, the memory access control unit 40 displays the data in the display line buffer 32 on the video composition display unit 30. Is written back to the frame buffer 200 (step S10). In this case, when the final screen capture is necessary, the memory access control unit 40 stops the writing of the video input unit 10 to prevent overwriting of the frame buffer 200 and stores it in the temporary storage line buffer 31 as described above. The composite result of the video and the external video input to the video input unit 10 is written back from the display line buffer 32 to the frame buffer 200 by the video composite display unit 30. If a bus (such as PCI Express) capable of transferring image data is separately connected to the CPU connected to the present apparatus, the process of writing back to the frame buffer 200 is omitted and directly performed. You may comprise so that the image data of a last screen may be transmitted.

  FIG. 6 is a diagram illustrating an example of a method of using the frame buffer 200 of the memory access control unit 40 when performing two-line video composition of the input video 1 and the input video 2 and final screen capture processing. At the time of image composition processing, as shown on the left side of FIG. 6, an area for saving input images 1 and 2 (double buffer configuration of A and B planes) and a temporary saving screen area (temporary saving storage area) are stored on the frame buffer 200. ) And the result of video composition halfway is written back to the frame buffer 200 only when necessary. In addition, during the screen capture process, in order to stop the writing of the video input unit 10 as described above, the image data for displaying the storage area of the input videos 1 and 2 as a single buffer configuration is stored, and the remaining area is stored on the final screen. By using it for capture, the frame buffer 200 can be effectively used.

  If it is not necessary to capture the final screen, the memory access control unit 40 determines whether the generation of the output image has been completed (step S11). If the generation of the output image has not been completed, the process moves to the next line, and the processes from step S5 to step S9 are repeated.

  If the generation of the output image has been completed, it is further determined whether or not image synthesis of the next frame is necessary (step S12). If necessary, the processing from step S1 to step S11 is repeated. If image composition is not necessary in the next video frame, the image composition is terminated.

  According to the first embodiment described above, the memory access control unit 40 determines the memory access timing from the temporary storage line buffer 31 and the display line buffer 32 of the video composition display unit 30 according to the processing load of the video composition processing. Control. Therefore, it is possible to realize high-speed screen capture processing, reduction of necessary memory bandwidth, reduction of memory cost, reduction of delay time from video input to display, and the like.

Embodiment 2. FIG.
In the first embodiment, the image composition display device 101 that combines and displays a plurality of input images from a video camera or the like has been described. In the second embodiment, the image composition display device 102 will be described. The image composition display device 102 is further provided with a drawing unit 50 with respect to the image composition display device 101, and the other points are the same as those of the image composition display device 101.

  The image composition display device 102 includes a drawing unit 50 that draws at least one of characters and graphics in addition to a plurality of input videos, and stores the drawn drawing data in the frame buffer 200. The drawing unit 50 draws the drawing data. Characters, graphics, etc. can be combined with video and displayed. In the image composition display device 102, the video composition display unit 30 generates a composite image using the drawing data.

  FIG. 7 is a configuration diagram showing the image composition display device 102 according to the second embodiment. In FIG. 7, the drawing unit 50 starts operating upon receiving a drawing start command from the CPU, and saves drawing results such as characters and graphics in a drawing window secured on the frame buffer 200. In the second embodiment, when the final screen capture is required, the memory access control unit 40 stops writing the video input unit 10 and the drawing unit 50 to prevent overwriting of the frame buffer 200, and temporarily stores the line buffer. It operates so as to write back the composite result of the composite video 31 and the external video input from the display line buffer 32 to the frame buffer 200.

  According to the second embodiment described above, since the drawing unit is provided, the same effects as those of the first embodiment can be obtained when combining characters and figures.

Embodiment 3 FIG.
The third embodiment will be described with reference to FIG.
FIG. 8 is a diagram illustrating a hardware configuration of the image composition display devices 101 and 102 which are computers described in the first and second embodiments.

  In FIG. 8, the image composition display device includes a CPU 810 (Central Processing Unit) for executing a program. The CPU 810 is connected to a ROM (Read Only Memory) 811, a RAM (Random Access Memory) 812 a, a buffer 812 b, a communication board 816, and a magnetic disk device 820 via a bus 825, and controls these hardware devices. Instead of the magnetic disk device 820, a storage device such as an optical disk device or a flash memory may be used.

  The RAM 812a, the buffer 812b, and the like are examples of a volatile memory. Storage media such as the ROM 811 and the magnetic disk device 820 are examples of nonvolatile memories. These are examples of a storage device, a storage unit, or a storage unit. Although only one buffer 812b is shown, the temporary storage line buffer 31, the display line buffer 32, the frame buffer 200, and the like. The communication board 816 is an example of an input device, and is also an example of an output unit and an output device.

  The magnetic disk device 820 stores an operating system 821 (OS), a program group 823, and a file group 824. The programs in the program group 823 are executed by the CPU 810 and the operating system 821.

  The program group 823 stores programs that execute the functions described as “˜units” in the description of the above embodiments. The program is read and executed by the CPU 810.

  In the description of the above embodiment, the file group 824 includes “to determination result”, “to calculation result”, “to extraction result”, “to generation result”, and “to processing result”. The described information, data, signal values, variable values, parameters, and the like are stored as items of “˜file” and “˜database”. The “˜file” and “˜database” are stored in a recording medium such as a disk or a memory. Information, data, signal values, variable values, and parameters stored in a storage medium such as a disk or memory are read out to the main memory or cache memory by the CPU 810 via a read / write circuit, and extracted, searched, referenced, compared, and calculated. Used for CPU operations such as calculation, processing, and output. Information, data, signal values, variable values, and parameters are temporarily stored in the main memory, cache memory, and buffer memory during the CPU operations of extraction, search, reference, comparison, calculation, calculation, processing, and output. .

  In addition, what has been described as “to part” in the description of the above embodiments may be “to means”, or may be “to step”, “to procedure”, and “to process”. Good. That is, what has been described as “˜unit” may be implemented by software alone, a combination of software and hardware, or a combination of firmware. The program is read by the CPU 810 and executed by the CPU 810. The program causes the computer to function as the “to part” described above. Alternatively, the computer executes the procedure and method of “to part” described above.

  In the above embodiment, the image composition display device has been described. However, it is understood from the above description that the operation of each “˜unit” in the image composition display device can be grasped as a program for causing the computer to function as the image composition display device. Of course. In addition, it is clear from the above description that the operations of each “˜unit” of the image composition display device can be grasped as a method.

  The image composition display device of the present invention can be applied to a control panel for consumer, industrial and in-vehicle devices. In particular, in a system employing an SOC (System-on-a-chip) in which a CPU and a graphics rendering engine are integrated on one chip, the video output from the SOC is connected to the external video input of the image composition display device of the present invention. Thus, it is possible to easily realize a display in which a camera image, characters, and the like are superimposed on the image output from the SOC.

  b display timing information, c load information, e image synthesis processing start command, 10 video input unit, 20 synchronization signal generation unit, 30 video synthesis display unit, 31 temporary storage line buffer, 32 display line buffer, 40 memory access control Part, 50 drawing part, 200 frame buffer, 101, 102 image composition display device.

Claims (6)

An image input unit that captures a plurality of input images and stores the plurality of input images in a storage area of a frame buffer;
An external image input unit that inputs an external image different from the plurality of input images captured by the image input unit;
An image composition unit for outputting load information indicating a load of image composition processing for compositing the plurality of input images stored in the frame buffer and the external image input by the external image input unit;
The load information output from the image composition unit is input, and the storage area of the frame buffer from which the plurality of input images are read and the reading order of the input images are determined based on the input load information. The plurality of input images stored in the frame buffer are read out and stored in the first line buffer according to the storage area and the reading order, and stored in the first line buffer. The plurality of stored input images and the external image input by the external image input unit are combined by the image combining unit, and the combined image which is a combination result is stored in the second line buffer by the image combining unit. An image composition apparatus comprising a memory access control unit. The memory access controller is
Based on the load information, whether to secure a temporary storage area for temporary storage different from the storage area to be stored by the image input unit, if determined to secure, 2. The image composition apparatus according to claim 1, wherein a temporary storage area used by the image composition unit is secured in the frame buffer. The memory access controller is
When the plurality of input images stored in the frame buffer are read by the image composition unit, the second input image is read by the image composition unit instead of the first line buffer. Whether to store in the line buffer is determined based on the load information, and when it is determined to store in the second line buffer, the plurality of input images read by the image composition unit is The image synthesizing apparatus according to claim 1, wherein the image synthesizing apparatus is stored in the second line buffer. The memory access controller is
It is determined whether or not to capture the final screen in the image composition processing, and when it is determined to capture, the image composition unit stores the final screen stored in the second line buffer in the frame buffer. The image synthesizing device according to claim 1, wherein The image composition device further includes:
A drawing unit that draws at least one of a character and a figure and stores the drawn drawing data in the frame buffer;
The image composition unit
The image composition apparatus according to claim 1, wherein the composite image is generated using the drawing data. In a computer having a first line buffer, a second line buffer, and a frame buffer,
Processing for capturing a plurality of input images and storing the plurality of input images in a storage area of the frame buffer;
A process of inputting an external image different from the plurality of input images;
A process of outputting load information indicating a load of an image combining process for combining the plurality of input images stored in the frame buffer and the input external image;
The storage area of the frame buffer for inputting the output load information and reading the plurality of input images and the reading order of the input images are determined based on the input load information, and the determined storage area and A process of reading the plurality of input images stored in the frame buffer and storing them in the first line buffer according to the reading order;
A process of combining the plurality of input images stored in the first line buffer and the input external image, and storing a combined image as a combination result in the second line buffer;
An image composition program for executing

Images