TWI815759B - Electronic system and image aggregation method thereof - Google Patents

Abstract

The application provides an electronic system and an image aggregation method thereof. The electronic system includes a plurality of image pickup devices and an image aggregation device coupled to the image pickup devices. The image aggregation device controls trigger of the image pickup devices. When the image pickup devices are triggered, the image pickup devices pickup a plurality of images and send to the image aggregation device. The image aggregation device flexibly selects the images pickup by the image pickup devices to flexibly aggregate into an aggregated image.

Description

Electronic system and image merging method

The invention relates to an electronic system and an image merging method thereof.

For security reasons, surveillance systems are currently installed in many places (such as buildings, streets, etc.). Figures 1A and 1B show two traditional surveillance systems. As shown in Figure 1A, the image capture devices 1_0~1_N (N is a positive integer) are coupled to the central server 2, and the central server 2 is coupled to the clients 3_0~3_M (M is a positive integer). As shown in Figure 1B, the image capture devices 4_0~4_N are coupled to the system-on-chip (SOC) 5_0~5_N, and the system SOC 5_0~5_N and the client 8_0~8_M are connected through the network 6 And connect to central server 7.

Traditional surveillance systems require the installation of a central server. As shown in Figures 1A and 1B, the image capture devices 1_0~1_N/4_0~4_N of the traditional surveillance system can capture image data, compress the image data and store it in the hard drive of the central server 2/7. The application software of client 3_0~3_M/8_0~8_M can grab image data from the hard drive of central server 2/7, decompress it and play it. Therefore, the central server 2/7 of the traditional surveillance system requires high computing power and a hard drive with large storage capacity, which is expensive.

In addition, the conventional surveillance system cannot perform "automatic selection/manual selection/flexible selection" of images captured by multiple image capture devices to merge the selected images into one image and output it to the client. The situation is more restricted.

In addition, the conventional surveillance system is limited in expanding the number of image capture devices due to the number of channels of the central server. Therefore, it is difficult to expand the number of image capture devices and cannot easily and flexibly expand the number of image capture devices. and/or change the placement.

In addition, head-mounted devices with virtual reality (VR)/augmented reality (AR) functions have gradually attracted people's attention. Head-mounted devices with conventional AR/VR functions can capture external images and display them on the screen for the user to view.

However, conventional AR/VR head-mounted devices cannot improve the effective frame rate, so their object tracking functions and/or dynamic trajectory detection functions still need to be improved.

In addition, conventional AR/VR head-mounted devices cannot reference dynamic trajectory information, so they are less able to efficiently detect object movement paths, and cannot predict the direction of head or object movement based on the tracked trajectories. The image from this direction cannot be captured in advance for display. Therefore, when a user moves his or her head quickly to track a moving object, a user using a conventional AR/VR head-mounted device is likely to encounter a dizzy problem due to persistence of vision and rapid head swing.

The invention relates to an electronic system and an image merging method thereof, which can achieve flexible image merging with multiple inputs and multiple outputs.

According to one aspect of this case, an electronic system is proposed, including: a plurality of image capture devices; and an image merging device coupled to the image capture devices. The image merging device controls the triggering of the image capture devices. When When the image capturing devices are triggered, the image capturing devices capture multiple images and send them to the image merging device. The image merging device flexibly selects the images captured by the image capturing devices to Flexible reorganization into a merged image.

In one embodiment of this case, the image merging device flexibly reuses the images captured by the image capturing devices according to the size of an internal buffer.

In one embodiment of this case, the image merging device includes: a trigger control unit coupled to the image capture devices to control the triggering of the image capture devices; an interface circuit coupled to the image capture devices device and the trigger control unit, when the image capture devices are triggered, the images captured by the image capture devices are sent to the interface circuit, and the interface circuit transmits the images of the image capture devices The image is decoded to obtain a synchronization signal and an image stream. The interface circuit sends the synchronization signal to the trigger control unit; a writing circuit is coupled to the interface circuit to receive the image decoded by the interface circuit. stream; a storage unit coupled to the writing circuit to store the image stream written by the writing circuit; and a reassembly circuit coupled to the storage unit, the reassembly circuit converts data from the storage unit The read image stream is flexibly reorganized into the merged image.

In an embodiment of this case, the interface circuit adjusts the screen size of the images.

In one embodiment of the present case, the writing circuit writes the image stream into the storage unit in a line-by-line scanning sequence.

In an embodiment of the present case, the storage unit includes a plurality of strip buffers or a frame buffer; wherein, when the storage unit includes the strip buffers, the size of the storage unit is related to an image capture device Quantity, an image width of the captured image and the number of image pixels of the captured image.

In an embodiment of the present case, the reorganization circuit includes a plurality of reorganization units. Each reorganization unit includes: a storage control unit, a reading unit and a frame control unit. When an image data in the storage unit is ready At this time, the storage control unit notifies the reading unit to read the storage unit according to a scheduling queue parameter. The scheduling queue parameter is set by the user. The scheduling queue parameter is selected from the image capture devices. Images are merged flexibly, and the schedule queue parameter flexibly specifies an image merging sequence.

In one embodiment of this case, in response to the notification from the storage control unit, the reading unit reads the image stream from the storage unit and sends it back to the storage control unit; the storage control unit reads the image stream transmitted to the frame control unit; and, the frame control unit flexibly reorganizes the image stream sent from the storage control unit to generate the merged image.

In one embodiment of this case, the trigger control unit generates a plurality of trigger signals to the image capture devices to control the image capture and an image output timing of the image capture devices. These trigger signals are independent of each other.

In an embodiment of this case, the trigger control unit selects one of the image capture devices as a reference image capture device. Compared with the reference image capture device, other non-reference images of the image capture devices The triggering of the capture device is delayed.

In one embodiment of this case, the trigger control unit includes: a multiplexer, a timing calculation unit and a delay control circuit. The delay control circuit includes a plurality of delay control units; the multiplexer is coupled to the interface circuit, The multiplexer transmits the synchronization signal of the reference image capture device to the timing calculation unit; the timing calculation unit performs timing calculation; and the delay control units count the timing of the image capture devices according to a delay time parameter. Individual delay times are used to trigger the image capture devices.

According to one aspect of this case, an image merging method is proposed, including: controlling the triggering of multiple image capture devices; when the image capture devices are triggered, the image capture devices capture multiple images; and, flexible selection The images captured by the image capturing devices are flexibly reorganized into a merged image.

In an embodiment of the present invention, the images captured by the image capturing devices are flexibly reused according to the size of an internal buffer.

In an embodiment of the present case, the images of the image capture devices are decoded to obtain a synchronization signal and an image stream; the image stream is stored in a storage unit; and read from the storage unit The image stream is reorganized into a merged image.

In an embodiment of this case, the screen size of the images is adjusted.

In one embodiment of the present case, the image stream is written to the storage unit in a line-by-line scanning sequence.

In an embodiment of the present case, the storage unit includes a plurality of strip buffers or a frame buffer; wherein, when the storage unit includes the strip buffers, the size of the storage unit is related to an image capture device Quantity, an image width of the captured image and the number of image pixels of the captured image.

In an embodiment of this case, when an image data of the storage unit is ready, the storage unit is read according to a schedule queue parameter. The schedule queue parameter is set by the user. The schedule queue Parameters are selected from the image capture devices to flexibly merge images, and the schedule queue parameters flexibly specify an image merging sequence.

In one embodiment of the present case, a plurality of trigger signals are generated to the image capture devices to control the image capture and an image output timing of the image capture devices. The trigger signals of the image capture devices are mutually exclusive. Independence.

In an embodiment of this case, one of the image capture devices is selected as a reference image capture device. Compared with the reference image capture device, the values of other non-reference image capture devices of the image capture devices are Trigger is delayed.

In an embodiment of the present case, timing calculation is performed based on the synchronization signal of the reference image capture device; and individual delay times of the image capture devices are counted according to a delay time parameter to trigger the image capture devices.

According to one aspect of this case, an electronic system is proposed, including: a plurality of image capture devices; and an image merging device coupled to the image capture devices. The image merging device controls the triggering of the image capture devices. When When the image capturing devices are triggered, the image capturing devices capture multiple images and send them to the image merging device. The image merging device flexibly selects the images captured by the image capturing devices to Flexible selection to reorganize into one or more merged images.

In order to have a better understanding of the above and other aspects of the present invention, examples are given below and are described in detail with reference to the accompanying drawings:

1_0~1_N, 4_0~4_N: Image capture device

2. 7: Central server

3_0~3_M, 8_0~8_M: Client

5_0~5_N: System single chip

6:Internet

100:Image merging device

105: Trigger control unit

110:Interface circuit

120:Writing circuit

130:Storage unit

140: Reorganization circuit

10_0~10_N: Image capture device

110_0~110_N:Interface

120_0~120_N: writing unit

140_0~140_M: Recombination unit

141:Storage control unit

143:Read unit

145: Picture frame control unit

150_0~150_M:Image processing unit

203:Multiplexer

205: Sequential calculation unit

210: Delay control circuit

210_0~210_N: Delay control unit

710~740: steps

810_0~810_M: Client

Figures 1A and 1B show two traditional surveillance systems.

Figure 2A shows a functional block diagram of an image merging device according to an embodiment of the present invention.

Figure 2B shows a functional block diagram of a trigger control unit according to an embodiment of the present invention.

Figure 2C shows a signal waveform diagram of a trigger control unit according to an embodiment of the present invention.

Figure 3 shows a schematic diagram of image merging according to an embodiment of the present invention.

Figures 4A and 4B show two image merging methods according to an embodiment of the present invention.

Figure 5 shows an example of image merging according to an embodiment of the present invention.

Figures 6A to 6D show different placement positions of the image capture devices in an embodiment of the present case.

Figure 7 shows an image merging method according to an embodiment of the present invention.

Figure 8 shows a schematic diagram of the image merging device of this embodiment applied to a surveillance system.

The technical terms in this specification refer to the idioms in the technical field. If there are explanations or definitions for some terms in this specification, the explanation or definition of this part of the terms shall prevail. Each embodiment of the present disclosure has one or more technical features. Under the premise that it is possible to implement, there are generally known people in this technical field. The skilled person may selectively implement some or all of the technical features in any embodiment, or selectively combine some or all of the technical features in these embodiments.

Please refer to Figure 2A, which illustrates a functional block diagram of an image merging device according to an embodiment of the present invention. The image merging device 100 according to an embodiment of the present invention includes: a trigger control unit 105, an interface circuit 110, a writing circuit 120, a storage unit 130 and a retiming circuit 140. The image merging device 100 can select and merge required images from a plurality of images captured by a plurality of image capturing devices 10_0~10_N (N is a positive integer). Here, the image capturing devices 10_0 to 10_N are, for example but not limited to, complementary metal-oxide semiconductor (Complementary Metal-Oxide-Semiconductor, CMOS) image sensors.

The trigger control unit 105 is coupled to the image capture devices 10_0~10_N, and controls the triggering of the image capture devices 10_0~10_N. When the image capturing devices 10_0~10_N are triggered by the trigger control unit 105, the image capturing devices 10_0~10_N capture images and send the captured images to the interface circuit 110. Details of the trigger control unit 105 will be described below. In other possible embodiments of this case, if the image capture devices 10_0~10_N have master-slave functions, the image capture devices can be connected in series with one of the image capture devices as the master, and the other image capture devices as the master. The image capture device is a servant. When triggered, the trigger control unit 105 can send a trigger signal to the image capture device with the master function, and the master image capture device transmits it to other slave image capture devices one by one.

The interface circuit 110 is coupled to the image capture devices 10_0~10_N and the trigger control unit 105. The interface circuit 110 includes a plurality of interfaces 110_0~110_N, respectively coupled to the image capture devices 10_0~10_N. The interfaces 110_0~110_N perform image windowing (for example, extracting a region of interest (ROI) from a complete image) and capture the images captured by the image capturing devices 10_0~10_N The decoded image stream is decoded, and the decoded synchronization signal is sent to the trigger control unit 105, and the decoded image stream is sent to the writing circuit 120. In addition, according to the user's needs, the interface circuit 110 can also adjust the The screen size of the image stream captured by some image capturing devices 10_0~10_N.

The writing circuit 120 is coupled to the interface circuit 110 and writes the image stream decoded by the interface circuit 110 into the storage unit 130 . The writing circuit 120 includes writing units 120_0~120_N. In an embodiment of the present case, for example but not limited to, the writing units 120_0~120_N of the writing circuit 120 can process the image streams decoded from the interfaces 110_0~110_N in the order of raster scan. written to storage unit 130.

The storage unit 130 is coupled to the writing circuit 120 and receives the image stream written by the writing circuit 120 . In one embodiment of this case, the storage unit 130 includes a plurality of line buffers to save costs. In this case, the image streams captured by the image capture devices 10_0~10_N are read by the reassembly circuit 140 After being output, it will be quickly overwritten by subsequent image streams, and therefore cannot be reused by the reassembly circuit 140. When the storage unit 130 includes several strip buffers, storing The size of the unit 130 can be expressed as (N+1)*W*1.5*B, where (N+1) represents the number of image capture devices, W is the image width of the captured image by the image capture device, and B is The number of image pixel bits in the image captured by the image capturing device. That is, when the storage unit 130 includes several strip buffers, the size of the storage unit 130 is related to the number of image capture devices, the image width of the captured image, and the number of image pixels of the captured image.

Or, in another example, the storage unit 130 includes a frame buffer to improve performance, and the image streams captured by the image capture devices 10_0 ~ 10_N can be shared/reused by the reassembly circuit 140 .

That is to say, in an embodiment of the present case, according to the size of the internal buffer of the storage unit 130, the image merging device 100 can flexibly reuse/share the image streams captured by the image capturing devices 10_0~10_N.

The reassembly circuit 140 is coupled to the storage unit 130 . The reassembly circuit 140 may reorganize the image stream read from the storage unit 130 into a merged image. For example, but not limited to, the merged image includes horizontal synchronization signals, vertical synchronization signals, pixel data, etc.

The recombination circuit 140 includes a plurality of recombination units 140_0~140_M (M is a positive integer). Each reorganization unit 140_0~140_M includes: a storage control unit 141, a reading unit 143 and a frame control unit 145.

When the image data in the storage unit 130 is ready, the storage control unit 141 notifies the reading unit 143 to read the storage unit 130 according to the spooling queue parameter SQ. That is, the storage control unit 141 can arrange The process determines which target images among the images stored in the storage unit 130 (captured by the image capturing devices 10_0~10_N) should be merged into one image, and accordingly notifies the reading unit 143 to read. The scheduling queue parameter SQ can be set by the user (manual setting), and can flexibly specify the scheduling order of one or more image capture devices. Moreover, the scheduling order of the specified image capture devices can be arranged arbitrarily. It is not limited to the consecutive numbering order of the image capture devices, or it is not limited to setting consecutive image capture devices into the same group. For example, but not limited to, the recombination unit 140_0 may select to recombine the images captured by the image capture devices 10_0~10_K (K≦N) into a complete merged image. In addition, the user can also specify the scheduling queue parameter SQ to specify the order of merging the images of the image capture devices 10_0~10_K. For example, but not limited to, in response to the scheduling queue parameter SQ, the reorganization unit 140_0 may specify the merging order as: images of the image capture device 10_2, images of the image capture device 10_1, images of the image capture device 10_K, etc., These are all within the spirit of this case. Or, in another possible example, in response to the scheduling queue parameter SQ, the reorganization unit 140_0 may specify the merging order as: the image of the image capture device 10_2, the image of the image capture device 10_4, the image of the image capture device 10_1 image.

That is to say, in an embodiment of the present case, the reorganization circuit 140 can "automatically select/manually select/flexibly select" to flexibly select/automatically select/select from the images captured by the image capture devices 10_0~10_N. Manually select the required images to flexibly merge them into a complete merged image. Therefore, the flexibility of use can be improved and the user experience can be enhanced.

In addition, in another embodiment of this case, the recombination circuit 140 can be from The image capture devices 10_0~10_N can flexibly select and reorganize one or more complete merged images (ie, multiple outputs) from multiple images captured by the image capture devices 10_0~10_N, thereby achieving multiple inputs and multiple outputs. Flexible image merging improves usage flexibility and enhances user experience.

In response to the notification from the storage control unit 141, the reading unit 143 reads the image stream from the storage unit 130 and sends it back to the storage control unit 141.

The frame control unit 145 reorganizes the image stream sent from the storage control unit 141 to generate a merged image. The merged images generated by the recombination units 140_0~140_M can be transmitted to the back-end image processing units 150_0~150_M. The architecture and operation of the image processing units 150_0~150_M are not particularly limited here.

For example, the reorganization unit 140_0 needs to reorganize images from the image capture devices 10_0~10_K (0≦K≦N). However, it should be noted that the present case is not limited to this. In response to the scheduling queue parameter SQ, when the first image strip of the captured image of the image capturing device 10_0 has been stored in the storage unit 130, the storage control unit 141 of the reassembly unit 140_0 instructs the reading unit 143 to retrieve the image from the storage unit 130. The first image strip of the captured image of the image capturing device 10_0 is read out and sent back to the storage control unit 141 and the frame control unit 145 . Continue accordingly. When the first image strip of the captured image of the image capturing device 10_K has been stored in the storage unit 130, the storage control unit 141 of the reorganization unit 140_0 instructs the reading unit 143 to read out the image strip of the image capturing device 10_K from the storage unit 130. The first image strip of the image is captured and returned to the storage control unit 141 and the frame control unit 145 . Recombination unit 140_0 The frame control unit 145 can reorganize the first image strip of the images of the image capture devices 10_0 ~ 10_K into the first image strip of a reorganized image. The rest are deduced in this way until the frame control unit 145 recombines the images of the image capture devices 10_0~10_K into a complete recombined image.

The trigger control unit 105 can generate trigger signals to the relevant image capture devices 10_0~10_N to control the image capture and image output timing of each image capture device 10_0~10_N. The trigger signals of each image capturing device 10_0 ~ 10_N can be independent. Therefore, each image capturing device 10_0 ~ 10_N can be triggered at any time point.

In an embodiment of this case, the trigger control unit 105 can select one of the image capture devices 10_0~10_N as the reference image capture device, and the triggering of other non-reference image capture devices is delayed.

Figure 2B shows a functional block diagram of the trigger control unit 105 according to an embodiment of the present application, and Figure 2C shows a signal waveform diagram of the trigger control unit according to an embodiment of the present application.

The trigger control unit 105 includes: a multiplexer 203, a timing calculation unit 205 and a delay control circuit 210. The delay control circuit 210 includes a plurality of delay control units 210_0~210_N.

The multiplexer 203 is coupled to the interfaces 110_0~110_N of the interface circuit 110. According to the control signal C (which can be determined by the user), one of the image capture devices 10_0~10_N is selected as the reference image capture device, and the multiplexer 203 will reference the synchronization signal (such as the horizontal synchronization signal) of the image capture device. ) sent to Timing calculation unit 205.

The timing calculation unit 205 performs timing calculations.

The delay control units 210_0~210_N count the individual delay times T0~TN of the image capture devices 10_0~10_N according to the delay time parameters (definable by the user), and send trigger signals CT0~CTN to the image capture devices 10_0~10_N.

In the following, the recombination unit 140_0 selects the image capture devices 10_0, 10_1 and 10_2 to merge images as an example for explanation, but it should be noted that the present case is not limited to this.

As shown in Figure 2C, assuming that the user selects the image capture device 10_1 as the reference image capture device, the delay time T1 of the image capture device 10_1=0, therefore, the delay control unit 210_1 of the trigger control unit 105 sends a trigger signal CT1 to image capture device 10_1. After the image capturing device 10_1 is triggered, the image capturing device 10_1 sends the image to the interface 110_1 of the interface circuit 110. After the interface 110_1 of the interface circuit 110 decodes the image of the image capture device 10_1, the interface 110_1 of the interface circuit 110 sends the frame valid signal FVS1 (can be regarded as a vertical synchronization signal) and the image bar valid signal LVS1 (can be regarded as a horizontal synchronization signal) to Trigger the multiplexer 203 of the control unit 105; and, the interface 110_1 of the interface circuit 110 sends the decoded image stream to the writing unit 120_1 of the writing circuit 120.

According to the control signal C (which can be determined by the user, in this example, the control signal C designates the image capture device 10_1 as the reference image capture device), the multiplexer 203 sends the frame valid signal FVS1 (which can be regarded as a vertical synchronization signal) with image bar The valid signal LVS1 (can be regarded as a horizontal synchronization signal) is provided to the timing calculation unit 205. The timing calculation unit 205 calculates the delay times T0˜T2 based on the frame valid signal FVS1 and the image strip valid signal LVS1.

After the delay time T0 (for example, but not limited to, the time for scanning 10 image strips), the delay control unit 210_0 of the trigger control unit 105 sends the trigger signal CT0 to the image capture device 10_0. After the image capturing device 10_0 is triggered, the image capturing device 10_0 sends the image to the interface 110_0 of the interface circuit 110. After the interface 110_0 of the interface circuit 110 decodes the image of the image capture device 10_0, the interface 110_0 of the interface circuit 110 sends the frame valid signal FVS0 (can be regarded as a vertical synchronization signal) and the image bar valid signal LVS0 (can be regarded as a horizontal synchronization signal) to Trigger the multiplexer 203 of the control unit 105; and, the interface 110_0 of the interface circuit 110 sends the decoded image stream to the writing unit 120_0 of the writing circuit 120.

Similarly, after the delay time T2 (for example, but not limited to, the time for scanning 20 image strips), the delay control unit 210_2 of the trigger control unit 105 sends the trigger signal CT2 to the image capture device 10_2. After the image capturing device 10_2 is triggered, the image capturing device 10_2 sends the image to the interface 110_2 of the interface circuit 110. After the interface 110_2 of the interface circuit 110 decodes the image of the image capture device 10_2, the interface 110_2 of the interface circuit 110 sends the frame valid signal FVS2 (can be regarded as a vertical synchronization signal) and the image bar valid signal LVS2 (can be regarded as a horizontal synchronization signal) to Trigger the multiplexer 203 of the control unit 105; and, the interface 110_2 of the interface circuit 110 sends the decoded image stream to the writing unit 120_2 of the writing circuit 120.

In addition, in Figure 2C, after the reassembly unit 140_0 of the reassembly circuit 140 combines the images captured by the image capture devices 10_0, 10_1 and 10_2 into a combined image, the reassembly unit 140_0 of the reassembly circuit 140 sends the combined image frame. The effective signal AF (can be regarded as a vertical synchronization signal) and the combined image strip effective signal LV (can be regarded as a vertical synchronization signal) are provided to the back-end image processing unit 150_0.

Specifically, during reassembly, only the image capture device 10_1 outputs image data before the delay time T0. After reaching the delay time T0, the image capture device 10_0 is triggered, and both the image capture devices 10_1 and 10_0 output image data synchronously. After the delay time T2 is reached, the image capture device 10_2 is triggered, and the image capture devices 10_1, 10_0 and 10_2 all output image data synchronously. However, when the frame sizes of the output images of the image capture devices 10_0, 10_1 and 10_2 are the same, there is a time difference in the image outputs of the image capture devices 10_1, 10_0 and 10_2, which means that when the image capture device 10_1 outputs the complete image Afterwards, the image capture devices 10_0 and 10_2 will still have data that has not been completely transmitted. Therefore, after the output of the image capturing devices is completed, a complete image can be reconstructed.

In an embodiment of this case, the delay times T0~TN are not limited to increasing sequentially, but may also decrease, etc., which is also within the spirit of this case. That is, when one of the image capturing devices is used as the reference image capturing device, the individual delay times of the other non-reference image capturing devices can be determined independently as needed.

Figure 3 shows a schematic diagram of image merging according to an embodiment of the present invention. As shown in FIG. 3 , the images ISO ~ ISN stored in the storage unit 130 respectively represent images captured by the image capture devices 10_0 ~ 10_N. Here, with emphasis on The grouping unit 140_0 reorganizes the images from the image capture devices 10_0~10_K (0≦K≦N), and the reorganization unit 140_M reorganizes the images from the image capture devices 10_P, 10_(P-2)..., 10_Q (0≦P, Q The image of ≦N) is for illustration, but it should be understood that this case is not limited thereto.

As shown in Figure 3, during reorganization, the reorganization unit 140_0 can reorganize the individual first image strips of the images ISO~ISK into the first image strip of the merged image AF0. The rest can be deduced in this way until the reassembly unit 140_0 reorganizes the images ISO~ISK into a complete merged image AF0. Similarly, when performing reorganization, the reorganization unit 140_M can reorganize the individual first image strips of the images ISP, IS(P-2)..., ISQ into the first image strip of a merged image AFM. The rest can be deduced in this way until the recombination unit 140_M recombines the images ISP, IS(P-2)..., ISQ into a complete merged image AFM.

Figures 4A and 4B show two image merging methods according to an embodiment of the present invention. Figure 4A shows the tracking frame mode (tracking frame mode), while Figure 4B shows the controlling frame mode (controlling frame mode).

In the tracking frame mode of Figure 4A, it is assumed that the shooting sequence is image capturing devices 10_2, 10_1, 10_0, 10_3 and 10_4. That is, the image output time axes of these image capture devices are staggered. In addition, when merging images, the relative positions of the images can be dynamically adjusted and are not limited to fixed heights. Tracking frame mode can be used for dynamic trajectory tracking.

In the control frame mode of Figure 4B, the output images of the image capture devices 10_0~10_4 are arranged horizontally and merged.

Figure 5 shows an example of image merging according to an embodiment of the present invention. As shown in FIG. 5 , the image capture devices are triggered at different times and output images to the image merging device 100 . The image merging device 100 combines the output images of the image capturing devices to obtain a merged image AF0.

In one embodiment of this case, after multiple images are merged into one image, image processing can be performed subsequently to detect or analyze objects in the image to perform object tracking.

Figures 6A to 6D show different placement positions of the image capture devices 10_0~10_4 in an embodiment of the present case. In an embodiment of the present case, the placement position of the image capturing devices is not limited to horizontal or vertical lines. In addition, the image capturing devices can be placed in any order. The placement positions of these image capture devices can be configured arbitrarily and are not limited to those shown in Figures 6A to 6D.

Figure 7 shows an image merging method according to an embodiment of the present invention. As shown in Figure 7, the image merging method includes: controlling the triggering of a plurality of image capturing devices (710); when the image capturing devices are triggered, the image capturing devices capture images. The image is decoded to obtain a synchronization signal and an image stream (720); the image stream is stored in a storage unit (730); and the image stream read from the storage unit is reorganized into a merged image (730). 740).

Figure 8 shows a schematic diagram of the image merging device of this embodiment applied to a surveillance system. As shown in Figure 8, clients 810_0~810_M have image processing units. Each client 810_0~810_M can configure/set the image merging device 100 to capture and merge images from any of the image capturing devices 10_0~10_M. Image merging The device 100 can transmit the merged image to each client 810_0~810_M. As shown in Figure 8, the surveillance system using the image merging device according to the embodiment of the present invention does not require a central server.

One embodiment of the present invention provides a flexible multi-input/multi-output architecture, which can control multiple image capture devices to capture images and merge them into one or more merged images for output. When necessary, the number of image capture devices can be expanded and/or the placement position of the image capture devices can be adjusted.

In one embodiment of this case, the effective frame rate of the merged image FPSe=FPSa*D, where FPSa represents the image frame rate of the image capture device, and D represents the number of merged images. In addition, in one embodiment of this case, the effective frame rate of the merged images can be increased through image merging, thereby improving object tracking efficiency.

The image merging device of this embodiment can be applied to many environments, such as, but not limited to, surveillance systems or head-mounted devices with virtual reality (VR)/augmented reality (AR) functions, etc. .

Conventional surveillance systems require the installation of a central server, so their costs are relatively high. However, in a surveillance system using the image merging device of this embodiment, image data can be processed in real time without a central server, and images can be captured from multiple image capture devices in real time and sent to the image destination. Therefore, the surveillance system using the image merging device of this embodiment has the advantage of low cost.

In addition, the conventional surveillance system is limited in expanding the number of image capture devices due to the number of channels of the central server. Therefore, it is difficult to expand the image capture devices. Get the number of devices. However, in a surveillance system using the image merging device of this embodiment, the number of image capture devices can be easily and flexibly expanded and/or their placement changed. Therefore, the surveillance system using the image merging device of this embodiment has the advantage of flexible expansion.

When the image merging device of this embodiment is applied to a head-mounted device with AR/VR functions, it can improve the object tracking function and/or dynamic trajectory detection function because the effective frame rate can be increased.

In addition, when the image merging device of this embodiment is applied to a head-mounted device with AR/VR functions, since the merged image includes dynamic trajectory information, the head-mounted device can efficiently detect the movement path of the object and perform tracking based on the tracking information. Trajectory, predicts the direction of head or object movement, and intercepts images in that direction for display in advance to avoid dizziness caused by viewing delayed (lag) images.

In summary, although the present invention has been disclosed above through embodiments, they are not intended to limit the present invention. Those with ordinary knowledge in the technical field to which the present invention belongs can make various modifications and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be determined by the appended patent application scope.

100:Image merging device

105: Trigger control unit

110:Interface circuit

120:Writing circuit

130:Storage unit

140: Reorganization circuit

10_0~10_N: Image capture device

110_0~110_N:Interface

120_0~120_N: writing unit

140_0~140_M: Recombination unit

141:Storage control unit

143:Read unit

145: Picture frame control unit

150_0~150_M:Image processing unit

Claims (13)

An electronic system includes: a plurality of image capturing devices; and an image merging device coupled to the image capturing devices. The image merging device controls the triggering of the image capturing devices. When the image capturing devices When triggered, the image capturing devices capture multiple images and send them to the image merging device. The image merging device flexibly selects the images captured by the image capturing devices to flexibly reorganize them into a merged image. , when merging, dynamically adjust the relative positional relationship between the images captured by the image capture devices. The electronic system as claimed in claim 1, wherein the image merging device flexibly reuses the images captured by the image capturing devices according to the size of an internal buffer. The electronic system of claim 1, wherein the image merging device includes: a trigger control unit coupled to the image capture devices to control the triggering of the image capture devices; an interface circuit coupled to The image capture devices and the trigger control unit, when the image capture devices are triggered, the images captured by the image capture devices are sent to the interface circuit, and the interface circuit captures the images Decode the images of the device to obtain a synchronization signal and an image stream, and the interface circuit sends the synchronization signal to the trigger control unit; A writing circuit, coupled to the interface circuit, receives the image stream decoded by the interface circuit; a storage unit, coupled to the writing circuit, stores the image stream written by the writing circuit stream; and a reorganization circuit coupled to the storage unit, the reorganization circuit flexibly reorganizes the image stream read from the storage unit into the merged image. The electronic system as claimed in claim 3, wherein the reorganization circuit includes a plurality of reorganization units, each reorganization unit includes: a storage control unit, a reading unit and a frame control unit, wherein when a storage unit When the image data is ready, the storage control unit notifies the reading unit to read the storage unit according to a schedule queue parameter. The schedule queue parameter is set by the user. The schedule queue parameter is obtained from the The image capture device is selected to flexibly merge images, and the schedule queue parameter flexibly specifies an image merging sequence. The electronic system as described in claim 3, wherein the trigger control unit generates a plurality of trigger signals to the image capture devices to control the image capture and an image output timing of the image capture devices, and the images The trigger signals of the capture device are independent of each other. The electronic system as described in claim 3, wherein the trigger control unit selects one of the image capture devices as a reference image capture device. Compared with the reference image capture device, the image capture devices The triggering of other non-reference image capture devices is delayed. An image merging method, applied to an electronic system, the image merging method includes: controlling the triggering of a plurality of image capturing devices by the electronic system; when the image capturing devices are triggered, the image capturing devices capture A plurality of images; and the electronic system flexibly selects the images captured by the image capture devices, and the electronic system flexibly reorganizes them into a merged image. During the merge, the images captured by the image capture devices are dynamically adjusted. The relative positional relationship between the captured images. The image merging method as described in claim 7 further includes: according to the size of an internal buffer, the electronic system flexibly reuses the images captured by the image capturing devices. The image merging method as described in claim 7, wherein the electronic system decodes the images of the image capture devices to obtain a synchronization signal and an image stream; the electronic system stores the image stream in a storage unit; and the image stream read from the storage unit is reorganized into a merged image by the electronic system. The image merging method as described in claim 9, wherein when an image data of the storage unit is ready, the storage unit is read from the electronic system according to a schedule queue parameter, and the schedule queue parameter As configured by the user, the schedule queue parameters are selected from the image capture devices to flexibly merge images by the electronic system, and the schedule queue parameters flexibly specify an image merging sequence. The image merging method described in claim 9, wherein the electronic system generates a plurality of trigger signals to the image capture devices to control the image capture and an image output timing of the image capture devices, the The trigger signals of the image capture device are independent of each other. The image merging method as described in claim 9, wherein the electronic system selects one of the image capture devices as a reference image capture device. Compared with the reference image capture device, the image capture devices Triggering of the device's other non-reference image capture devices is delayed. An electronic system includes: a plurality of image capturing devices; and an image merging device coupled to the image capturing devices. The image merging device controls the triggering of the image capturing devices. When the image capturing devices When triggered, the image capturing devices capture multiple images and send them to the image merging device. The image merging device flexibly selects the images captured by the image capturing devices to flexibly reorganize them into one or more images. When multiple merged images are merged, the relative positional relationship between the images captured by the image capture devices is dynamically adjusted.

Images