CN104601977A - Sensing device and signal processing method thereof - Google Patents

Abstract

The sensing device provided by the invention comprises a first sensor, a second sensor, a synchronizer and a combiner. The first sensor has a first resolution and generates a first data stream. The second sensor has a second resolution and generates a second data stream, wherein the second resolution is different from the first resolution. The synchronizer is electrically connected to the first sensor and the second sensor, and is configured to control timing of the first data stream and the second data stream, so that the first data stream and the second data stream have synchronous vertical synchronization signals. The combiner is configured to combine the first data stream and the second data stream to form a data output stream.

Description

Sensing device and signal processing method thereof

technical field

The invention relates to a sensing device and a signal processing method thereof, in particular to a sensing device incorporating multiple sensors with different resolutions and a signal processing method thereof.

Background technique

In recent years, with the development of stereoscopic display technology, the processing of stereoscopic images is becoming more and more important. Generally speaking, stereoscopic images can be formed in the following ways, such as shooting with a depth camera that can obtain depth information, shooting with a dual camera that simulates human binocular vision, or obtaining a 2D image through appropriate image processing. Stereoscopic image.

Among them, the depth photography technology uses a traditional RGB camera and another depth camera for shooting. The depth camera uses the principle of Time of Flight (ToF). By calculating the time required for the infrared rays emitted by the camera to reflect back to the depth camera after it irradiates the object, the distance between the object and the camera is calculated. distance.

Common products of depth photography technology include Microsoft’s Kinect sensor, which is used in somatosensory games, or augmented reality (augmented reality, AR), such as the application of three-dimensional fitting rooms. However, both the RGB camera and the depth camera of the aforementioned depth photography technology use their own image processors, which have problems of bulkiness and high cost. In addition, the data generated by the RGB camera and the depth camera are generally processed independently, so there is a problem of poor experience in existing practical applications. For example, in the game, there may be problems with insensitivity or slow response. However, in the example of augmented reality, there may be a situation where characters and backgrounds do not match, and there are still shortcomings in use.

Contents of the invention

In view of this, the object of the present invention is to provide a sensing device that can synchronize and combine the data streams of two sensors with different resolutions, so that the frames of the two sensors are in a synchronized state to provide subsequent better results. Applications.

Another object of the present invention is to provide a signal processing method for a sensing device, which can control the timing of the data streams of the above two sensors with different resolutions, so that the above two sensors have synchronous vertical synchronization signals to solve the problem of The imagery and depth mismatch issues mentioned above.

To achieve the above object, the sensing device provided by the present invention includes a first sensor, a second sensor, a synchronizer and a combiner. The first sensor has a first resolution and generates a first data stream. The second sensor has a second resolution and generates a second data stream, wherein the second resolution is different from the first resolution. The synchronizer is electrically connected to the first sensor and the second sensor, and is used to control the timing of the first data stream and the second data stream, so that the first data stream and the second data stream The second data stream has a synchronous vertical sync signal. The combiner is used for combining the first data stream and the second data stream to form a data output stream.

In a preferred embodiment, the sensing device further includes a signal processor for processing the data output stream for external use.

In a preferred embodiment, the data output stream generates a plurality of data lines forming a frame. In addition, the frame includes a first frame of the first data stream and a second frame of the second data stream, and the first frame and the second frame The sizes respectively correspond to the first resolution and the second resolution. Specifically, the size of the frame is twice the larger frame of the first frame and the second frame.

In a preferred embodiment, the first sensor is a depth sensor, the second sensor is an image sensor, and the first resolution of the depth sensor is smaller than the first resolution of the image sensor. Second resolution.

In a preferred embodiment, the combiner comprises a line buffer or a frame buffer.

In a preferred embodiment, the synchronizer provides a control frequency signal to the first sensor and the second sensor. In addition, the synchronizer provides a control delay signal to one of the first sensor and the second sensor.

To achieve the above object, the signal processing method of the sensing device provided by the present invention includes the following steps: generating a first data stream through a first sensor, the first sensor having a first resolution; generating a first data stream through a second sensor generating a second data stream, the second sensor having a second resolution, wherein the second resolution is different from the first resolution; controlling the first data stream and the second data stream the timing of the streams such that the first data stream and the second data stream have a synchronized vertical synchronization signal; combining the first data stream and the second data stream to form a data output stream stream; and processing the data output stream for external use.

In a preferred embodiment, the data output stream generates a plurality of data lines forming a frame. The frame includes a first frame of the first data stream and a second frame of the second data stream, and the sizes of the first frame and the second frame are respectively Corresponding to the first resolution and the second resolution. Further, the size of the frame is twice the larger frame of the first frame and the second frame.

In a preferred embodiment, the first data stream and the second data stream have synchronous data line signals.

In a preferred embodiment, controlling the timing of the first data stream and the second data stream specifically includes: providing a control frequency signal to the first sensor and the second sensor; and providing a control Delaying a signal to one of the first sensor and the second sensor.

Compared with the prior art, the present invention uses a synchronizer to synchronize the frequencies of the first sensor and the second sensor, and delays the data line signal sent by the sensor with a smaller resolution, so that the first sensor and the second sensor Synchronous frame transfer can be achieved. That is to say, the first sensor and the second sensor have synchronous vertical synchronization signals, so as to overcome the disadvantages of mismatching and poor experience in the prior art.

Description of drawings

In order to make the above and other purposes, features, and advantages of the present invention more obvious and understandable, the detailed description is as follows in conjunction with the accompanying drawings:

FIG. 1 is a schematic diagram of a sensing device according to a preferred embodiment of the present invention;

FIG. 2 is a timing diagram of a first data stream and a second data stream;

FIG. 3 is a timing diagram of a first data stream, a second data stream, and a data output stream according to a preferred embodiment of the present invention;

Figure 4 is a schematic diagram of the combined frame;

FIG. 5 is a flowchart of a signal processing method of a sensing device according to a preferred embodiment of the present invention.

[Description of main component symbols]

100 Sensing device 120 First sensor

140 Second sensor 160 Synchronizer

180 Combiner 182 Buffer 190 Signal Processor

220 First Data Stream 222 Data Line Signal

222(1)~ 222(240) The data line signal of the first line to the data line signal of the 240th line

240 Second Data Stream 244 Data Line Signal

244(1)~ 244(480) The data line signal of the first line to the data line signal of the 480th line

260 data output stream 310 control frequency signal

320 Control delay signal 400 Frame

420 First Frame 440 Second Frame.

Detailed ways

A number of preferred embodiments of the present invention are described in detail by the accompanying drawings and the following descriptions. In different drawings, the same component symbols represent the same or similar components.

Please refer to FIG. 1 , which is a schematic diagram of a sensing device according to a preferred embodiment of the present invention. For clarity, the sensing device 100 in this embodiment is represented by a dotted line. The sensing device 100 includes a first sensor 120 , a second sensor 140 , a synchronizer 160 , a combiner 180 and a signal processor 190 . It should be noted that this embodiment is described with two sensors, however, the present invention is not limited to two sensors, and more than two sensors are also within the scope of the present invention. The above-mentioned synchronizer 160 , combiner 180 and signal processor 190 can each be a chip or be integrated into a System-on-a-Chip (SoC) to reduce volume and cost.

Specifically, the first sensor 120 has a first resolution and generates a first data stream 220 . The first resolution of the first sensor 120 refers to the fineness it can sense. If an RGB camera is taken as an example, it is the resolution that the RGB camera can capture, such as VGA, which is 640*480. Taking a depth camera as an example, it is the resolution of the depth map captured by the RGB camera, such as QVGA, which is 320*240. Similarly, the second sensor 140 has a second resolution and generates a second data stream 240 . The second resolution of the second sensor 140 refers to the fineness it can sense. If an RGB camera is taken as an example, it is the resolution that the RGB camera can capture, such as VGA, which is 640*480. Taking a depth camera as an example, it is the resolution of the depth map captured by the RGB camera, such as QVGA, which is 320*240. In this embodiment, the second resolution is different from the first resolution. It should be noted that the present invention is not limited to the combination of a depth camera and an RGB camera, other such as light sensors, infrared sensors, ultrasonic sensors, CCD (Charge-Coupled Device) image sensors, CMOS (Complementary Metal Oxide Semiconductor) Any combination of image sensors and the like is within the scope of the present invention.

In this embodiment, the first sensor 120 is a depth sensor (such as a depth camera), the second sensor 140 is an image sensor (such as an RGB camera), and the depth sensor has the first resolution rate is less than the second resolution that the image sensor has. For example, the first resolution (that is, the resolution of the depth map) is QVGA, 320*240. The second resolution (that is, the resolution of the captured image) is VGA, 640*480. However, the present invention is not limited thereto, for example, the resolutions of the two can be other different multiples, and even the disproportionate relationship between the resolutions of the two is also within the scope of the present invention.

Please refer to FIG. 1 and FIG. 2 , wherein FIG. 2 is a timing diagram of the first data stream 220 and the second data stream 240 . The synchronizer 160 is electrically connected to the first sensor 120 and the second sensor 140, and is used to control the timing of the first data stream 220 and the second data stream 240, so that the first A data stream 220 and the second data stream 240 have a vertical sync signal (Vertical sync, VSYNC) synchronous. As shown in FIG. 2 , the first data stream 220 has a plurality of data row signals 222, each data row signal 222 includes the information amount of each data row, and the plurality of data rows constitute the data captured by the first sensor 120. A first picture frame (or frame frame) data. Therefore, the plurality of data row signals 222 includes the data row signal 222(1) of the first row, the data row signal 222(2) of the second row, the data row signal 222(3) of the third row... to the 240th row Row data row signal 222 (240). Similarly, the second data stream 240 has a plurality of data row signals 244, each data row signal 244 includes the information amount of each data row, and the plurality of data rows constitute a second data row captured by the second sensor 140. plot frame data. Therefore, the plurality of data row signals 244 includes the data row signal 244(1) of the first row, the data row signal 244(2) of the second row, the data row signal 244(3) of the third row... to the 480th row Row data row signal 244 (480).

In this embodiment, the synchronizer 160 controls the timing when the first sensor 120 outputs the data line signal 222(1) of the first line to be the same as the time when the second sensor 140 outputs the data line signal 244(1) of the first line. ) at the same moment, as shown in Figure 2. That is to say, the first sensor 120 and the second sensor 140 output the timings of the first frame and the second frame synchronously to achieve a synchronized vertical synchronization signal.

Referring to FIG. 1 again, the combiner 180 is used to combine the first data stream 220 and the second data stream 240 to form a data output stream 260 . Since the first sensor 120 and the second sensor 140 have a synchronous vertical synchronous signal, within a frame time (frame time), what the combiner 180 receives is that the first sensor 120 and the second sensor 140 are in the same position. Time to retrieve the data. Therefore, the first sensor 120 and the second sensor 140 with different resolutions can be synchronized, thereby solving the problem of mismatch between the existing depth camera and the RGB camera.

In this embodiment, the combiner 180 cooperates with the synchronizer 160 to generate a synchronized data output stream 260 . Specifically, the combiner 180 includes a buffer 182 . In addition, the synchronizer 160 provides a control frequency signal 310 to the first sensor 120 and the second sensor 140 . At the same time, the synchronizer 160 provides a control delay signal 320 to one of the first sensor 120 and the second sensor 140 . In detail, please refer to FIG. 3 , which is a timing diagram of the first data stream 220 , the second data stream 240 and the data output stream 260 in a preferred embodiment of the present invention. In this example, the synchronizer 160 provides the control delay signal 320 to the first sensor 120 so that each data row signal 222 output by the first sensor 120 is delayed by one data row period. That is to say, the first sensor 120 outputs the next data line signal 222 after the second sensor 140 outputs two data line signals 244 .

On the other hand, the buffer 182 of the combiner 180 is preferably a line buffer (line buffer), which receives the data line signal 222(1) of the first line of the first data stream 220 and receives the second data After the data row signal 244(1) of the first row and the data row signal 244(2) of the second row of the serial stream 240, the above three are combined and sent out. Subsequently, the line buffer receives the data line signal 222(2) of the 2nd line of the first data stream 220 and receives the data line signal 244(3) of the 3rd line of the second data stream 240 along with the data line signal 244(3). After the data line signal 244 (4) of 4 lines, the above three are combined and sent out, and so on, to form a data output stream 260 . Therefore, the capacity of the row buffer can accommodate one row of data row signals 222 and two rows of data row signals 244 .

Therefore, in the example where the first sensor 120 is a depth sensor (320*240) and the second sensor 140 is an image sensor (640*480), when the first sensor 120 outputs the data lines of a whole frame , that is, line 240, the second sensor 140 also outputs data lines of a complete picture frame, that is, line 480, so that the first data stream 220 and the second data stream 240 can be synchronized . Next, the data output stream 260 is provided to the signal processor 190, and the signal processor 190 formats the data output stream 260 into a format suitable for external use, such as providing a signal displayed on a monitor, or providing an external host Do other processing. Therefore, only a single signal processor can be used for the above-mentioned two sensors, thereby saving the cost of using its own signal processor for each sensor.

It is worth mentioning that the frequency of the combiner 180 may be twice or other multiples of the frequency of the first sensor 120 or the second sensor 140 to synchronize data processing.

However, in other embodiments, the synchronizer 160 controls the timing of the first data stream 220 and the second data stream 240 such that the first data stream 220 and the second data stream Stream 240 has a synchronized data line signal. That is, the data line signals of the first data stream 220 can be synchronized with the data line signals of the second data stream 240 .

Please refer to Figure 4, a schematic diagram of the combined frame. In this embodiment, the data output stream 260 generates a plurality of data rows forming a frame 400 . The frame 400 includes a first frame 420 of the first data stream 220 and a second frame 440 of the second data stream 240, and the first frame 420 and the second The sizes of the two frames 440 respectively correspond to the first resolution and the second resolution. That is to say, the size of the first frame 420 is 320*240, and the size of the second frame 440 is 640*480. Further, the size of the frame 400 is approximately twice as large as that of the first frame 420 and the second frame 440 . In this example, the size of the frame 400 is 1280*480. It is worth noting that, in other embodiments, the content in the frame 400 may also be the mixed information of the first frame 420 and the second frame 440, not just two frames in one Together, then the actual image can be derived from external decoding.

In another embodiment, the buffer 182 of the combiner 180 is a frame buffer. That is to say, the capacity of the frame buffer can accommodate one first frame 420 and one second frame 440 . The synchronizer 160 provides the control delay signal 320 to the first sensor 120, so that the first sensor 120 stops outputting the time of the first frame 420 after outputting the first frame 420, and then outputs the next one. A first frame 420 is drawn. That is to say, the first sensor 120 may wait for the second sensor 140 to output the second frame 440 before starting to output the next first frame 420 to achieve a synchronized vertical synchronization signal. After the frame buffer has received the first data stream 220 and the second data stream 240 of the first picture frame 420 and a second picture frame 440, it generates the combined data output stream 260 to achieve synchronization. requirements.

The signal processing method using the sensing device 100 described in this embodiment will be described in detail below. Please refer to FIG. 1 and FIG. 5 together, wherein FIG. 5 is a flowchart of a signal processing method of a sensing device according to a preferred embodiment of the present invention. The signal processing method described in this embodiment is used in the above-mentioned sensing device 100 . For the components mentioned below, please refer to the above-mentioned description, and details will not be repeated here.

The signal processing method starts at step S10. In step S10, a first data stream 220 is generated by a first sensor 120, and the first sensor 120 has a first resolution.

In step S20 , a second data stream 240 is generated by a second sensor 120 having a second resolution, wherein the second resolution is different from the first resolution.

In step S30, the timing of the first data stream 220 and the second data stream 240 is controlled so that the first data stream 220 and the second data stream 240 have synchronous vertical synchronization signals , and then execute step S40. It should be noted that steps S10 to S30 are preferably performed simultaneously.

In step S40, combine the first data stream 220 and the second data stream 240 to form a data output stream 260, and then execute step S50.

In step S50, the data output stream 260 is processed for external use.

Likewise, referring to FIG. 4 , the data output stream 260 generates a plurality of data rows forming a frame 400 . The frame 400 includes a first frame 420 of the first data stream 220 and a second frame 440 of the second data stream 240, and the first frame 420 and the second The sizes of the two frames 440 respectively correspond to the first resolution and the second resolution. Further, the size of the frame 400 is approximately twice as large as that of the first frame 420 and the second frame 440 .

Specifically, in step S30, controlling the timing of the first data stream 220 and the second data stream 240 specifically includes: providing a control frequency signal 310 to the first sensor 120 and the second sensor 120 a sensor 140 ; and providing a control delay signal 320 to one of the first sensor 120 and the second sensor 140 . However, in other embodiments, two different control delay signals can also be provided to the first sensor 120 and the second sensor 140 at the same time, so as to achieve more complex multi-sensor synchronization.

In summary, the preferred embodiment of the present invention uses the synchronizer 160 to synchronize the frequencies of the first sensor 120 and the second sensor 140, and delays the data line signal sent by the sensor with a smaller resolution, so that the first sensor 120 and the second sensor 140 The second sensor 140 can achieve synchronous frame transmission. That is to say, the first sensor 120 and the second sensor 140 have synchronous vertical synchronization signals, so as to overcome the disadvantages of mismatching and poor experience in the prior art.

The above is only a preferred embodiment of the present invention, it should be pointed out that for those of ordinary skill in the art, without departing from the structure of the present invention, some improvements and modifications can also be made, and these improvements and modifications should also be regarded as protection scope of the present invention.

Claims (17)

1. A sensing device, characterized in that it comprises: a first sensor having a first resolution and generating a first data stream; a second sensor having a second resolution and generating a second data stream, wherein the second resolution is different from the first resolution; A synchronizer, electrically connected to the first sensor and the second sensor, for controlling the timing of the first data stream and the second data stream, so that the first data stream and the second data stream the second data stream has a synchronized vertical sync signal; and A combiner is used for combining the first data stream and the second data stream to form a data output stream. 2. The sensing device according to claim 1, further comprising a signal processor for processing the data output stream for external use. 3. The sensing device according to claim 1, wherein the data output stream generates a plurality of data lines forming a frame. 4. The sensing device according to claim 3, wherein the frame comprises a first frame of the first data stream and a second frame of the second data stream, And the sizes of the first frame and the second frame respectively correspond to the first resolution and the second resolution. 5 . The sensing device according to claim 4 , wherein the size of the frame is twice the larger frame of the first frame and the second frame. 6. The sensing device according to claim 1, wherein the first data stream and the second data stream have synchronous data line signals. 7. The sensing device according to claim 1, wherein the first sensor is a depth sensor, and the second sensor is an image sensor. 8. The sensing device according to claim 7, wherein the first resolution of the depth sensor is smaller than the second resolution of the image sensor. 9. The sensing device of claim 1, wherein the combiner comprises a row of buffers. 10. The sensing device according to claim 1, wherein the combiner comprises a frame buffer. 11. The sensing device according to claim 1, wherein the synchronizer provides a control frequency signal to the first sensor and the second sensor. 12. The sensing device according to claim 11, wherein the synchronizer provides a control delay signal to one of the first sensor and the second sensor. 13. A signal processing method for a sensing device, comprising: generating a first data stream through a first sensor, the first sensor having a first resolution; generating a second data stream through a second sensor, the second sensor having a second resolution, wherein the second resolution is different from the first resolution; controlling the timing of the first data stream and the second data stream so that the first data stream and the second data stream have synchronous vertical synchronization signals; combining the first data stream and the second data stream to form a data output stream; and The data output stream is processed for external use. 14. The signal processing method according to claim 13, wherein the data output stream generates a plurality of data lines forming a frame. 15. The signal processing method according to claim 14, wherein the frame comprises a first frame of the first data stream and a second frame of the second data stream, And the sizes of the first frame and the second frame respectively correspond to the first resolution and the second resolution. 16. The signal processing method according to claim 15, wherein the size of the frame is twice the size of the larger frame of the first frame and the second frame. 17. The signal processing method according to claim 13, wherein controlling the timing of the first data stream and the second data stream specifically comprises: providing a control frequency signal to the first sensor and the second sensor; and A control delay signal is provided to one of the first sensor and the second sensor.