CN110460824B - Frame synchronization method of image data and camera - Google Patents
Frame synchronization method of image data and camera Download PDFInfo
- Publication number
- CN110460824B CN110460824B CN201910595552.4A CN201910595552A CN110460824B CN 110460824 B CN110460824 B CN 110460824B CN 201910595552 A CN201910595552 A CN 201910595552A CN 110460824 B CN110460824 B CN 110460824B
- Authority
- CN
- China
- Prior art keywords
- frame
- color image
- depth image
- time
- depth
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/103—Static body considered as a whole, e.g. static pedestrian or occupant recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/23—Recognition of whole body movements, e.g. for sport training
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/10—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/70—Circuitry for compensating brightness variation in the scene
- H04N23/73—Circuitry for compensating brightness variation in the scene by influencing the exposure time
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/04—Synchronising
- H04N5/06—Generation of synchronising signals
- H04N5/067—Arrangements or circuits at the transmitter end
- H04N5/0675—Arrangements or circuits at the transmitter end for mixing the synchronising signals with the picture signal or mutually
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Human Computer Interaction (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Social Psychology (AREA)
- Psychiatry (AREA)
- General Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
Abstract
The invention discloses a frame synchronization method of image data and a camera. The method of the invention comprises the following steps: acquiring a color image data stream and a depth image data stream transmitted by different signal channels, wherein a color image frame in the color image data stream and a depth image frame in the depth image data stream comprise synchronous frames synchronously output after exposure of a color camera and the depth camera, transmission delays of the different signal channels are different, and data volume structures of the color image frame and the depth image frame both carry a timestamp and exposure time; according to the time stamp and the exposure time, respectively acquiring SOF time of the color image frame and SOF time of the depth image frame; and acquiring synchronous frame images from the color image data stream and the depth image data stream according to the SOF time. The invention can avoid the asynchronism of image data caused by transmission delay and ensure that the upper layer obtains synchronous color images and depth images.
Description
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to a frame synchronization method for image data and a camera.
Background
The 3D skeleton recognition technology is applied to a plurality of fields, such as various beautifying functional software, has the skeleton recognition technology, and can accurately position the position of human skeleton, thereby locally adjusting the human image and realizing the function of beautifying the whole body. The bone recognition point technology, also called limb detection technology, can estimate the position of human body joint points from images or videos by using a computer vision technology, and gives an image containing a human body, so that the positions of all joints of the human body in the image, such as elbow joints, wrist joints, knee joints and the like, can be accurately positioned, and finally the positioned joint points can form a human body bone map which can reflect the information of the current posture and the like of the human body.
The bone recognition technology uses color images and depth images, has strict requirements on real-time performance, and two frames of data taken by an algorithm must be guaranteed to be shot at the same time, so that the technology has requirements on hardware to guarantee synchronous output of color data and depth data, and a strategy must be provided for a software layer to guarantee synchronous output of data taken by an upper layer sdk.
Under the condition of ensuring the synchronous output of the bottom layer, due to the problems of different exposure time, transmission delay and the like, the data taken by the upper layer is difficult to ensure to be the image shot at the same moment. In this case, since the joint points recognized by the bones are not accurate and stable recognition cannot be ensured, there is a defect that the positions of the recognized bones are deviated.
Disclosure of Invention
The invention provides a frame synchronization method of image data and a camera, which aim to solve the problem that two types of image data acquired by an upper layer are not synchronized due to different exposure time and transmission delay in the prior art.
In a first aspect, the present invention provides a method for frame synchronization of image data, including: acquiring a color image data stream and a depth image data stream transmitted by different signal channels, wherein a color image frame in the color image data stream and a depth image frame in the depth image data stream comprise synchronous frames synchronously output after exposure of a color camera and the depth camera, transmission delays of the different signal channels are different, and data volume structures of the color image frame and the depth image frame both carry a timestamp and exposure time; according to the time stamp and the exposure time, respectively acquiring SOF time of the color image frame and SOF time of the depth image frame; and acquiring synchronous frame images from the color image data stream and the depth image data stream according to the SOF time.
In some embodiments, acquiring a color image data stream and a depth image data stream transmitted by different signal channels comprises: constructing a frame buffer queue, and sequentially buffering the depth image frames in the obtained depth image data stream into the frame buffer queue; wherein a transmission delay of the depth image data stream is less than a transmission delay of the color image data stream; and acquiring color image frames of the color image data stream, and determining synchronous depth image frames corresponding to the color image frames from the depth image frames buffered in the frame buffer queue.
In some embodiments, acquiring a sync frame image from the color image data stream and the depth image data stream according to the SOF time comprises: when a first frame color image of the color image data stream is acquired, performing SOF time comparison on the first frame color image and a first frame depth image in the frame buffer queue; when the SOF time comparison result is not greater than a preset time threshold value, determining that the first frame color image and the first frame depth image are synchronous frame images; and when the SOF time comparison result is greater than the preset time threshold, acquiring the synchronous frame image according to the SOF time comparison result.
In some embodiments, acquiring the sync frame image according to the SOF time comparison comprises: when the SOF time of the first frame depth image is greater than the SOF time of the first frame color image, discarding the first frame color image and the first frame depth image, and performing SOF time comparison on a second frame color image of the color image data stream and a depth image frame in a frame buffer queue corresponding to the second frame color image to obtain a synchronous frame image of the second frame color image; and when the SOF time of the first frame depth image is not more than the SOF time of the first frame color image, discarding the first frame depth image, comparing the first frame color image with other depth image frames in the corresponding frame buffer queue by the SOF time, and determining the synchronous frame image of the first frame color image according to the SOF time comparison result.
In some embodiments, performing SOF time comparison of the first frame color image with other depth image frames in the frame buffer queue corresponding thereto, and determining a synchronization frame image of the first frame color image according to the SOF time comparison result includes: and when the synchronous frame image of the first frame color image is not matched in other depth image frames in the frame buffer queue corresponding to the first frame color image, if the SOF time of the other depth image frames is not greater than the SOF time of the first frame color image, generating prompt information indicating bottom layer transmission error.
In some embodiments, sequentially buffering the depth image frames in the acquired depth image data stream into the frame buffer queue includes: when a set number of depth image frames are cached in the frame cache queue, if a new depth image frame in the depth image data stream is acquired, discarding the first frame depth image in the frame cache queue, and caching the new depth image frame to the tail of the frame cache queue.
In some embodiments, obtaining the SOF times of the color image frames and the SOF times of the depth image frames from the time stamps and the exposure times, respectively, comprises: calculating the time sum of a time stamp carried by a data volume structure of the color image frame and exposure time, wherein the time sum is SOF time of the color image frame; and calculating the time sum of the time stamp carried by the data volume structure of the depth image frame and the exposure time, wherein the time sum is the SOF time of the depth image frame.
In some embodiments, acquiring a sync frame image from the color image data stream and the depth image data stream according to the SOF time further comprises: and carrying out bone recognition according to the synchronous frame image, identifying recognized bone key points on the color image, and displaying the color image with the identified bone key points.
In a second aspect, the present invention provides a camera comprising: the system comprises a color camera, a depth camera and a processor; the color camera collects color image frames and sends color image data streams formed by the color image frames to the processor through a first signal channel; the depth camera collects depth image frames and sends a depth image data stream formed by the depth image frames to the processor through a second signal channel; wherein, the color image frame in the color image data stream and the depth image frame in the depth image data stream include a synchronization frame synchronously output after the color camera and the depth camera are exposed, the transmission delay of the first signal channel is different from that of the second signal channel, and the data volume structures of the color image frame and the depth image frame both carry a timestamp and an exposure time; the processor is used for respectively acquiring SOF time of the color image frame and SOF time of the depth image frame according to the time stamp and the exposure time; and acquiring synchronous frame images from the color image data stream and the depth image data stream according to the SOF time.
In some embodiments, the camera further comprises a display; the processor is used for carrying out bone recognition according to the synchronous frame image, identifying recognized bone key points on the color image and sending the color image with the identified bone key points to the display; and the display displays the color images marked with the bone key points.
Aiming at the problem that the color image and the depth image which are simultaneously acquired by an upper layer are asynchronous when color image data and depth image data which are synchronously output from a bottom layer are transmitted through different signal channels, the SOF time of the image data is acquired based on the timestamp and the exposure time carried by the image data volume structure, and the image frame with the closest SOF time of the color image and the depth image is determined as a synchronous frame image, so that the asynchronous image data caused by transmission delay is avoided.
Drawings
Fig. 1 is a schematic diagram of a time node for acquiring a synchronous RGB image and a depth image by a Framework layer according to an embodiment of the present invention;
FIG. 2 is a flow chart illustrating a method for frame synchronization of image data according to an embodiment of the present invention;
FIG. 3 is a SOF time diagram of an RGB image and a depth image according to an embodiment of the present invention;
FIG. 4 is a schematic diagram illustrating a process of acquiring a sync frame image according to an embodiment of the present invention;
FIG. 5 is a schematic view of human skeletal points illustrating an embodiment of the present invention;
fig. 6 is a block diagram of a camera according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention will be described in detail with reference to the accompanying drawings.
Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. It is to be understood that this description is made only by way of example and not as a limitation on the scope of the invention. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present invention.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The words "a", "an" and "the" and the like as used herein are also intended to include the meanings of "a plurality" and "the" unless the context clearly dictates otherwise. Furthermore, the terms "comprises," "comprising," and the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.
All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It is noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of this specification and should not be interpreted in an idealized or overly formal sense.
Some block diagrams and/or flowcharts are shown in the figures. It will be understood that some blocks of the block diagrams and/or flowchart illustrations, or combinations thereof, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the instructions, which execute via the processor, create means for implementing the functions/acts specified in the block diagrams and/or flowchart block or blocks.
Thus, the techniques of the present invention may be implemented in hardware and/or in software (including firmware, microcode, etc.). Furthermore, the techniques of this disclosure may take the form of a computer program product on a machine-readable medium having instructions stored thereon for use by or in connection with an instruction execution system. In the context of the present invention, a machine-readable medium may be any medium that can contain, store, communicate, propagate, or transport the instructions. For example, a machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium. Specific examples of the machine-readable medium include: magnetic storage devices, such as magnetic tape or Hard Disk Drives (HDDs); optical storage devices, such as compact disks (CD-ROMs); a memory, such as a Random Access Memory (RAM) or a flash memory; and/or wired/wireless communication links.
The skeleton recognition technology is accurately applied, on one hand, the synchronism of Hardware Abstraction Layer (HAL) data is ensured, namely a color image and a depth image are shot at the same time, and the time difference is stabilized within 5 ms; on the other hand, the color image and the depth image obtained by the SDK layer are required to be ensured to be synchronous data. Only in this way the accuracy and stability of the bone identification technique can be guaranteed.
In the prior art, it can be ensured that the bottom layer synchronously outputs color image data and depth image data, that is, under the condition of hardware support, the transmission synchronization of the synchronization frame after exposure can be ensured. When data transmission is performed through a Mobile Industry Processor Interface (MIPI) after exposure is completed, since color image data and depth image data are transmitted through different signal channels, a color image and a depth image obtained by a frame layer are asynchronous.
In the prior art, in a system integrating a color camera and a depth camera, the depth camera is used as a master camera, and the color camera is used as a slave camera, i.e., a synchronization signal of the depth camera triggers the color camera to complete bottom layer synchronous transmission. However, due to the fact that exposure time of the two cameras is different, transmission delay in the transmission process is different, and the like, when the frame layer receives color image data, depth image data of a synchronous frame already arrives, and if a depth image is recalled based on a time point of the color image, the recalled depth image and the color image are not synchronous. As shown in fig. 1, the color image rgb1 and the depth image tof1 in fig. 1 are synchronous frame images.
In order to solve the above problem, in the embodiments of the present invention, a frame transmission start time, that is, SOF time, is obtained by using a timestamp and exposure time carried by a data volume of a frame image in an image data stream, and a frame depth image, in which the current color image SOF time is closest to the SOF time in a depth image data stream, is searched as a synchronization frame image.
The invention provides a frame synchronization method of image data.
Fig. 2 is a flowchart illustrating a frame synchronization method for image data according to an embodiment of the present invention, and as shown in fig. 2, the method of the embodiment includes:
s210, color image data streams and depth image data streams transmitted by different signal channels are obtained, wherein color image frames in the color image data streams and depth image frames in the depth image data streams comprise synchronous frames synchronously output after exposure of a color camera and the depth camera, transmission delays of the different signal channels are different, and data volume structures of the color image frames and the depth image frames both carry time stamps and exposure time.
In the embodiment, the time stamp can be understood as a time point when the color camera and the depth camera start to be exposed, and the long exposure time can be understood as the time used by the color camera and the depth camera to be exposed. Since the exposure time of the color image is much longer than that of the depth image, the time stamps of the color image and the depth image in the sync frame image are not the same. Referring to fig. 3, taking an RGB image as an example, an exposure time used by the RGB camera to capture the RGB image is about 25ms, and an exposure time used by the depth camera to capture a depth image is about 0.03ms, in order to ensure that the bottom layer synchronously outputs two images (i.e., SOF time corresponds to the same time), it is necessary to ensure that a time stamp of the RGB image is greater than a time stamp of the depth image, and a difference between the two time stamps is about 25-0.03ms.
And S220, respectively acquiring SOF time of the color image frame and SOF time of the depth image frame according to the time stamp and the exposure time.
And S230, acquiring synchronous frame images from the color image data stream and the depth image data stream according to the SOF time.
In the embodiment, for the problem that the color image and the depth image acquired at the same time by the upper layer are not synchronized when the color image data and the depth image data synchronously output from the bottom layer are transmitted through different signal channels, the SOF time of the image data is acquired based on the timestamp and the exposure time carried by the image data volume structure, and the image frame with the closest SOF time of the color image and the depth image is determined as the synchronous frame image, so that the image data is not synchronized due to transmission delay.
The above steps S210-S230 are described in detail below with reference to fig. 3-5.
Firstly, step S210 is executed, namely, a color image data stream and a depth image data stream transmitted by different signal channels are obtained, wherein a color image frame in the color image data stream and a depth image frame in the depth image data stream include a synchronization frame synchronously output by a color camera and a depth camera after exposure, transmission delays of the different signal channels are different, and data volume structures of the color image frame and the depth image frame both carry a timestamp and an exposure time.
In a system integrated with a color camera and a depth camera, the depth camera is used as a main camera, the color camera is used as a slave camera, and a synchronous signal of the depth camera triggers the color camera to acquire images so as to finish synchronous acquisition of bottom images.
Due to the problem of transmission delay of a signal channel, referring to fig. 1, when a frame layer is taken to two adjacent frames (namely, a depth image and a color image) of data, the SOF time of the depth image is faster than that of the color image, so the depth image data needs to be cached in a queue, after the color image arrives, the SOF time of the depth image is matched in the queue, if the time difference is within 5ms, the color image and the depth image are adjusted back together, and the two are synchronous frame images.
Specifically, when a color image data stream and a depth image data stream are acquired, a frame buffer queue is constructed, and depth image frames in the acquired depth image data stream are sequentially buffered in the frame buffer queue; wherein a transmission delay of the depth image data stream is less than a transmission delay of the color image data stream; and acquiring color image frames of the color image data stream, and determining synchronous depth image frames corresponding to the color image frames from the depth image frames buffered in the frame buffer queue.
When a set number of depth image frames are cached in the frame cache queue, if a new depth image frame in the depth image data stream is acquired, discarding the first frame depth image in the frame cache queue, and caching the new depth image frame to the tail of the frame cache queue. For example, the size of the frame buffer queue is size =3, when there are 3 frames of depth image data in the frame buffer queue, and new depth image data comes again, the oldest data is dropped (pop), and then the newest depth image data is pushed to the tail (push) of the queue, so that the data in the frame buffer queue is always guaranteed to be less than or equal to 3.
In an example, the frame interval time of adjacent color image frames in a color image data stream is 30ms, and the frame interval time of adjacent depth image frames in a depth image data stream is also 30ms, because a depth camera is used to trigger the color camera, and the exposure time and the transmission delay of the depth camera are both smaller than those of the color camera, when a color image is obtained on a frame layer, a synchronized depth image corresponding to the color image is cached in a queue, and because the depth image data cached in the frame cache queue is updated, when a time node of each frame of the color image is received, a depth image cached in the frame cache queue under the time node needs to be determined, and then a synchronized frame image matching the current color image is matched in the depth image cached in the frame cache queue under the time node.
After the color image data stream and the depth image data stream are acquired, step S220 is continuously performed, that is, SOF times of the color image frame and the depth image frame are respectively acquired according to the time stamp and the exposure time.
In some embodiments, the SOF time is obtained by: calculating the time sum of a time stamp carried by a data volume structure of the color image frame and exposure time, wherein the time sum is SOF time of the color image frame; and calculating the sum of the time stamp carried by the data volume structure of the depth image frame and the exposure time, wherein the sum of the time is the SOF time of the depth image frame.
In this embodiment, the data structures Metadata of each frame of image in the acquired color image data stream and depth image data stream both carry a TIMESTAMP SENSOR _ TIMESTAMP and an EXPOSURE TIME SENSOR _ EXPOSURE _ TIME, and the two are added to obtain the SOF TIME, that is, SOF = SENSOR _ TIMESTAMP + SENSOR _ EXPOSURE _ TIME, so that only one frame of the color image and the depth image whose SOF TIMEs are closest (within 5 ms) needs to be recalled together, that is, the synchronous frame image can be obtained, and the influence of transmission delay is avoided.
After the SOF time of the color image frame and the SOF time of the depth image frame are acquired, step S230 is continuously performed, that is, the synchronization frame image is acquired from the color image data stream and the depth image data stream according to the SOF times.
In some embodiments, when a first frame of color image of the color image data stream is acquired, performing SOF time comparison on the first frame of color image and a first frame of depth image in the frame buffer queue; when the SOF time comparison result is not greater than a preset time threshold (for example, the preset time threshold is 5 ms), determining that the first frame color image and the first frame depth image are synchronous frame images; and when the SOF time comparison result is greater than the preset time threshold, acquiring the synchronous frame image according to the SOF time comparison result.
When the SOF time of the first frame depth image is greater than the SOF time of the first frame color image, discarding the first frame color image and the first frame depth image, and performing SOF time comparison on a second frame color image of the color image data stream and a depth image frame in a frame buffer queue corresponding to the second frame color image to obtain a synchronous frame image of the second frame color image;
specifically, when the SOF time of the first frame depth image is not greater than the SOF time of the first frame color image, discarding the first frame depth image, performing SOF time comparison between the first frame color image and other depth image frames in the frame buffer queue corresponding to the first frame depth image, and determining a synchronous frame image of the first frame color image according to the SOF time comparison result. When the SOF time of the first frame depth image is not more than the SOF time of the first frame color image, discarding the first frame depth image, and successively comparing the first frame color image with the second frame depth image and the third frame depth image in the frame buffer queue corresponding to the first frame color image according to the comparison rule of the SOF time of the first frame color image and the first frame color image to perform SOF time comparison, so as to obtain a comparison result.
When the synchronous frame image of the first frame color image is not matched in other depth image frames in the frame buffer queue corresponding to the first frame color image, if the SOF time of the other depth image frames is not greater than the SOF time of the first frame color image, prompt information indicating bottom layer transmission error is generated.
The following describes the process of obtaining the sync frame image from the color image data stream and the depth image data stream in detail with reference to the sync frame matching method shown in fig. 4.
Wherein, the judgment condition (1) | rgb. SOF-tof. SOF | ≦ 5ms shown in FIG. 4 can be understood as: judging whether the two groups of data are the basis of synchronous frames, namely whether the time difference value of the two groups of data (depth images and color images) transmitted to the frame layer is within 5ms, if so, determining the two groups of data as synchronous frame images, otherwise, determining the two groups of data as asynchronous frame images. SOF denotes SOF time of a color image, and tof denotes SOF time of a depth image.
The judgment conditions shown in FIG. 4 (2) tof. SOF > rgb. SOF can be understood as: when the SOF time of the depth image in the frame buffer queue is greater than the SOF time of the color image, it is not necessary to compare the SOF time of the color image with the SOF time of the next frame of depth image in the frame buffer queue, because the SOF time of the next frame of depth image is greater, at this time, the color image and the compared depth image are discarded, and then the color image data of the next frame is waited to reach the Framework layer.
Assuming that the size of the frame buffer queue is size =3, since the color image data is slow down to the frame layer, the present embodiment takes the color image data as the master and the depth image data as the slave. When color image data rgb1 arrives, depth image data tof1, tof2 and tof3 of3 frames are cached in a frame buffer queue at this time, first comparison is carried out, if | rgb1. SOF-toff 1.SOF | ≦ 5ms, it is proved that tof1 and rgb1 in the frame buffer queue are synchronous frame images, the images can be combined together and called back to an upper layer SDK, if | rgb1. SOF-toff 1.SOF | > greater than 5ms, the SOF time of two groups of data is continuously judged according to a judgment condition (2), if tof1.SOF > rgb1.SOF, it is unnecessary to continuously compare tof2 and tof3 in the frame buffer queue, at this time, rgb1 is lost, newest rgb2 data is continuously waited, and comparison is carried out again according to the matching flow shown in fig. 4. Otherwise, the tof1 of the frame buffer queue needs to be dropped, and the tof2 and the rgb1 are continuously compared, and the comparison operation is repeated.
If the data in the frame buffer queue can not be combined with rgb1 to form a synchronous frame image, and the SOF time values of the 3 frames of depth image data in the frame buffer queue are all smaller than the SOF time value of rgb1, it indicates that the bottom layer transmission has a problem, and an error code is returned.
By the method, the frame loss problem can be avoided, frame synchronization is strictly ensured, when the synchronous frame image is obtained, bone identification is carried out according to the synchronous frame image, the identified bone key points are identified on the color image, and the color image with the identified bone key points is displayed. Wherein the skeletal key points include skeletal points shown in fig. 5.
By using the synchronous color image and the depth image, the accuracy and the stability of the bone recognition algorithm can be ensured, and the accuracy of the bone recognition is greatly improved. When the human body moves, the position of the skeleton can be quickly positioned, and the motion of the human body can be predicted.
The invention also provides a camera.
Fig. 6 is a block diagram of a camera according to an embodiment of the present invention, and as shown in fig. 6, the camera according to the embodiment includes: the system comprises a color camera, a depth camera and a processor;
the color camera collects color image frames and sends color image data streams formed by the color image frames to the processor through a first signal channel;
the depth camera collects depth image frames and sends a depth image data stream formed by the depth image frames to the processor through a second signal channel; wherein, the color image frame in the color image data stream and the depth image frame in the depth image data stream include a synchronization frame synchronously output after the color camera and the depth camera are exposed, the transmission delay of the first signal channel is different from that of the second signal channel, and the data volume structures of the color image frame and the depth image frame both carry a timestamp and an exposure time;
the processor is used for respectively acquiring SOF time of the color image frame and SOF time of the depth image frame according to the time stamp and the exposure time; and acquiring synchronous frame images from the color image data stream and the depth image data stream according to the SOF time.
In some embodiments, the processor constructs a frame buffer queue, and sequentially buffers depth image frames in the obtained depth image data stream into the frame buffer queue; wherein a transmission delay of the depth image data stream is less than a transmission delay of the color image data stream; and acquiring a color image frame of the color image data stream, and determining the frame buffer queue corresponding to the color image frame.
Correspondingly, when a first frame color image of the color image data stream is acquired, the processor compares the first frame color image with a first frame depth image in the frame buffer queue by the SOF time; when the SOF time comparison result is not greater than a preset time threshold value, determining that the first frame color image and the first frame depth image are synchronous frame images; and when the SOF time comparison result is greater than the preset time threshold, acquiring the synchronous frame image according to the SOF time comparison result. Specifically, when the SOF time of the first frame depth image is greater than the SOF time of the first frame color image, discarding the first frame color image and the first frame depth image, and performing SOF time comparison on a second frame color image of the color image data stream and a depth image frame in a frame buffer queue corresponding to the second frame color image to obtain a synchronous frame image of the second frame color image; and when the SOF time of the first frame depth image is not more than the SOF time of the first frame color image, discarding the first frame depth image, performing SOF time comparison on the first frame color image and other depth image frames in the frame buffer queue corresponding to the first frame color image, and determining the synchronous frame image of the first frame color image according to the SOF time comparison result.
The processor is further configured to generate a prompt message indicating a bottom layer transmission error if SOF times of other depth image frames are not greater than SOF times of the first frame color image when the synchronous frame image of the first frame color image is not matched in other depth image frames in the frame buffer queue corresponding to the first frame color image.
In some embodiments, the processor calculates a sum of time of a timestamp and an exposure time carried by a data volume structure of the color image frame, the sum of time being a SOF time of the color image frame; and calculating the time sum of the time stamp carried by the data volume structure of the depth image frame and the exposure time, wherein the time sum is the SOF time of the depth image frame.
In some embodiments, the camera shown in fig. 5 further comprises a display;
the processor is used for carrying out bone recognition according to the synchronous frame image, identifying recognized bone key points on the color image and sending the color image with the identified bone key points to the display;
and the display displays the color image marked with the bone key points.
For the camera embodiment, since it basically corresponds to the method embodiment, the relevant points may be referred to the partial description of the method embodiment. The above-described embodiments of the camera are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
While the foregoing is directed to embodiments of the present invention, other modifications and variations of the present invention may be devised by those skilled in the art in light of the above teachings. It should be understood by those skilled in the art that the foregoing detailed description is for the purpose of better explaining the present invention, and the scope of the present invention should be determined by the scope of the appended claims.
Claims (7)
1. A method for frame synchronization of image data, comprising:
acquiring a color image data stream and a depth image data stream transmitted by different signal channels, wherein a color image frame in the color image data stream and a depth image frame in the depth image data stream comprise synchronous frames synchronously output after exposure of a color camera and the depth camera, transmission delays of the different signal channels are different, and data volume structures of the color image frame and the depth image frame both carry a timestamp and exposure time;
according to the time stamp and the exposure time, respectively obtaining the SOF time of the color image frame and the SOF time of the depth image frame;
acquiring synchronous frame images from the color image data stream and the depth image data stream according to the SOF time;
the obtaining of the color image data stream and the depth image data stream transmitted by different signal channels, where the transmission delay of the depth image data stream is smaller than the transmission delay of the color image data stream, includes:
constructing a frame buffer queue, sequentially buffering depth image frames in an acquired depth image data stream into the frame buffer queue, discarding a first frame depth image in the frame buffer queue if a set number of depth image frames are buffered in the frame buffer queue and a new depth image frame in the depth image data stream is acquired, and buffering the new depth image frame to the tail of the frame buffer queue;
acquiring color image frames of the color image data stream, and determining synchronous depth image frames corresponding to the color image frames from the depth image frames buffered in the frame buffer queue;
wherein acquiring a sync frame image from the color image data stream and the depth image data stream according to the SOF time comprises: when a first frame color image of the color image data stream is acquired, performing SOF time comparison on the first frame color image and a first frame depth image in the frame buffer queue; when the SOF time of the first frame depth image is not greater than the SOF time of the first frame color image, acquiring the synchronous frame image according to the SOF time comparison result; wherein, obtaining the synchronous frame image according to the SOF time comparison result comprises: when the SOF time of the first frame depth image is not greater than the SOF time of the first frame color image, discarding the first frame depth image, performing SOF time comparison on the first frame color image and other depth image frames in the frame buffer queue corresponding to the first frame color image, and when the synchronous frame image of the first frame color image is not matched in other depth image frames in the frame buffer queue corresponding to the first frame color image, if the SOF times of the other depth image frames are not greater than the SOF time of the first frame color image, generating prompt information indicating bottom layer transmission errors.
2. The method of claim 1, wherein said obtaining a synchronized frame image from the color image data stream and the depth image data stream according to the SOF time comprises:
and when the SOF time comparison result is not greater than a preset time threshold value, determining that the first frame color image and the first frame depth image are synchronous frame images.
3. The method of claim 2, wherein said obtaining the synchronization frame image according to the SOF time comparison comprises:
and when the SOF time of the first frame depth image is greater than the SOF time of the first frame color image, discarding the first frame color image and the first frame depth image, and performing SOF time comparison on a second frame color image of the color image data stream and a depth image frame in a frame buffer queue corresponding to the second frame color image to obtain a synchronous frame image of the second frame color image.
4. The method of claim 1, wherein the obtaining the SOF times of the color image frame and the depth image frame according to the time stamp and the exposure time, respectively, comprises:
calculating the time sum of a time stamp carried by a data volume structure of the color image frame and exposure time, wherein the time sum is SOF time of the color image frame; and the number of the first and second groups,
and calculating the time sum of the time stamp carried by the data volume structure of the depth image frame and the exposure time, wherein the time sum is the SOF time of the depth image frame.
5. The method of claim 1, wherein the obtaining a synchronized frame image from the color image data stream and the depth image data stream according to the SOF time further comprises:
and carrying out bone identification according to the synchronous frame image, identifying the identified bone key points on the color image, and displaying the color image identified with the bone key points.
6. A camera, comprising: the system comprises a color camera, a depth camera and a processor;
the color camera collects color image frames and sends color image data streams formed by the color image frames to the processor through a first signal channel;
the depth camera collects depth image frames and sends a depth image data stream formed by the depth image frames to the processor through a second signal channel; wherein, the color image frame in the color image data stream and the depth image frame in the depth image data stream include a synchronization frame synchronously output after the color camera and the depth camera are exposed, the transmission delay of the first signal channel is different from that of the second signal channel, and the data volume structures of the color image frame and the depth image frame both carry a timestamp and an exposure time;
the processor is used for respectively acquiring SOF time of the color image frame and SOF time of the depth image frame according to the time stamp and the exposure time; acquiring synchronous frame images from the color image data stream and the depth image data stream according to the SOF time;
the processor is used for constructing a frame cache queue, sequentially caching depth image frames in an acquired depth image data stream into the frame cache queue, discarding a first frame depth image in the frame cache queue if a set number of depth image frames are cached in the frame cache queue and a new depth image frame is acquired in the depth image data stream, and caching the new depth image frame to the tail of the frame cache queue; wherein a transmission delay of the depth image data stream is less than a transmission delay of the color image data stream; acquiring a color image frame of the color image data stream, and determining the frame buffer queue corresponding to the color image frame;
wherein acquiring a sync frame image from the color image data stream and the depth image data stream according to the SOF time comprises: when a first frame color image of the color image data stream is acquired, performing SOF time comparison on the first frame color image and a first frame depth image in the frame buffer queue; when the SOF time of the first frame depth image is not greater than the SOF time of the first frame color image, acquiring the synchronous frame image according to the SOF time comparison result; wherein, obtaining the synchronous frame image according to the SOF time comparison result comprises: when the SOF time of the first frame depth image is not greater than the SOF time of the first frame color image, discarding the first frame depth image, performing SOF time comparison on the first frame color image and other depth image frames in the frame buffer queue corresponding to the first frame color image, and when the synchronous frame image of the first frame color image is not matched in other depth image frames in the frame buffer queue corresponding to the first frame color image, if the SOF times of the other depth image frames are not greater than the SOF time of the first frame color image, generating prompt information indicating bottom layer transmission errors.
7. The camera of claim 6, further comprising a display;
the processor is used for carrying out bone recognition according to the synchronous frame image, identifying recognized bone key points on the color image and sending the color image with the identified bone key points to the display;
and the display displays the color image marked with the bone key points.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910595552.4A CN110460824B (en) | 2019-07-03 | 2019-07-03 | Frame synchronization method of image data and camera |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910595552.4A CN110460824B (en) | 2019-07-03 | 2019-07-03 | Frame synchronization method of image data and camera |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110460824A CN110460824A (en) | 2019-11-15 |
| CN110460824B true CN110460824B (en) | 2022-10-11 |
Family
ID=68481977
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910595552.4A Active CN110460824B (en) | 2019-07-03 | 2019-07-03 | Frame synchronization method of image data and camera |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110460824B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112449172B (en) * | 2021-02-01 | 2021-04-23 | 南京爱奇艺智能科技有限公司 | VR equipment multi-camera exposure synchronization method |
| CN112751983B (en) * | 2021-04-02 | 2021-06-18 | 湖北亿咖通科技有限公司 | Image time synchronization method and device, electronic equipment and storage medium |
| CN113411499B (en) * | 2021-06-17 | 2023-03-24 | 浙江大华技术股份有限公司 | Image fusion method and device, storage medium and electronic device |
| CN116859407A (en) * | 2022-03-24 | 2023-10-10 | 北京小米移动软件有限公司 | Multi-sensor fusion system and autonomous mobile device |
| CN116847059A (en) * | 2022-03-24 | 2023-10-03 | 北京小米移动软件有限公司 | Depth camera, depth image acquisition device and multi-sensor fusion system |
| CN115883815A (en) * | 2022-10-28 | 2023-03-31 | 珠海视熙科技有限公司 | Image data output method and device, lower computer and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9626803B2 (en) * | 2014-12-12 | 2017-04-18 | Qualcomm Incorporated | Method and apparatus for image processing in augmented reality systems |
| CN107610041A (en) * | 2017-08-16 | 2018-01-19 | 南京华捷艾米软件科技有限公司 | Video portrait based on 3D body-sensing cameras scratches drawing method and system |
| CN108419017A (en) * | 2018-04-28 | 2018-08-17 | Oppo广东移动通信有限公司 | Control method, apparatus, electronic equipment and the computer readable storage medium of shooting |
| CN109803087A (en) * | 2018-12-17 | 2019-05-24 | 维沃移动通信有限公司 | A kind of image generating method and terminal device |
-
2019
- 2019-07-03 CN CN201910595552.4A patent/CN110460824B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9626803B2 (en) * | 2014-12-12 | 2017-04-18 | Qualcomm Incorporated | Method and apparatus for image processing in augmented reality systems |
| CN107610041A (en) * | 2017-08-16 | 2018-01-19 | 南京华捷艾米软件科技有限公司 | Video portrait based on 3D body-sensing cameras scratches drawing method and system |
| CN108419017A (en) * | 2018-04-28 | 2018-08-17 | Oppo广东移动通信有限公司 | Control method, apparatus, electronic equipment and the computer readable storage medium of shooting |
| CN109803087A (en) * | 2018-12-17 | 2019-05-24 | 维沃移动通信有限公司 | A kind of image generating method and terminal device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110460824A (en) | 2019-11-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110460824B (en) | Frame synchronization method of image data and camera | |
| US20150271469A1 (en) | Image synchronization method for cameras and electronic apparatus with cameras | |
| CN110399110B (en) | Multi-screen synchronous display method and system, display equipment and storage medium | |
| US20180308253A1 (en) | Method and system for time alignment calibration, event annotation and/or database generation | |
| US20210274064A1 (en) | Systems and methods for synchronizing sensor capture | |
| CN102474570B (en) | Information display device, display control integrated circuit and display control method | |
| CN110312056B (en) | Synchronous exposure method and image acquisition equipment | |
| CN103999455B (en) | The crossover-platform video that cooperates is caught | |
| CN113811920A (en) | Distributed pose estimation | |
| US9392322B2 (en) | Method of visually synchronizing differing camera feeds with common subject | |
| CN112154669B (en) | Correlation of video stream frame timestamps based on system clock | |
| EP2693393A1 (en) | Information processing device, information processing method, and data structure of location information | |
| US9628777B2 (en) | Method of 3D reconstruction of a scene calling upon asynchronous sensors | |
| WO2019047606A1 (en) | Method and fpga for realizing synchronization of video frame data and gps time position | |
| US11238604B1 (en) | Densifying sparse depth maps | |
| CN108665522A (en) | A kind of short delay dynamic scene simulation generation system and method for high frame frequency | |
| CN111343401B (en) | Frame synchronization method and device | |
| WO2021070814A1 (en) | Synchronization device, synchronization method, and synchronization program | |
| KR100824016B1 (en) | Systme and method for executiing synchronization of output data created in asynchronous dual camera | |
| CN111277896A (en) | Method and device for splicing network video stream images | |
| US11694409B1 (en) | Augmented reality using a split architecture | |
| JP2019140483A (en) | Image processing system, image processing system control method, transmission device, transmission method, and program | |
| CN113556366A (en) | Multi-sensor data synchronization method and system and electronic equipment | |
| CN113296723A (en) | Method, system, device and medium for synchronously displaying target frame and video frame | |
| CN107534736A (en) | Method for registering images, device and the terminal of terminal |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |