WO2024060763A1 - 一种无线智能可穿戴装置及其图像采集方法 - Google Patents
一种无线智能可穿戴装置及其图像采集方法 Download PDFInfo
- Publication number
- WO2024060763A1 WO2024060763A1 PCT/CN2023/103757 CN2023103757W WO2024060763A1 WO 2024060763 A1 WO2024060763 A1 WO 2024060763A1 CN 2023103757 W CN2023103757 W CN 2023103757W WO 2024060763 A1 WO2024060763 A1 WO 2024060763A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- camera
- wireless communication
- image acquisition
- clock
- trigger signal
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 54
- 238000004891 communication Methods 0.000 claims abstract description 169
- 239000004984 smart glass Substances 0.000 claims description 9
- 230000004044 response Effects 0.000 claims description 8
- 238000001514 detection method Methods 0.000 claims description 6
- 238000010276 construction Methods 0.000 claims description 5
- 230000001360 synchronised effect Effects 0.000 description 30
- 230000005540 biological transmission Effects 0.000 description 13
- 238000010586 diagram Methods 0.000 description 13
- 239000011521 glass Substances 0.000 description 6
- 238000012545 processing Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000002411 adverse Effects 0.000 description 3
- 239000013078 crystal Substances 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000004927 fusion Effects 0.000 description 3
- 230000014509 gene expression Effects 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000001960 triggered effect Effects 0.000 description 3
- 108010076504 Protein Sorting Signals Proteins 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 101000827703 Homo sapiens Polyphosphoinositide phosphatase Proteins 0.000 description 1
- 102100023591 Polyphosphoinositide phosphatase Human genes 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000001364 causal effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 230000002427 irreversible effect Effects 0.000 description 1
- 230000000704 physical effect Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/04—Synchronising
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
- G02B27/0176—Head mounted characterised by mechanical features
-
- G—PHYSICS
- G02—OPTICS
- G02C—SPECTACLES; SUNGLASSES OR GOGGLES INSOFAR AS THEY HAVE THE SAME FEATURES AS SPECTACLES; CONTACT LENSES
- G02C11/00—Non-optical adjuncts; Attachment thereof
- G02C11/10—Electronic devices other than hearing aids
-
- G—PHYSICS
- G04—HOROLOGY
- G04B—MECHANICALLY-DRIVEN CLOCKS OR WATCHES; MECHANICAL PARTS OF CLOCKS OR WATCHES IN GENERAL; TIME PIECES USING THE POSITION OF THE SUN, MOON OR STARS
- G04B47/00—Time-pieces combined with other articles which do not interfere with the running or the time-keeping of the time-piece
-
- G—PHYSICS
- G04—HOROLOGY
- G04G—ELECTRONIC TIME-PIECES
- G04G21/00—Input or output devices integrated in time-pieces
-
- G—PHYSICS
- G04—HOROLOGY
- G04G—ELECTRONIC TIME-PIECES
- G04G21/00—Input or output devices integrated in time-pieces
- G04G21/04—Input or output devices integrated in time-pieces using radio waves
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04J—MULTIPLEX COMMUNICATION
- H04J3/00—Time-division multiplex systems
- H04J3/02—Details
- H04J3/06—Synchronising arrangements
- H04J3/0635—Clock or time synchronisation in a network
- H04J3/0638—Clock or time synchronisation among nodes; Internode synchronisation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/76—Television signal recording
- H04N5/765—Interface circuits between an apparatus for recording and another apparatus
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
- H04W4/30—Services specially adapted for particular environments, situations or purposes
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
- G02B2027/0178—Eyeglass type
Definitions
- the present application relates to a wireless device and an image acquisition method, and more specifically, to a wireless intelligent wearable device and an image acquisition method thereof.
- Smart wearable devices such as smart glasses, smart watches, etc., are gradually entering people's work and life. These smart wearable devices are wirelessly connected to each other and to other smart devices (such as mobile phones, pads, personal computers, multimedia TVs, etc.).
- these smart wearable devices are usually equipped with multiple cameras at multiple locations (such as the left and right eye parts of smart glasses) and provide various video-related functions, such as augmented reality, virtual reality, etc. Reality, panoramic view, etc. These functions require multiple cameras to capture video or images simultaneously.
- the current video or image collection of each camera has the problem of insufficient synchronization. For example, if the videos or images collected by multiple cameras are spliced to generate a panoramic video, defects such as image blur and motion ghosting will occur, affecting User experience.
- the purpose of this application is to provide a wireless smart wearable device and an image acquisition method thereof, which can enable multiple cameras arranged in multiple divisions of the wireless smart wearable device to achieve image shooting and acquisition in a hardware-triggered manner. Dynamic precise synchronization.
- a wireless smart wearable device includes a first part and a second part capable of wireless communication with each other.
- the first part includes a first processor, a first wireless communication module, a first camera and a first image acquisition module and has a first clock;
- the second part includes a second processor, a second wireless communication module, a first Two cameras and a second image acquisition module and a second clock.
- the first image acquisition module is configured to trigger the first camera to capture a first image and obtain a first image by sending a first hardware trigger signal based on the first clock to the first camera;
- the second image acquisition module is configured to trigger the second camera to capture a second image and obtain a second image by sending a second hardware trigger signal based on the second clock to the second camera.
- the first processor and the second processor At least one is configured to: during continuous use of the first image acquisition module and the second image acquisition module, the first wireless communication module and the second wireless communication module perform wireless communication with each other and/or Or both wirelessly communicate with the smart device, and determine the clock difference between the first wireless communication module and the second wireless communication module to perform the wireless communication, and the clock difference is used to implement the first hardware trigger signal and synchronization with the second hardware trigger signal.
- an image acquisition method for a wireless smart wearable device includes a first part and a second part that can communicate wirelessly with each other.
- the first part includes a first processor, a first wireless communication module, a first camera and a first image acquisition module and has a first clock;
- the second part includes a second processor, a second wireless communication module, a first Two cameras and a second image acquisition module and a second clock.
- the image acquisition method includes the following steps.
- the first wireless communication module and the second wireless communication module are caused to perform wireless communication with each other and/or both with intelligent wireless communication of the device, and determine the clock difference of each of the first wireless communication module and the second wireless communication module for performing the wireless communication.
- the first image acquisition module sends a first hardware trigger signal based on the first clock to the first camera to trigger the first camera to capture the first image and acquire the first image.
- the second image acquisition module sends a second hardware trigger signal based on the second clock to the second camera to trigger the second camera to capture a second image and obtain a second image.
- the clock difference is utilized to achieve synchronization of the first hardware trigger signal and the second hardware trigger signal.
- the wireless smart wearable device Utilizing the wireless smart wearable device and its image acquisition method according to the present application, it can use hardware triggering to enable multiple cameras arranged in multiple divisions of the wireless smart wearable device to achieve dynamic and accurate image capture and collection. Synchronize.
- Figure 1 shows a schematic structural diagram of a wireless smart wearable device according to the first embodiment of the present application
- Figure 2 shows a schematic diagram of a synchronous control method for image shooting and collection of different cameras in a wireless smart wearable device according to the second embodiment of the present application
- Figure 3 shows a schematic diagram of a synchronous control method for image shooting and collection of different cameras in a wireless smart wearable device according to the third embodiment of the present application
- Figure 4 shows a schematic diagram of a synchronous control method for image shooting and collection of different cameras in a wireless smart wearable device according to the fourth embodiment of the present application
- Figure 5 shows a schematic diagram of a synchronous control method for image shooting and collection of different cameras in a wireless smart wearable device according to the fifth embodiment of the present application
- Figure 6 shows a timing diagram of hardware trigger signals used to trigger different cameras to capture and collect images according to the sixth embodiment of the present application
- Figure 7 shows a flow chart of an image acquisition method for a wireless smart wearable device according to the seventh embodiment of the present application.
- Figure 8 shows a flow chart of an image acquisition method for a wireless smart wearable device according to the eighth embodiment of the present application.
- Figure 9 shows a timing diagram of an SOT or EOT signal according to the ninth embodiment of the present application.
- Figure 10 shows a schematic structural diagram of a wireless smart wearable device according to the tenth embodiment of the present application.
- Figure 11 shows a flow chart of an image acquisition method of a wireless smart wearable device according to the eleventh embodiment of the present application.
- first, second and third in this application are only intended to make descriptive distinctions, and do not imply limitations on quantity or sequence, nor are they intended to limit the expressions immediately following the expression “first”. “,””second” and “third” after the “second” and “tertiary” components, devices and systems in the physical properties of the difference.
- a “first system on a chip” may include a system implemented on a single chip, or may include system(s) implemented on multiple chips.
- FIG. 1 shows a schematic structural diagram of a wireless smart wearable device according to the first embodiment of the present application.
- the wireless smart wearable device may include a first part 101a and a second part 101b that can wirelessly communicate with each other.
- a wireless smart glasses device is shown in FIG. 1 as an example of a wireless smart wearable device.
- One of the first part 101a and the second part 101b is a left glasses part and the other is a right glasses part, but This is only an example.
- the wireless smart wearable device can adopt other structures, and can even be formed into components, including two or more separate components (devices), as long as each component is equipped with its own camera and needs to synthesize images or videos from the respective cameras.
- the wireless smart wearable device may include a wireless smart helmet, a wireless smart bracelet, etc., or may include components of a wireless smart necklace and a wireless smart bracelet, etc.
- the following description takes a wireless smart glasses device as an example.
- the following description can also be flexibly applied to wireless smart wearable devices with other structures, and will not be described again here.
- the first part 101a includes a first processor 102a, a first wireless communication module 103a, a first camera 105a and a first image acquisition module 104a and has a first clock 106a.
- the second part 101b includes a second processor 102b, a second wireless communication module 103b, a second camera 105b and a second image acquisition module 104b and has a second clock 106b.
- the first camera 105a and the second camera 105b are respectively disposed on the left and right sides of the upper beam of the glasses frame, so that the captured image/video can contain the surrounding environment information within the user's perspective of interest as much as possible.
- the first processor 102a, the first wireless communication module 103a, the first image acquisition module 104a and the first clock 106a are built into the temple of the first part 101a (shown as the right eye part in Figure 1)
- the second processor 102b, the second wireless communication module 103b, the second image acquisition module 104b and the second clock 106b are built in the temple of the second part 101b (shown as the left eyeglass part in Figure 1).
- the first part 101a and the second part 101b are connected wirelessly, and the first clock 106a and the second clock 106b belong to two parts. Although they may have the same nominal frequency, they are each based on different crystals or crystal oscillators. Or crystal oscillators often have a certain frequency offset, such as within 10ppm, 5ppm, or 1ppm.
- the starting values of the timers of the first clock 106a and the second clock 106b may also deviate.
- the first clock 106a may belong to the first wireless communication module 103a, and the second clock 106b may belong to the second wireless communication module 103b.
- the first wireless communication module 103a is a Bluetooth communication module 103a, its own Bluetooth clock can be used as the first clock 106a.
- the first clock 106a can be represented by a first clock counter or a first clock counter; the second clock 106b can be represented by a second clock counter or a second clock counter. Since the counting start time of the first clock counter and the second clock counter may be different, and even the initial value may be different, and the first clock 106a and the second clock 106b may have a frequency deviation, the first clock counter and the second clock counter are often different at the same time. For example, at a certain time, the value of the first clock counter may be 20, and the value of the second clock counter may be 24.
- the first processor 102a, the first wireless communication module 103a, the first image acquisition module 104a and the first clock 106a are implemented on the same chip , also said to be implemented as the same system-on-chip (hereinafter referred to as the first system-on-chip), and the first camera 105a is implemented as another independent chip (hereinafter referred to as the second chip). That is to say, the first camera 105a has an independent clock (timer). Even if the set shooting and collection time is known, the frequency offset between the independent clocks causes the timing difference between the first system-on-chip and the second chip to be irreversible to each other. control. This will further cause the image capture and collection of the first camera 105a and the second camera 105b to be unable to be accurately synchronized.
- SOC System-on-a-chip
- various RISC (reduced instruction set computer) processor IPs purchased from ARM and others can be used as the processor of the SOC to perform corresponding functions, and thus can be implemented as an embedded system.
- IP modules
- modules such as but not limited to memory, various communication modules (such as WiFi communication module, Bluetooth communication module, etc.), image acquisition module, buffer, Clock etc.
- chip manufacturers can also develop customized versions of these modules based on off-the-shelf IP.
- other components such as antennas, sensor components, speakers, microphones, etc. can be externally connected to the IP. Users can implement various communication modules, image acquisition modules, etc.
- ASICs Application Specific Integrated Circuits
- FPGA Field Programmable Gate Array
- the first image acquisition module 104a is configured to trigger the first camera 105a to capture a first image and obtain a first image by sending a first hardware trigger signal S1 based on the first clock 106a to the first camera 105a. image.
- the second image acquisition module 104b is configured to trigger the second camera 105b to capture a second image and obtain a second image by sending a second hardware trigger signal S2 based on the second clock 106b to the second camera 105b. image.
- At least one of the first processor 102a and the second processor 102b is configured to: during the continuous use of the first image acquisition module 104a and the second image acquisition module 104b, the A wireless communication module 103a and the second wireless communication module 103b perform wireless communication with each other (as shown in the communication signal S3) and/or wireless communication with the smart device (as shown in the communication signal S4), and determine the The first wireless communication module 103a and the second wireless communication module 103b each perform a clock difference of the wireless communication, and the dynamic clock difference can be used to implement the first hardware trigger signal S1 and the second hardware trigger. Synchronization of signal S2.
- the so-called "clock difference" can be implemented as time2-time1 or time4-time3. Please refer to the detailed description in the following embodiments.
- At least one of the first processor 102a and the second processor 102b may be configured to: during continuous use of the first image acquisition module 104a and the second image acquisition module 104b , so that one of the first wireless communication module 103a and the second wireless communication module 103b sends the wireless signal S3 to the other party.
- the difference time2-time1 between the value time1 of the clock counter at the first time when the one party sends the wireless signal S3 and the value time2 of the clock counter at the second time when the other party receives the wireless signal S3 can be determined. .
- the air time for both parties to send and receive the wireless signal S3 can usually be ignored, so the numerical difference time2-time1 is determined by the difference between the initial value and the initial counting time of the first clock timer and the second clock timer, as well as the first clock 106a and the second clock timer. Caused by the frequency offset between the two clocks 106b.
- time2-time1 will be the same fixed value or an approximately fixed value.
- the difference between time2 and time1 can be used to adjust the second clock 106b, so that the first clock 106a and the second clock 106b are synchronized, so that time2 and time1 maintain the same fixed value or approximately a fixed value.
- At least one of the first processor 102a and the second processor 102b is configured to: during continued use of the first image acquisition module 104a and the second image acquisition module 104b , so that the first wireless communication module 103a and the second wireless communication module 103b each receive the wireless signal S4 from the smart device; determine the third moment when the first wireless communication module 103a receives the wireless signal S4. clock timer value time3 and the The second wireless communication module 103b receives the value time4 of the clock counter at the fourth moment of the wireless signal S4. The air time deviation of the wireless signal S4 received by both parties from the smart device can usually be ignored.
- the numerical difference time4-time3 is also determined by the difference between the initial value and the initial counting time of the first clock timer and the second clock timer, and the first clock 106a and the second clock 106b.
- the first wireless communication module 103a can be used to receive the wireless signal S4 from the smart device to achieve synchronization between the first clock 106a and the smart device wireless clock;
- the first wireless communication module 103b can be used to receive The wireless signal S4 from the smart device realizes synchronization between the second clock 106b and the wireless clock of the smart device; thereby realizing time synchronization between the first clock 106a and the second clock 106b; thus also keeping time3-time4 at the same fixed value or approximately fixed value.
- At least one of the first processor 102a and the second processor 102b may utilize the numerical difference to achieve synchronization of the first hardware trigger signal S1 and the second hardware trigger signal S2.
- the numerical difference (the numerical difference between time4-time3 or time2-time1 that can be dynamically changed) can be considered and compensated when sending the first hardware trigger signal S1 and the second hardware trigger signal S2 respectively, so that the first hardware Dynamic precise synchronization of the trigger signal S1 and the second hardware trigger signal S2.
- the first wireless communication module 103a and the second wireless communication module 103b can adopt various wireless communication methods, such as but not limited to Bluetooth module, WiFi module, UWB module, etc.
- Bluetooth module through the related processing of the Bluetooth access code or part of the access code at the physical layer, the Bluetooth clock of the first Bluetooth module 101a can be synchronized with the Bluetooth clock of the second Bluetooth module 101b.
- the Bluetooth clock is in Bluetooth mode. wireless transceiver clock.
- the WiFi device will receive the beacon sent by the access device at the same time (such as every interval of 50ms, 102.4ms, 500ms, etc.).
- the WiFi device can use the time when the beacon is received as the WiFi clock. , that is, the wireless transceiver clock in WiFi mode.
- the wireless transceiver clock of the first part 101a can be used as the first clock 106a; the wireless transceiver clock of the second part 101b can be used as the second clock 106b.
- FIG. 3 shows a schematic diagram of a synchronous control method for image capturing and collection by different cameras in a wireless smart wearable device according to the third embodiment of the present application.
- the first image acquisition module 104a and the first camera 105a are connected through the first (pair) CSI interfaces CSI1a-CSI1b and the first (pair) GPIO interfaces GPIO1a-GPIO1b
- the second image acquisition module 104b and The second camera 105b is connected to the second (pair) GPIO interfaces GPIO2a-GPIO2b through the second (pair) CSI interfaces CSI2a-CSI2b.
- the first image acquisition module 104a is further configured to be connected to the first camera 105a via the first GPIO interfaces GPIO1a-GPIO1b to send the first hardware trigger signal S1 to the first camera 105a.
- the first camera 105a is further configured to: in response to receiving the first hardware trigger signal S1, start exposure and image capture and transmit images to the first image acquisition module 104a via the first CSI interface CSI1a-CSI1b S5.
- the second image acquisition module 104b is further configured to connect to the second camera 105b via the second GPIO interface GPIO2a-GPIO2b to send the second hardware trigger signal S2 to the second camera 105b.
- the second camera 105b is further configured to start exposure and image capture in response to receiving the second hardware trigger signal S2 and transmit the image S6 to the second image acquisition module 104b via the second CSI interface CSI2a-CSI2b.
- CSI also known as camera serial interface
- GPIO also known as general-purpose input/output
- each image acquisition module can be connected to the camera through the CSI interface and GPIO interface.
- the left and right glasses parts 101a and 101b can send signals to the first camera 105a and the second camera 105b simultaneously from the first image acquisition module 104a and the second image acquisition module 104b through the GPIO port.
- Output hardware trigger signals S1 and S2 to start image acquisition at the same time.
- the first The image acquisition module 104a and the second image acquisition module 104b simultaneously output hardware trigger signals S1 and S2 to the first camera 105a and the second camera 105b respectively, thereby correcting the subsequent acquisition of the left and right glasses caused by the frequency offset of the pixel clocks of the left and right glasses. Images are out of sync.
- the first camera 105a and the second camera 105b respectively transmit the video images S5 and S6, and causes the left and right eyeglass portions 101a and 101b to obtain each frame of image with the same nominal number of pixel clocks.
- the so-called triggering of the corresponding camera's shooting (exposure) and the return of the video image (image acquisition) by sending a hardware trigger signal means that the shooting and return actions are triggered by receiving the hardware trigger signal (there is a causal relationship), but the shooting and return time can be different from the reception time of the hardware trigger signal (not necessarily at the same time), at least after the reception time.
- taking the first camera 105a as an example after the first camera 105a receives the first hardware trigger signal S1, it can start exposure after a predetermined delay.
- the predetermined delay is greater than the time offset caused by the frequency deviation of the above-mentioned interval N frames, so as to ensure that the first camera 105a has not started exposure when it receives the first hardware trigger signal S1 after the interval N frames. This avoids the first camera 105a receiving the first hardware trigger signal S1 during the exposure process, which causes frame loss or image errors in the frame.
- FIG. 4 shows a schematic diagram of a synchronous control method for image capturing and collection by different cameras in a wireless smart wearable device according to the fourth embodiment of the present application.
- FIG. 5 shows a schematic diagram of a synchronous control method for image capturing and collection by different cameras in a wireless smart wearable device according to the fifth embodiment of the present application.
- the following is a detailed description of the synchronization control methods of image shooting and collection of different cameras with reference to Figures 4 and 5 respectively.
- the first processor 102a can be further configured to generate the first hardware trigger signal S1 when the value of the clock counter of the first part 101a is the first predetermined value t1; the second processor 102b can be further configured to generate the second hardware trigger signal S2 when the value of the clock counter of the second part 101b is the second predetermined value t1+ ⁇ t.
- the difference ⁇ t between the first predetermined value t1 and the second predetermined value t1+ ⁇ t is set based on the above-mentioned value difference time4-time3 or time2-time1, so that the first predetermined value t1 and the second predetermined value t1+ ⁇ t represent substantially the same time for the clock timer of the first clock 106a and the clock timer of the second clock 106b, respectively.
- the difference ⁇ t can be set to time4-time3 or time2-time1, and updated with the dynamic change of time4-time3 or time2-time1.
- the first hardware trigger signal S1 and the second hardware trigger signal S2 can be generated substantially at the same time, thereby realizing the synchronous triggering of image capture and acquisition of the first camera 105a and the second camera 105b.
- the first processor 102a may be further configured to: generate a reference hardware trigger signal (not shown) and obtain the reference value t0 of the clock counter of the first part 101a at the trigger time; so that the first wireless The communication module 103a transmits the reference value t0 to the second wireless communication module 103b; and generates the first hardware trigger signal S1 after a predetermined delay td after the trigger time t0.
- the second processor 102b is further configured to: based on the reference value t0, the predetermined time delay td and the value difference ⁇ t of the clock counter (such as time4-time3 or time2-time1), to determine the value of the clock counter used to generate the second hardware trigger signal S2, such as t0+td+ ⁇ t; when the value of the clock counter of the second part 101b reaches a determined value (for example, t0+td+ ⁇ t), the second hardware trigger signal S2 is generated, so that the first hardware trigger signal S1 and the The second hardware trigger signal S2 is generated substantially at the same time, thereby achieving synchronous triggering of image capture and collection by the first camera 105a and the second camera 105b.
- the predetermined time delay td and the value difference ⁇ t of the clock counter such as time4-time3 or time2-time1
- the second hardware trigger signal S2 is generated, so that the first hardware trigger signal S1 and the The second hardware trigger signal S2 is generated substantially at the same time,
- the predetermined time delay td may be greater than the time offset caused by the frequency offset of the above-mentioned N frames to ensure that the first camera 105a (second camera 105b) receives the first hardware after an interval of N frames.
- the trigger signal S1 the second hardware trigger signal S2
- the exposure has not yet started. This prevents the first camera 105a (the second camera 105b) from receiving the first hardware trigger signal S1 (the second hardware trigger signal S2) during the exposure process, causing frame loss or frame image errors.
- the time period of the predetermined delay td can also be used by the first wireless communication module 103a to transmit the reference value t0 or t0+td to the second wireless communication module 103b.
- the first part 101a and the second part 101b each include a brightness detection unit 107a and 107b configured to detect the brightness of the ambient light of the corresponding first camera 105a and the second camera 105b.
- a brightness detection unit 107a and 107b configured to detect the brightness of the ambient light of the corresponding first camera 105a and the second camera 105b.
- At least one of the first processor 102a and the second processor 102b is further configured to: based on the brightness respectively detected by the first part 101a and the second part 101b, set the first camera 105a and the The exposure time of the second camera 105b is such that the lower the detected brightness corresponding to the camera is, the longer the exposure time is. That is to say, if the brightness of the ambient light detected by the brightness detection unit 107a of the first part 101a is lower, the exposure time of the first camera 105a is longer to make up for the lack of brightness, so that the first camera 105a exposes the captured image.
- /Video has consistent brightness with the image/video exposed and captured by the second camera 105b, thereby helping to improve the quality of subsequent synthesized (fusion) images.
- At least one of the first processor 102a and the second processor 102b may be further configured to: for the first camera 105a or the second camera 105b to set a longer exposure time, use Compared with the image acquisition module connected to another camera, the first image acquisition module 104a or the second image acquisition module 104b connected thereto generates and sends the corresponding hardware trigger signal S1 or S2 at a predetermined time in advance.
- the advance predetermined time can be set according to the excess ⁇ tp.
- the setting method can be adapted according to the setting rules of the time points to which the images collected during the entire exposure period belong.
- the collected image can be used as an image at an intermediate time point in the exposure period.
- the excess of the exposure time of the first camera 105a compared to the second camera 105b is ⁇ tp, as shown in FIG. 6
- the timing of the first hardware trigger signal S1 is earlier than the timing of the second hardware trigger signal S2
- the excess of the exposure time is approximately half, that is, approximately 1/2 ⁇ tp is advanced. In this way, it can not only make up for the lack of brightness of the first camera 105a, but also keep the images collected by the first camera 105a and the second camera 105b during each other's exposure time periods synchronized without being affected by the longer duration of the exposure time period. .
- the first image and the second image can be used for fusion to meet the increasing various needs of users.
- at least one of the first processor 102a and the second processor 102b may be further configured to utilize the first image and the second image for generating a panoramic video or synchronizing positioning and map construction.
- FIG. 7 shows a flowchart of an image acquisition method for a wireless smart wearable device according to the seventh embodiment of the present application.
- the wireless smart wearable device may, for example, adopt a structure as shown in Figures 1 and 2 but is not limited thereto. It includes a first part and a second part that can communicate wirelessly with each other, and the first part includes a first processor. , a first wireless communication module, a first camera and a first image acquisition module and having a first clock, the second part includes a second processor, a second wireless communication module, a second camera and a second image acquisition module and having Second clock.
- the first clock and the second clock are independent of each other and may have frequency offsets; each camera may be set on an independent chip or may be set on the same on-chip system with other components, which is not limited here.
- the image acquisition method may include the following steps.
- step 701 during the continuous use of the first image acquisition module and the second image acquisition module, the first wireless communication module and the second wireless communication module are caused to perform wireless communication with each other and/or The two wirelessly communicate with the smart device, and determine the clock difference for each of the first wireless communication module and the second wireless communication module to perform the wireless communication.
- the first image acquisition module may send a first hardware trigger signal based on the first clock to the first camera to trigger the first camera to capture a first image and acquire the first image.
- the second image acquisition module 104b sends a second hardware trigger signal based on the second clock 106b to the second camera 105b to trigger the second camera 105b to capture a second image and obtain a second image. Two images.
- step 702 and step 703 are parallel processing steps controlled by the first image acquisition module and the second image acquisition module respectively.
- the first hardware trigger signal and the second hardware trigger signal are synchronized using the dynamic clock difference determined in step 701 (that is, the value difference of the clock counter of the transmission and reception timing), for example, but not limited to, the The numerical difference in the corresponding clock counter values of the first hardware trigger signal and the second hardware trigger signal is compensated, so that the first hardware trigger signal and the second hardware trigger signal are sent at substantially the same time.
- the frequency offset of the first clock and the second clock changes dynamically, it can be dynamically captured and compensated in time and accurately, and then the image shooting and collection of the first camera and the second camera can be started at the same time without considering The frequency deviation of the clocks on the first and second independent chips where the first camera and the second camera are located.
- the captured first image and the second image can maintain precise dynamic synchronization and can be used for fusion to meet the increasing various needs of users, such as for generating panoramic videos or synchronized positioning and map construction.
- FIG. 8 shows a flow chart of an image acquisition method of a wireless smart wearable device according to the eighth embodiment of the present application.
- the first image acquisition module is connected to the first camera via a first GPIO interface to send the first hardware trigger signal to the first camera.
- the first camera in response to receiving the first hardware trigger signal, the first camera starts exposure and image capturing and transmits images to the first image acquisition module via the first CSI interface.
- the second image acquisition module is connected to the second camera via a second GPIO interface to send the second hardware trigger signal to the second camera.
- the second camera in response to receiving the second hardware trigger signal, the second camera starts exposure and image capturing and transmits images to the second image acquisition module via the second CSI interface.
- each image acquisition module can be connected to the camera through the CSI interface and GPIO interface.
- the clock difference for each of the first wireless communication module and the second wireless communication module to perform the wireless communication may be determined through the following steps.
- the first image taken During the continuous use of the collection module and the second image acquisition module, one of the first wireless communication module and the second wireless communication module sends a wireless signal to the other party; it is determined that the one party sends the wireless signal
- the numerical difference of the clock counter between the first moment of the signal and the second moment when the other party receives the wireless signal is used as the clock difference.
- the wireless signal includes a signal sequence known to the receiver. For example, for a Bluetooth module, it may be a Bluetooth access code of the physical layer; for example, for a WIFI module, it may be a beacon signal.
- the clock difference for each of the first wireless communication module and the second wireless communication module to perform the wireless communication may be determined through the following steps.
- the first wireless communication module and the second wireless communication module each receive wireless signals from the smart device; determining that the The numerical difference of the clock counter between the third time when the first wireless communication module receives the wireless signal and the fourth time when the second wireless communication module receives the wireless signal is used as the clock difference.
- the wireless signal includes a signal sequence known to the receiver. For example, for a Bluetooth module, it may be a Bluetooth access code of the physical layer; for example, for a WIFI module, it may be a beacon signal.
- Part 1-Part 2 mutual communication and Part 1/Part 2-joint communication of smart devices are routine communications and occur continuously for wireless smart wearable devices such as smart glasses, thus dynamically determining The clock difference will not affect the user experience.
- the first hardware trigger signal may be generated when the value of the clock counter of the first part is a first predetermined value; and the second hardware trigger signal may be generated when the value of the clock counter of the second part is a second predetermined value.
- the difference between the first predetermined value and the second predetermined value is set based on the value difference so as to represent the same moment.
- a reference hardware trigger signal can be generated in the first part, and the reference value of the clock counter of the first part can be obtained at the trigger time; and the reference value can be transmitted to the second wireless communication module through the first wireless communication module.
- Module generate the first hardware trigger signal after a predetermined delay after the trigger moment; in the second part, determine the value used to generate the reference value, the predetermined delay and the value difference of the clock counter.
- the value of the clock counter of the second hardware trigger signal when the value of the clock counter of the second part reaches the determined value, the second hardware trigger signal is generated, so that the first hardware trigger signal and the third Two hardware trigger signals are generated at the same time.
- the image acquisition method may also include detection and compensation equalization of the brightness of ambient light of each camera.
- the brightness of the ambient light of the first camera and the second camera can be detected; the exposure time of the first camera and the second camera is set so that the lower the brightness of the ambient light, the longer the exposure time of the corresponding camera.
- the image acquisition method may also include: for the first camera or the second camera to be set with a longer exposure time, the image acquisition module connected to it is scheduled in advance compared to the image acquisition module connected to the other camera. Time to generate and send the corresponding hardware trigger signal.
- the predetermined time advanced is approximately half of the excess of the exposure time.
- the wireless smart wearable device of this modification can adopt the hardware configuration shown in Figure 10, in which the first clock 106a and the second clock 106b are independent of each other and also include the function of a clock counter. To simplify the description, they are respectively referred to as the first clock 106a and the second clock 106b in the following.
- a clock counter 106a and a second clock counter 106b are respectively referred to as the first clock 106a and the second clock 106b in the following.
- a clock counter 106a and a second clock counter 106b A difference from the hardware configuration shown in Figure 2 is that this variant does not rely on the GPIO interface and does not need to send hardware trigger signals from the chip on the image acquisition module side to the chip on the camera side. Instead, it uses each frame of image captured by a certain camera.
- the following description takes a wireless smart glasses device as an example of a wireless smart wearable device, and uses the first image captured by the first camera 105a of the first part 101a as a reference image.
- the wireless smart wearable device can adopt other structures.
- the first part 101a and the second part 101b can switch between the left spectacle part and the right spectacle part.
- the first camera 105a captures the first image and the collected image.
- the second image captured by the second camera 105b can be switched and used as a reference image.
- the first image acquisition module 104a is configured to interconnect with the first camera 105a through the first CSI interface CSI1a-CSI1b and acquire the first image of each frame captured by the first camera 105a S5 as reference image.
- the second image acquisition module 104b is configured to To: interconnect with the second camera 105b through the second CSI interface CSI2a-CSI2b and obtain the second image S6 of each frame captured by the second camera 105b.
- the first processor 102a is configured to obtain a first value of the first clock counter 106a when the first image acquisition module 104a receives a SOT or EOT signal for transmitting each frame of the first image from the first CSI interface, or after a preset time delay.
- the second processor 102b is configured to obtain a second value of the second clock counter 106b when the second image acquisition module 104b receives a SOT or EOT signal for transmitting each frame of the second image from the second CSI interface, or after the preset time delay.
- the first value of the first clock counter 106a and the second value of the second clock counter 106b may be based on the count values of the first clock counter 106a and the second clock counter 106b at the same time. Differences are used to make compensation adjustments, and then subsequent analysis is performed. That is to say, the value of the first clock counter 106a (such as but not limited to the first value) and the corresponding value of the second clock counter 106b (such as but not limited to the second value) used for interpolation processing below can be Compensation adjustment is performed using the numerical difference between the first clock counter 106a and the second clock counter 106b at the same time, so as to eliminate differences at the actual same time caused by starting count values and/or frequency offsets of the two clock counters. numerical difference. The numerical difference may be, for example, time2-time1 or time4-time3. Please refer to the detailed description in the following embodiments.
- At least one of the first processor 102a and the second processor 102b may be configured to: in the first image acquisition module 104a and the second image acquisition module 104b During continuous use, one of the first wireless communication module 103a and the second wireless communication module 103b is caused to send a wireless signal S3 to the other party (return to FIG. 2).
- the difference time2-time1 between the value time1 of the clock counter at the first time when the one party sends the wireless signal S3 and the value time2 of the clock counter at the second time when the other party receives the wireless signal S3 can be determined. .
- the air time for both parties to send and receive the wireless signal S3 can usually be ignored, so the numerical difference time2-time1 is determined by the difference between the initial value and the initial counting time of the first clock timer and the second clock timer, as well as the first clock 106a and the second clock timer. Caused by the frequency offset between the two clocks 106b.
- time2-time1 will be the same fixed value or an approximately fixed value.
- the difference between time2 and time1 can be used to adjust the second clock 106b, so that the first clock 106a and the second clock 106b are synchronized, so that time2 and time1 maintain the same fixed value or approximately a fixed value.
- At least one of the first processor 102a and the second processor 102b is configured to: during continued use of the first image acquisition module 104a and the second image acquisition module 104b , so that the first wireless communication module 103a and the second wireless communication module 103b each receive the wireless signal S4 from the smart device (return to FIG. 2); determine that the first wireless communication module 103a receives the wireless signal The value time3 of the clock timer at the third time of S4 and the value time4 of the clock counter at the fourth time when the second wireless communication module 103b receives the wireless signal S4. The air time deviation of the wireless signal S4 received by both parties from the smart device can usually be ignored.
- the numerical difference time4-time3 is also determined by the difference between the initial value and the initial counting time of the first clock timer and the second clock timer, and the first clock 106a and the second clock 106b.
- the first wireless communication module 103a can be used to receive the wireless signal S4 from the smart device to achieve synchronization between the first clock 106a and the smart device wireless clock;
- the first wireless communication module 103b can be used to receive The wireless signal S4 from the smart device realizes synchronization between the second clock 106b and the wireless clock of the smart device; thereby realizing time synchronization between the first clock 106a and the second clock 106b; thus also keeping time3-time4 at the same fixed value or approximately fixed value.
- Compensation adjustment can be made by using the numerical difference time2-time1 or time4-time3 based on the count values of the first clock counter 106a and the second clock counter 106b. Specifically, if time2 is the value of the clock timer at the first time when the second wireless communication module 103b sends the wireless signal S3, and time1 is the second time when the first wireless communication module 103a receives the wireless signal S3 The value of the clock counter, and time2>time1, then the dynamic count value of the second clock counter 106b can be subtracted (time2-time1), thereby eliminating the initial value and initial count of the first clock timer and the second clock timer. The adverse effects of time differences and frequency offset between the first clock 106a and the second clock 106b.
- the dynamic count value of the second clock counter 106b can be subtracted (time4-time3), thereby eliminating the difference between the initial value and the initial counting time of the first clock timer and the second clock timer and Adverse effects of frequency offset between first clock 106a and second clock 106b.
- the adverse effects of the difference between the initial value and the initial counting time of the first clock timer and the second clock timer and the frequency offset between the first clock 106a and the second clock 106b can be eliminated.
- receiving the SOT or EOT signal of the first CSI interface for transmitting the first image of each frame or after a preset delay time and receiving the SOT or EOT signal of the second CSI interface for transmitting each frame.
- the SOT or EOT signal of the second image is received or after the preset time delay
- the real time sequence relationship between the two is to facilitate better synchronization between the subsequent interpolated image and the reference image.
- FIG 9 it shows the timing diagram of the CSI interface connection defined according to the MIPI CSI-2 protocol when transmitting a data packet.
- the CSI interface is a one-way transmission, transmitted from the camera to the outside, including a clock lane (data lane) and one to four data lanes.
- the four data lanes are LANE 1, LANE 2, LANE 3 and LANE shown in Figure 9. 4.
- SOT also known as transmission start signal
- EOT also known as transmission end signal
- the first value and the second value essentially represent the first image and the second value.
- the payload (byte 0-byte N-1, N is a positive integer) is transmitted between the SOT and EOT signals.
- LPS state low power state
- the first processor 102a or the second processor 102b is further configured to cause the first wireless communication module 103a to transmit the first value to the second wireless communication module 103b, or to cause the first The wireless communication module 103a and the second wireless communication module 103b transmit the first value, the second value and the second image to the smart device, so that the second processor 102b or the smart device
- the third processor 102c obtains the first value, the second value together with the second image.
- the second processor 102b or the third processor 102c in the smart device that obtains the first numerical value, the second numerical value together with the second image is further configured to: based on using the The first numerical value and the second numerical value after the numerical difference between the first clock counter 106a and the second clock counter 106b at the same time are compensated and adjusted, and the second image of each frame is interpolated to obtain The first image of each frame is synchronized with the third image of each frame. Therefore, by using the conventional signal interaction method of the CSI interface, through simple interpolation operations, without the need for interactive configuration of other hardware interfaces, precise dynamic synchronization of each frame image of the first part 101a and the second part 101b can be easily achieved. This level of precise dynamic synchronization is such that the first image of each frame and the synchronized third image of each frame can be used to generate panoramic videos or to synchronize positioning and map construction.
- the clock difference between the first clock 106a and the second clock 106b can also be considered, and the first value and/or the second value can be compensated and adjusted accordingly. , and then used to interpolate the second image of each frame.
- the method of using the numerical difference between the first clock counter 106a and the second clock counter 106b at the same time to compensate and adjust the first numerical value and/or the second numerical value described in various embodiments of the present application can be achieved. Suitable for this.
- the numerical difference may be, for example, time2-time1 or time4-time3. Please refer to the specific descriptions in each embodiment.
- At least one of the first processor 102a and the second processor 102b is further configured to: during continued use of the first image acquisition module 104a and the second image acquisition module 104b, such that The first wireless communication module 103a and the second wireless communication module 103b perform wireless communication with each other and/or wireless communication with smart devices; calculations utilize the first wireless communication module 103a and the second wireless communication module 103a.
- the communication module 103b performs the wireless communication, the numerical difference between the first clock counter 106a and the second clock counter 106b; the first numerical value and/or the second numerical value is compensated based on the numerical difference. Adjustment is then used to interpolate the second image of each frame.
- the second processor 102b is further configured to obtain the second CSI interface CSI2a-CSI2b received by the second image acquisition module 104b for transmitting the N-1th frame and the Nth frame.
- the corresponding two second values of the second clock counter 106b are T0 and T2.
- the second processor 102b can obtain the SOT or EOT signal used to transmit the first image of the Nth frame when the first image acquisition module 104a receives the first CSI interface CSI1a-CSI1b, or after the pre-processing Assume that after a time delay, the corresponding first value T1 of the first clock counter is T1, where T2 is greater than or equal to T1. That is to say, the first value T1 of the timing corresponding to the first image of the Nth frame may be located between the two values T0 and T2 of the timing corresponding to the N-1th frame and the second image of the Nth frame.
- New_image represents the third image of the Nth frame that is synchronized with the first image of the Nth frame
- image_N_1 represents the second image of the N-1th frame
- image_N represents the second image of the Nth frame.
- the first portion 101a and the second portion 101b may each include a brightness detection unit 107a and 107b, which are configured to detect the brightness of the corresponding first camera 105a and the second camera 105b.
- At least one of the first processor 102a and the second processor 102b is further configured to set the exposure time of the first camera 105a and the second camera 105b based on the brightness detected by the first part 101a and the second part 101b, so that the lower the detected brightness of the camera, the longer the exposure time.
- the second image acquisition module 104b When the second image acquisition module 104b receives the SOT or EOT signal of the second CSI interface CSI2a-CSI2b for transmitting the second image of the Nth frame, or after the preset delay, the The corresponding second value T2 of the second clock counter 106b can be adjusted according to the difference in exposure time of the first camera 105a and the second camera 105b.
- the second processor 102b may be further configured to: obtain the first exposure time t00 of the first camera and the second exposure time t11 of the second camera; calculate according to the following formula (2)
- the adjusted second value T2′ of T2: T2′ T2-(t11-t00)/2, formula (2).
- the collected image can be used as an image at an intermediate time point in the exposure period.
- T2 is advanced by approximately half of the excess amount of exposure time t11-t00/2. In this way, it can not only make up for the lack of brightness of the second camera 105b, but also keep the images collected by the first camera 105a and the second camera 105b during each other's exposure time periods synchronized without being affected by the longer duration of the exposure time period.
- the above adjusted second value T2' can be substituted into T2 of formula (1) to interpolate to obtain the third image of the Nth frame that is synchronized with the first image of the Nth frame.
- FIG 11 shows a flow chart of an image acquisition method of a wireless smart wearable device according to the eleventh embodiment of the present application.
- the wireless smart wearable device includes a first part and a second part that communicate wirelessly with each other.
- the first part includes a first processor, a first wireless communication module, a first camera, a first image acquisition module and a first clock counter
- the second part includes a second processor, a second wireless communication module, a first Two cameras, a second image acquisition module and a second clock counter.
- the first clock counter and the second clock timer are independent of each other, and may have dynamic clock deviations from each other.
- the wireless smart wearable device can adopt the structure according to various embodiments of the present application, as long as it can collaboratively implement the process of the image acquisition method, which will not be described again here.
- the image acquisition method includes the following steps.
- the first image acquisition module is interconnected with the first camera through the first CSI interface, and acquires the first image of each frame captured by the first camera as a reference image.
- the second image acquisition module is interconnected with the second camera through the second CSI interface, and acquires second images of each frame captured by the second camera.
- the first processor obtains when the first image acquisition module receives the SOT or EOT signal of the first CSI interface for transmitting the first image of each frame, or after a preset delay. After that, the first value of the first clock counter.
- step 1104 the second processor obtains when the second image acquisition module receives the SOT or EOT signal of the second CSI interface for transmitting the second image of each frame, or after the pre-processing Assume that after the time delay, the second value of the second clock counter.
- the first wireless communication module transmits the first value to the second wireless communication module, or the first wireless communication module and the second wireless communication module transmit the first value to the smart device. the first value, the second value and the second image, so that the second processor or the third processor in the smart device obtains the first value, the second value together with the Second image.
- step 1106 the second processor or the third processor in the smart device that obtains the first numerical value, the second numerical value together with the second image, based on the first numerical value and the The second value is used to interpolate the second image of each frame to obtain the third image of each frame that is synchronized with the first image of each frame.
- the image acquisition method may further include: during the continuous use of the first image acquisition module and the second image acquisition module, the first wireless communication module and the second wireless communication module The modules perform wireless communication with each other and/or wireless communication with the smart device; calculate when the first wireless communication module and the second wireless communication module are used to perform the wireless communication, the first clock counter and the The numerical difference of the second clock counter; and the first numerical value and/or the second numerical value are compensated and adjusted based on the numerical difference, and then used to interpolate the second image of each frame.
- the image acquisition method further includes: utilizing the first image of each frame and the synchronized third image of each frame for generating a panoramic video or synchronizing positioning and map construction.
- the image acquisition method further includes, by the second processor: obtaining the N-1th frame and the Nth frame received by the second image acquisition module from the second CSI interface.
- the second processor receives the SOT or EOT signal of the second image of the frame, or after the preset time delay, the second The corresponding two second values T0 and T2 of the clock counter are obtained;
- the first image acquisition module receives the SOT or EOT signal of the first CSI interface for transmitting the first image of the Nth frame, or after passing After the preset time delay, the corresponding first value T1 of the first clock counter is obtained, where T2 is greater than or equal to T1;
- the second image of the N-1th frame and the second image of the Nth frame are obtained, and
- the acquired second image of the N-1th frame and the second image of the Nth frame are interpolated to obtain the third image of the Nth frame that is synchronized with the first image of the Nth frame.
- image: New_image [image_N_1*(T2-T1)+
- New_image represents the third image of the Nth frame that is synchronized with the first image of the Nth frame
- image_N_1 represents the second image of the N-1th frame
- image_N represents the second image of the Nth frame.
- the image acquisition method further includes: detecting the brightness of ambient light of the first camera and the second camera; and setting the exposure time of the first camera and the second camera so that the brightness of the ambient light is greater. The lower the value, the longer the exposure time of the corresponding camera.
- the image acquisition method further includes, by the second processor: obtaining the first exposure time t00 of the first camera and the second exposure time t11 of the second camera; according to the following formula (2)
- the above adjusted second value T2' can be substituted into T2 of formula (1) to interpolate to obtain the third image of the Nth frame that is synchronized with the first image of the Nth frame.
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Signal Processing (AREA)
- Optics & Photonics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Multimedia (AREA)
- Health & Medical Sciences (AREA)
- Acoustics & Sound (AREA)
- General Health & Medical Sciences (AREA)
- Otolaryngology (AREA)
- Ophthalmology & Optometry (AREA)
- Studio Devices (AREA)
Abstract
本申请涉及一种无线智能可穿戴装置及其图像采集方法。该装置包括可彼此无线通信的第一部和第二部。第一部和第二部分别包括第一和第二处理器、第一和第二无线通信模块、第一和第二摄像头、第一和第二图像采集模块以及第一和第二时钟。第一图像采集模块向第一摄像头发送基于第一时钟的第一硬件触发信号;第二图像采集模块向第二摄像头发送基于第二时钟的第二硬件触发信号。至少一个处理器使得所述第一无线通信模块和所述第二无线通信模块持续执行彼此的无线通信和/或两者与智能设备的无线通信,并确定时钟差异,以实现述第一和第二硬件触发信号的同步。如此使得设置在该装置的多个分部的多个摄像头实现图像拍摄和采集的动态精确同步。
Description
本申请涉及一种无线装置和图像采集方法,更具体地,涉及一种无线智能可穿戴装置及其图像采集方法。
智能可穿戴装置,例如智能眼镜、智能手表等,逐渐进入到人们的工作和生活中。这些智能可穿戴装置彼此之间,以及与其他智能设备(例如手机、pad、个人电脑、多媒体电视等)利用无线方式连接。
随着用户需求的发展,这些智能可穿戴装置通常在多个位置(例如智能眼镜的左眼镜部和右眼镜部)分别设置有多个摄像头,且提供各种视频相关功能,例如增强现实、虚拟现实、全景视野等,这些功能需要多个摄像头同步地采集视频或图像。但是,当前各个摄像头的视频或图像采集存在同步不够精确的问题,这样例如如果要对多个摄像头采集的视频或图像进行拼接以生成全景视频,就会产生图像模糊、运动鬼影等缺陷,影响用户的使用体验。
发明内容
提供了本申请以解决现有技术中存在的技术问题。
本申请旨在提供一种无线智能可穿戴装置及其图像采集方法,其能够以硬件触发的方式,使得设置在无线智能可穿戴装置的多个分部的多个摄像头,实现图像拍摄和采集的动态精确同步。
根据本申请的第一方案,提供了一种无线智能可穿戴装置。该无线智能可穿戴装置包括可彼此无线通信的第一部和第二部。所述第一部包括第一处理器、第一无线通信模块、第一摄像头和第一图像采集模块且具有第一时钟;所述第二部包括第二处理器、第二无线通信模块、第二摄像头和第二图像采集模块且具有第二时钟。所述第一图像采集模块配置为:通过向所述第一摄像头发送基于所述第一时钟的第一硬件触发信号,以触发所述第一摄像头拍摄第一图像并获取第一图像;所述第二图像采集模块配置为:通过向所述第二摄像头发送基于所述第二时钟的第二硬件触发信号,以触发所述第二摄像头拍摄第二图像并获取第二图像。所述第一处理器和所述第二处理器中的至
少一个配置为:在所述第一图像采集模块和所述第二图像采集模块的持续使用过程中,使得所述第一无线通信模块和所述第二无线通信模块执行彼此的无线通信和/或两者与智能设备的无线通信,并确定所述第一无线通信模块和所述第二无线通信模块各自执行所述无线通信的时钟差异,该时钟差异用于实现所述第一硬件触发信号和所述第二硬件触发信号的同步。
根据本申请的第二方案,提供一种无线智能可穿戴装置的图像采集方法。所述无线智能可穿戴装置包括可彼此无线通信的第一部和第二部。所述第一部包括第一处理器、第一无线通信模块、第一摄像头和第一图像采集模块且具有第一时钟;所述第二部包括第二处理器、第二无线通信模块、第二摄像头和第二图像采集模块且具有第二时钟。所述图像采集方法包括如下步骤。在所述第一图像采集模块和所述第二图像采集模块的持续使用过程中,使得所述第一无线通信模块和所述第二无线通信模块执行彼此的无线通信和/或两者与智能设备的无线通信,并确定所述第一无线通信模块和所述第二无线通信模块各自执行所述无线通信的时钟差异。由所述第一图像采集模块向所述第一摄像头发送基于所述第一时钟的第一硬件触发信号,以触发所述第一摄像头拍摄第一图像并获取第一图像。由所述第二图像采集模块向所述第二摄像头发送基于所述第二时钟的第二硬件触发信号,以触发所述第二摄像头拍摄第二图像并获取第二图像。其中,利用该时钟差异以实现所述第一硬件触发信号和所述第二硬件触发信号的同步。
利用根据本申请的无线智能可穿戴装置及其图像采集方法,其能够以硬件触发的方式,使得设置在无线智能可穿戴装置的多个分部的多个摄像头,实现图像拍摄和采集的动态精确同步。
在不一定按比例绘制的附图中,相同的附图标记可以在不同的视图中描述相似的部件。具有字母后缀或不同字母后缀的相同附图标记可以表示相似部件的不同实例。附图大体上通过举例而不是限制的方式示出各种实施例,并且与说明书以及权利要求书一起用于对所公开的实施例进行说明。在适当的时候,在所有附图中使用相同的附图标记指代同一或相似的部分。这样的实施例是例证性的,而并非旨在作为本装置或方法的穷尽或排他实施例。
图1示出根据本申请第一实施例的无线智能可穿戴装置的结构示意图;
图2示出根据本申请第二实施例的无线智能可穿戴装置中不同摄像头的图像拍摄和采集的同步控制方式的示意图;
图3示出根据本申请第三实施例的无线智能可穿戴装置中不同摄像头的图像拍摄和采集的同步控制方式的示意图;
图4示出根据本申请第四实施例的无线智能可穿戴装置中不同摄像头的图像拍摄和采集的同步控制方式的示意图;
图5示出根据本申请第五实施例的无线智能可穿戴装置中不同摄像头的图像拍摄和采集的同步控制方式的示意图;
图6示出根据本申请第六实施例的用于触发不同摄像头进行图像拍摄和采集的硬件触发信号的时序图;
图7示出根据本申请第七实施例的无线智能可穿戴装置的图像采集方法的流程图;
图8示出根据本申请第八实施例的无线智能可穿戴装置的图像采集方法的流程图;
图9示出根据本申请第九实施例的SOT或EOT信号的时序图;
图10示出根据本申请第十实施例的无线智能可穿戴装置的结构示意图;以及
图11示出根据本申请第十一实施例的无线智能可穿戴装置的图像采集方法的流程图。
为使本领域技术人员更好的理解本申请的技术方案,下面结合附图和具体实施方式对本申请作详细说明。下面结合附图和具体实施例对本申请的实施例作进一步详细描述,但不作为对本申请的限定。本文中所描述的各个步骤,如果彼此之间没有前后关系的必要性,则本文中作为示例对其进行描述的次序不应视为限制,本领域技术人员应知道可以对其进行顺序调整,只要不破坏其彼此之间的逻辑性导致整个流程无法实现即可。本申请中的表述“第一”、“第二”和“第三”仅仅旨在描述上的区分,并不意味着对数量、顺序上的限定,也不旨在限定紧跟表述“第一”、“第二”和“第三”后的元件、装置和系统在物理属性上的差异。例如,“第一片上系统”可以包括在单个芯片上实现的系统,也可以包括在多个芯片上实现的(一个或多个)系统。
图1示出根据本申请第一实施例的无线智能可穿戴装置的结构示意图。该无线智能可穿戴装置可以包括可彼此无线通信的第一部101a和第二部101b。作为示例,在图1中示出了无线智能眼镜装置作为无线智能可穿戴装置的示例,所述第一部101a和第二部101b中的一个为左眼镜部而另一个为右眼镜部,但这仅仅作为示例。无线智能可穿戴装置可以采用其他构造,甚至可以形成为组件,包括两个以上的分立的构件(装置),只要各个构件均设置有各自的摄像头且需要合成来自各自的摄像头的图像或视频即可。例如,该无线智能可穿戴装置可以包括无线智能头盔、无线智能手环等,也可以包括无线智能项链与无线智能手环的组件等等。
下面以无线智能眼镜装置作为示例进行说明,以下说明也可以灵活专用于其他构造的无线智能可穿戴装置,在此不赘述。
如图1和图2所示,所述第一部101a包括第一处理器102a、第一无线通信模块103a、第一摄像头105a和第一图像采集模块104a且具有第一时钟106a,所述第二部101b包括第二处理器102b、第二无线通信模块103b、第二摄像头105b和第二图像采集模块104b且具有第二时钟106b。作为示例,第一摄像头105a和第二摄像头105b分设于眼镜架的上梁的左右两侧,以便拍摄的图像/视频能够尽可能包含用户的关注视角内的周边环境信息。在一些实施例中,第一处理器102a、第一无线通信模块103a、第一图像采集模块104a和第一时钟106a内置于第一部101a(图1中示出为右眼镜部)的镜腿内,而第二处理器102b、第二无线通信模块103b、第二图像采集模块104b和第二时钟106b则内置于第二部101b(图1中示出为左眼镜部)的镜腿内。通常第一部101a和第二部101b是无线连接的,第一时钟106a和第二时钟106b分属于两部,虽然可以具有相同的标称频率,但它们各自基于不同的晶体或晶振,其晶体或晶振往往会存在一定的频偏,频偏例如10ppm,5ppm,1ppm之内。再者,第一时钟106a和第二时钟106b的计时器的起始数值也可能有偏差。
在一些实施例中,第一时钟106a可以属于第一无线通信模块103a,而第二时钟106b可以属于第二无线通信模块103b。例如,如果第一无线通信模块103a是蓝牙通信模块103a,则其自带的蓝牙时钟就可以用作第一时钟106a。
第一时钟106a可以由第一时钟计数器或者说第一部时钟计数器来表征;第二时钟106b可以由第二时钟计数器或者说第二部时钟计数器来表征。由于第一时钟计数器、第二时钟计数器的计数起始时刻可能有差异,甚至初始值也可以不同,另外,第一时钟106a、第二时钟106b可能有频偏,因此在同一时刻,第一时钟计数器、第二时钟计数器往往不同。例如可能在某一时刻,第一时钟计数器的数值是20,第二时钟计数器的数值是24。
在一些实施例中,如图2所示,对于第一部101a而言,第一处理器102a、第一无线通信模块103a、第一图像采集模块104a和第一时钟106a在同一个芯片上实现,也称为实现为同个片上系统(下文中称为第一片上系统),而第一摄像头105a则实现为另一个独立的芯片(下文中称为第二芯片)。也就是说,第一摄像头105a具有独立的时钟(计时器),即便知晓所设定的拍摄采集时间,独立时钟之间的频偏导致第一片上系统和第二芯片的定时差异对于彼此不可控。这会进一步导致第一摄像头105a和第二摄像头105b的图像拍摄采集无法精确同步。
片上系统也称SOC,例如,可以利用从ARM公司等购买的各种RISC(精简指令集计算机)处理器IP来作为SOC的处理器来执行对应的功能,从而可以实现为嵌入式系统。具体说来,在市场上可购买到的模组(IP)上具有很多模块,例如但不限于存储器、各种通信模块(例如WiFi通信模块、蓝牙通信模块等)、图像采集模块、缓存器、时钟等等。在一些实施例中,芯片制造商也可以在现成的IP上自主开发出这些模块的定制版本。此外,其它的比如天线、传感器组件、扬声器、麦克风等可以外接到IP上。用户可以通过基于购买的IP或自主研发的模块构建ASIC(特定用途集成电路),来实现各种通信模块、图像采集模块等,以便降低功耗和成本。例如,用户也可以利用FPGA(现场可编程门阵列)来实现各种通信模块、图像采集模块等,可以用于对硬件设计的稳定性进行验证。
所述第一图像采集模块104a配置为通过向所述第一摄像头105a发送基于所述第一时钟106a的第一硬件触发信号S1,以触发所述第一摄像头105a拍摄第一图像并获取第一图像。所述第二图像采集模块104b配置为通过向所述第二摄像头105b发送基于所述第二时钟106b的第二硬件触发信号S2,以触发所述第二摄像头105b拍摄第二图像并获取第二图像。通过分别以第一硬件触发信号S1触发第一摄像头105a的拍摄,以第二硬件触发信号S2触发第
二摄像头105b的拍摄,只要从第一图像采集模块104a和第二图像采集模块104b侧同时地输出第一硬件触发信号S1和第二硬件触发信号S2,就可以同时启动图像采集,而无需考虑第一摄像头105a和第二摄像头105b所在的独立芯片上的时钟的频偏情况。
所述第一处理器102a和所述第二处理器102b中的至少一个配置为:在所述第一图像采集模块104a和所述第二图像采集模块104b的持续使用过程中,使得所述第一无线通信模块103a和所述第二无线通信模块103b执行彼此的无线通信(如通信信号S3所示)和/或两者与智能设备的无线通信(如通信信号S4所示),并确定所述第一无线通信模块103a和所述第二无线通信模块103b各自执行所述无线通信的时钟差异,可以利用该动态的时钟差异来实现所述第一硬件触发信号S1和所述第二硬件触发信号S2的同步。所谓的“时钟差异”可以实现为time2-time1或者time4-time3,可以参见以下实施例中的详细说明。
具体说来,所述第一处理器102a和所述第二处理器102b中的至少一个可以配置为:在所述第一图像采集模块104a和所述第二图像采集模块104b的持续使用过程中,使得所述第一无线通信模块103a和所述第二无线通信模块103b中的一方向另一方发送无线信号S3。可以确定所述一方发送所述无线信号S3的第一时刻的时钟计时器的数值time1和所述另一方接收所述无线信号S3的第二时刻的时钟计数器的数值time2之间的差异time2-time1。双方彼此之间收发无线信号S3的空中时间通常可以忽略,则该数值差异time2-time1由第一时钟计时器与第二时钟计时器的初始值与初始计数时刻差异,以及第一时钟106a和第二时钟106b之间的频偏导致。当第一时钟106a和第二时钟106b同步后,time2-time1会是同一固定值或近似固定值。在一些实施例中,可以利用time2-time1的差值来调整第二时钟106b,使得第一时钟106a和第二时钟106b同步,从而使得time2-time1保持同一固定值或近似固定值。
在一些实施例中,所述第一处理器102a和所述第二处理器102b中的至少一个配置为:在所述第一图像采集模块104a和所述第二图像采集模块104b的持续使用过程中,使得所述第一无线通信模块103a和所述第二无线通信模块103b各自接收来自智能设备的无线信号S4;确定所述第一无线通信模块103a接收所述无线信号S4的第三时刻的时钟计时器的数值time3和所述第
二无线通信模块103b接收所述无线信号S4的第四时刻的时钟计数器的数值time4。双方接收来自智能设备的无线信号S4的空中时间偏差通常可以忽略,则该数值差异time4-time3也是由第一时钟计时器与第二时钟计时器的初始值与初始计数时刻差异,以及第一时钟106a和第二时钟106b之间的频偏导致。在一些实施例中,可以利用所述第一无线通信模块103a接收来自智能设备的无线信号S4实现第一时钟106a与智能设备无线时钟之间的同步;可以利用所述第一无线通信模块103b接收来自智能设备的无线信号S4实现第二时钟106b与智能设备无线时钟之间的同步;从而实现第一时钟106a与第二时钟106b的时间同步;从而也使得time3-time4保持同一固定值或近似固定值。
第一处理器102a和所述第二处理器102b中的至少一个可以利用所述数值差异来实现所述第一硬件触发信号S1和所述第二硬件触发信号S2的同步。具体说来,可以在分别发送第一硬件触发信号S1和所述第二硬件触发信号S2时考虑并补偿该数值差异(time4-time3或者time2-time1可以动态变化的数值差异),使得第一硬件触发信号S1和所述第二硬件触发信号S2的动态精确同步。
进一步地,该数值差异也可以用于实现第一部101a和第二部101b之间的无线收发时钟的精准同步。第一无线通信模块103a和所述第二无线通信模块103b可以采用各种无线通信方式,例如但不限于蓝牙模块、WiFi模块、UWB模块等。例如对于蓝牙模块,通过物理层的蓝牙访问码或部分访问码的相关处理,第一部101a的蓝牙模块的蓝牙时钟能与第二部101b的蓝牙模块的蓝牙时钟同步,蓝牙时钟即蓝牙模式下的无线收发时钟。例如对于WIFI模块,根据WiFi协议,WiFi设备会在相同时刻(比如每间隔50ms,102.4ms,500ms等)接收到接入设备发送的信标,WiFi设备可以将接收到该信标的时间作为WiFi时钟,即WiFi模式下的无线收发时钟。在上述例子中,第一部101a的无线收发时钟即可以用作第一时钟106a;第二部101b的无线收发时钟即可以用作第二时钟106b。
图3示出根据本申请第三实施例的无线智能可穿戴装置中不同摄像头的图像拍摄和采集的同步控制方式的示意图。如图3所示,所述第一图像采集模块104a和所述第一摄像头105a通过第一(对)CSI接口CSI1a-CSI1b和第一(对)GPIO接口GPIO1a-GPIO1b相连,所述第二图像采集模块104b和
所述第二摄像头105b通过第二(对)CSI接口CSI2a-CSI2b和第二(对)GPIO接口GPIO2a-GPIO2b相连。
具体说来,所述第一图像采集模块104a进一步配置为:经由第一GPIO接口GPIO1a-GPIO1b连接到所述第一摄像头105a以向所述第一摄像头105a发送所述第一硬件触发信号S1。所述第一摄像头105a进一步配置为:响应于接收到所述第一硬件触发信号S1,开启曝光和图像拍摄并经由所述第一CSI接口CSI1a-CSI1b向所述第一图像采集模块104a传输图像S5。
所述第二图像采集模块104b进一步配置为:经由第二GPIO接口GPIO2a-GPIO2b连接到所述第二摄像头105b以向所述第二摄像头105b发送所述第二硬件触发信号S2。所述第二摄像头105b进一步配置为:响应于接收到所述第二硬件触发信号S2,开启曝光和图像拍摄并经由所述第二CSI接口CSI2a-CSI2b向所述第二图像采集模块104b传输图像S6。
CSI又称相机串行接口,是摄像头所在芯片通常配备用于与外界进行图像信息交互的接口,也是摄像头与主处理器之间的接口。GPIO又称通用型输入/输出,通常芯片上会配备有多个这样的引脚(pin脚)。由此,可以将各个图像采集模块与摄像头通过CSI接口和GPIO接口连接。通过让GPIO接口独立地负责第一硬件触发信号S1或第二硬件触发信号S2的传输,并且让CSI接口独立地负责图像/视频的传输,为硬件触发信号和图像/视频信息的传输提供彼此独立的硬件通路,能够确保传输速度且避免彼此的信息干扰。
就图1所示的无线智能眼镜装置而言,左右眼镜部101a和101b可以通过GPIO口,由第一图像采集模块104a和第二图像采集模块104b向第一摄像头105a和第二摄像头105b分别同时输出硬件触发信号S1和S2,以此同时启动图像采集。并在每帧或每隔N(N是大于1的整数)帧(由各自的时钟来计时,例如每当达到每帧或每隔N帧对应的时钟计数器的数值时),再同时由第一图像采集模块104a和第二图像采集模块104b向第一摄像头105a和第二摄像头105b分别同时输出硬件触发信号S1和S2,以此纠正由于左右眼镜的像素时钟的频偏引起的后继采集的左右眼镜图像不同步。启动图像采集后,第一摄像头105a和第二摄像头105b基于曝光时间、曝光行数、行长度和帧长度等参数配置分别向第一图像采集模块104a和第二图像采集模块104b传送视频图像S5和S6,并使得左右眼镜部101a和101b以相同的标称像素时钟数获取每帧图像。
请注意,本申请中所谓通过发送硬件触发信号触发对应摄像头的拍摄(曝光)和视频图像的回传(图像采集),意味着拍摄和回传行为因为收到硬件触发信号而被触发(存在因果关系),但拍摄和回传的时间可以与硬件触发信号的接收时间不同(未必同时),至少在接收时间以后。在一些实施例中,以第一摄像头105a为例,第一摄像头105a接收到第一硬件触发信号S1后,可以经过预定时延再开始曝光,该预定时延大于上述间隔N帧由于频偏引起的时间偏移,以确保第一摄像头105a在间隔N帧后接收到该第一硬件触发信号S1时,还没有开始曝光。从而避免让第一摄像头105a在曝光过程中接收到该第一硬件触发信号S1,而引起丢帧或该帧图像错误。
图4示出根据本申请第四实施例的无线智能可穿戴装置中不同摄像头的图像拍摄和采集的同步控制方式的示意图。图5示出根据本申请第五实施例的无线智能可穿戴装置中不同摄像头的图像拍摄和采集的同步控制方式的示意图。下面分别结合图4和图5对各自的不同摄像头的图像拍摄和采集的同步控制方式进行具体说明。
如图4所示,所述第一处理器102a可以进一步配置为:在第一部101a的时钟计数器的数值为第一预定数值t1时产生所述第一硬件触发信号S1;所述第二处理器102b进一步配置为:在第二部101b的时钟计数器的数值为第二预定数值t1+Δt时产生所述第二硬件触发信号S2。其中,所述第一预定数值t1和所述第二预定数值t1+Δt的差异Δt基于上述数值差异time4-time3或者time2-time1来设定,使得所述第一预定数值t1和所述第二预定数值t1+Δt分别对于第一时钟106a的时钟计时器和第二时钟106b的时钟计时器而言表征实质上的同一时刻。作为示例,可以将差异Δt设置为time4-time3或者time2-time1,且随着time4-time3或者time2-time1的动态变化而更新。由此,可以使得所述第一硬件触发信号S1和所述第二硬件触发信号S2实质上在同一时刻产生,从而实现第一摄像头105a和第二摄像头105b的图像拍摄和采集的同步触发。
如图5所示,所述第一处理器102a可以进一步配置为:产生基准硬件触发信号(未示出)并获取触发时刻第一部101a的时钟计数器的基准数值t0;使得所述第一无线通信模块103a将所述基准数值t0传输给所述第二无线通信模块103b;在所述触发时刻t0后经过预定时延td产生所述第一硬件触发信号S1。相应地,所述第二处理器102b进一步配置为:基于所述基准数值
t0、所述预定时延td以及时钟计数器的数值差异Δt(例如time4-time3或者time2-time1),来确定用于产生所述第二硬件触发信号S2的时钟计数器的数值,例如t0+td+Δt;在第二部101b的时钟计数器的数值达到所确定的数值(例如t0+td+Δt)时,产生所述第二硬件触发信号S2,从而使得所述第一硬件触发信号S1和所述第二硬件触发信号S2实质上在同一时刻产生,从而实现第一摄像头105a和第二摄像头105b的图像拍摄和采集的同步触发。在一些实施例中,所述预定时延td可以大于上述间隔N帧由于频偏引起的时间偏移,以确保第一摄像头105a(第二摄像头105b)在间隔N帧后接收到该第一硬件触发信号S1(第二硬件触发信号S2)时,还没有开始曝光。从而避免让第一摄像头105a(第二摄像头105b)在曝光过程中接收到该第一硬件触发信号S1(第二硬件触发信号S2),而引起丢帧或该帧图像错误。同时,所述预定时延td这一时间段还可以用来由所述第一无线通信模块103a将所述基准数值t0或t0+td传输给所述第二无线通信模块103b。
返回图1,在一些实施例中,所述第一部101a和第二部101b各自包括亮度检测单元107a和107b,其配置为检测对应的第一摄像头105a和第二摄像头105b的环境光的亮度。就无线智能眼镜而言,由于镜片相对于环境光源的角度区别会导致第一摄像头105a和第二摄像头105b曝光拍摄时的入光量不同。所述第一处理器102a和所述第二处理器102b中的至少一个进一步配置为:基于所述第一部101a和第二部101b各自检出的亮度,设置所述第一摄像头105a和所述第二摄像头105b的曝光时间,使得摄像头对应的检出亮度越低则曝光时间越长。也就是说,如果第一部101a的亮度检测单元107a检测的环境光的亮度更低,那么就使得第一摄像头105a的曝光时间更长,以弥补亮度不足,使得第一摄像头105a曝光拍摄的图像/视频与第二摄像头105b曝光拍摄的图像/视频具有一致的亮度,从而有助于提升后续合成(融合)图像的质量。
在一些实施例中,所述第一处理器102a和所述第二处理器102b中的至少一个可以进一步配置为:对于要设置更长的曝光时间的第一摄像头105a或第二摄像头105b,使其相连的第一图像采集模块104a或第二图像采集模块104b相较另一摄像头相连的图像采集模块,提前预定时间产生和发送对应的硬件触发信号S1或S2。为了便于说明,假设某一摄像头相较另一摄像头的曝光时间的超出量为Δtp,则提前的预定时间可以根据该超出量Δtp来设
置。设置方式则可以根据整个曝光时间段所采集的图像所归属的时间点的设定规则来进行适配。例如,在曝光时间段持续较长的情况下,可以将所采集的图像作为曝光时间段中的中间时间点的图像。相应地,在第一摄像头105a相较第二摄像头105b的曝光时间的超出量为Δtp的情况下,如图6所示,第一硬件触发信号S1的时序比第二硬件触发信号S2的时序提前所述曝光时间的超出量的大约一半,即提前大约1/2Δtp。如此,不仅可以弥补第一摄像头105a的亮度不足,还可以让第一摄像头105a和第二摄像头105b在彼此曝光时间段采集的图像保持同步,而不会受到曝光时间段的较长持续时间的影响。
受益于第一图像和第二图像的动态精确同步,第一图像和第二图像可以用于融合以满足用户日益增长的各种需求。例如,所述第一处理器102a和所述第二处理器102b中的至少一个可以进一步配置为:利用所述第一图像和所述第二图像用于生成全景视频或同步定位和地图构建。
图7示出根据本申请第七实施例的无线智能可穿戴装置的图像采集方法的流程图。所述无线智能可穿戴装置例如可以采用如图1和图2所示的构造但不限于此,其包括可彼此无线通信的第一部和第二部,所述第一部包括第一处理器、第一无线通信模块、第一摄像头和第一图像采集模块且具有第一时钟,所述第二部包括第二处理器、第二无线通信模块、第二摄像头和第二图像采集模块且具有第二时钟。其中,第一时钟和第二时钟是彼此独立的,可以存在频偏;各个摄像头可以设置在独立的芯片上,也可以与其他构件设置在同一个片上系统上,在此不做限定。
如图7所示,所述图像采集方法可以包括如下步骤。
在步骤701,在所述第一图像采集模块和所述第二图像采集模块的持续使用过程中,使得所述第一无线通信模块和所述第二无线通信模块执行彼此的无线通信和/或两者与智能设备的无线通信,并确定所述第一无线通信模块和所述第二无线通信模块各自执行所述无线通信的时钟差异。
在步骤702,可以由所述第一图像采集模块向所述第一摄像头发送基于所述第一时钟的第一硬件触发信号,以触发所述第一摄像头拍摄第一图像并获取第一图像。
在步骤703,由所述第二图像采集模块104b向所述第二摄像头105b发送基于所述第二时钟106b的第二硬件触发信号,以触发所述第二摄像头105b拍摄第二图像并获取第二图像。
请注意,步骤702和步骤703是分别由第一图像采集模块和第二图像采集模块侧控制的并行的处理步骤。
利用步骤701中所确定的动态的时钟差异(也就是收发定时的时钟计数器的数值差异),对所述第一硬件触发信号和所述第二硬件触发信号进行同步,例如但不限于对所述第一硬件触发信号和所述第二硬件触发信号的对应的时钟计数器的数值进行数值差异的补偿,从而使得第一硬件触发信号和所述第二硬件触发信号实质上在相同的时间发送。如此,不管第一时钟和第二时钟的频偏如何动态变化,都可以被动态地捕捉到,并及时准确地弥补,进而同时启动第一摄像头和第二摄像头的图像拍摄和采集,而无需考虑第一摄像头和第二摄像头所在的第一部和第二部各自的独立芯片上的时钟的频偏情况。所拍摄和采集的第一图像和第二图像可以保持精确的动态同步,可以用于融合以满足用户日益增长的各种需求,例如可以用于生成全景视频或同步定位和地图构建。
图8示出根据本申请第八实施例的无线智能可穿戴装置的图像采集方法的流程图。在步骤801,将所述第一图像采集模块经由第一GPIO接口连接到所述第一摄像头以向所述第一摄像头发送所述第一硬件触发信号。在步骤802,响应于接收到所述第一硬件触发信号,由所述第一摄像头开启曝光和图像拍摄并经由第一CSI接口向所述第一图像采集模块传输图像。
在步骤803,将所述第二图像采集模块经由第二GPIO接口连接到所述第二摄像头以向所述第二摄像头发送所述第二硬件触发信号。在步骤804,响应于接收到所述第二硬件触发信号,由所述第二摄像头开启曝光和图像拍摄并经由第二CSI接口向所述第二图像采集模块传输图像。由此,可以将各个图像采集模块与摄像头通过CSI接口和GPIO接口连接。通过让GPIO接口独立地负责第一硬件触发信号或第二硬件触发信号的传输,并且让CSI接口独立地负责图像/视频的传输,为硬件触发信号和图像/视频信息的传输提供彼此独立的硬件通路,能够确保传输速度且避免彼此的信息干扰。
在一些实施例中,可以通过如下步骤来确定所述第一无线通信模块和所述第二无线通信模块各自执行所述无线通信的时钟差异。在所述第一图像采
集模块和所述第二图像采集模块的持续使用过程中,使得所述第一无线通信模块和所述第二无线通信模块中的一方向另一方发送无线信号;确定所述一方发送所述无线信号的第一时刻和所述另一方接收所述无线信号的第二时刻的时钟计数器的数值差异作为所述时钟差异。所述无线信号包括接收方已知的信号序列,例如对于蓝牙模块,可以是物理层的蓝牙访问码;例如对于WIFI模块,可以是信标信号。
在一些实施例中,可以通过如下步骤来确定所述第一无线通信模块和所述第二无线通信模块各自执行所述无线通信的时钟差异。在所述第一图像采集模块和所述第二图像采集模块的持续使用过程中,使得所述第一无线通信模块和所述第二无线通信模块各自接收来自智能设备的无线信号;确定所述第一无线通信模块接收所述无线信号的第三时刻和所述第二无线通信模块接收所述无线信号的第四时刻的时钟计数器的数值差异作为所述时钟差异。所述无线信号包括接收方已知的信号序列,例如对于蓝牙模块,可以是物理层的蓝牙访问码;例如对于WIFI模块,可以是信标信号。
第一部-第二部相互通信,第一部/第二部-智能设备的联合通信,对于诸如智能眼镜等无线智能可穿戴装置而言都是常规的通信且持续地发生,从而在动态确定时钟差异的同时也不会影响用户的使用体验。
可以采用各种方式来实现第一和第二硬件触发信号的定时同步。
例如,可以在第一部的时钟计数器的数值为第一预定数值时产生所述第一硬件触发信号;在第二部的时钟计数器的数值为第二预定数值时产生所述第二硬件触发信号,其中,所述第一预定数值和所述第二预定数值的差异基于所述数值差异来设定,使得表征同一时刻。
又例如,可以在第一部产生基准硬件触发信号,并获取触发时刻第一部的时钟计数器的基准数值;经由所述第一无线通信模块,将所述基准数值传输给所述第二无线通信模块;在所述触发时刻后经过预定时延产生所述第一硬件触发信号;在第二部,基于所述基准数值、所述预定时延以及时钟计数器的数值差异,来确定用于产生所述第二硬件触发信号的时钟计数器的数值;在第二部的时钟计数器的数值达到所确定的数值时,产生所述第二硬件触发信号,从而使得所述第一硬件触发信号和所述第二硬件触发信号在同一时刻产生。
以上定时同步控制方式在上文中已经结合图4和图5进行了详细描述,在此不赘述。
在一些实施例中,图像采集方法还可以包括对各个摄像头的环境光的亮度的检测和补偿均衡。可以检测第一摄像头和第二摄像的环境光的亮度;设置所述第一摄像头和所述第二摄像头的曝光时间,使得环境光的亮度越低,对应的摄像头的曝光时间越长。
在一些实施例中,图像采集方法还可以包括:对于要设置更长的曝光时间的第一摄像头或第二摄像头,使其相连的图像采集模块相较另一摄像头相连的图像采集模块,提前预定时间产生和发送对应的硬件触发信号。作为示例,所提前的预定时间为所述曝光时间的超出量的大约一半。
对各个摄像头的环境光的亮度的检测和补偿均衡在上文中已经结合图6进行了详细描述,在此不赘述。
下面对无线智能可穿戴装置的变型例进行说明。该变型例的无线智能可穿戴装置可以采用图10所示的硬件配置,其中,第一时钟106a和第二时钟106b彼此独立且也包含时钟计数器的功能,为了简化说明,下文中分别称为第一时钟计数器106a和第二时钟计数器106b。与图2所示的硬件配置的区别在于,该变型例不依赖于GPIO接口也无需从图像采集模块侧的芯片向摄像头侧的芯片发送硬件触发信号,而是以某个摄像头拍摄的各帧图像作为参考图像,考虑到另一个摄像头与其的拍摄和采集对应帧(例如相邻帧)的定时的时钟差异,以参考图像的拍摄和采集定时为标准进行插值,以得到同步的图像。图10中与图2中相同的构造部分请参见上文中的描述,在此不赘述,仅仅对两者之间的区别进行说明。
下面以无线智能眼镜装置作为无线智能可穿戴装置的示例进行说明,且以第一部101a的第一摄像头105a拍摄的第一图像作为参考图像为例进行说明。但须知无线智能可穿戴装置可以采用其他的构造,所述第一部101a和所述第二部101b能够在左眼镜部与右眼镜部之间切换,第一摄像头105a拍摄采集的第一图像和第二摄像头105b拍摄采集的第二图像能够被切换用作参考图像。
具体说来,所述第一图像采集模块104a配置为:通过第一CSI接口CSI1a-CSI1b,与所述第一摄像头105a互连并获取由所述第一摄像头105a拍摄的各帧的第一图像S5作为参考图像。所述第二图像采集模块104b配置
为:通过第二CSI接口CSI2a-CSI2b,与所述第二摄像头105b互连并获取由所述第二摄像头105b拍摄的各帧的第二图像S6。
所述第一处理器102a配置为:获取所述第一图像采集模块104a接收到所述第一CSI接口的用于传输各帧第一图像的SOT或EOT信号时,或者再经过预设时延后,所述第一时钟计数器106a的第一数值。所述第二处理器102b配置为:获取所述第二图像采集模块104b接收到所述第二CSI接口的用于传输各帧第二图像的SOT或EOT信号时,或者再经过所述预设时延后,所述第二时钟计数器106b的第二数值。
在一些实施例中,所述第一时钟计数器106a的第一数值和所述第二时钟计数器106b的第二数值可以基于相同时刻第一时钟计数器106a与所述第二时钟计数器106b的计数值的差异来进行补偿调整,然后再进行后续的分析。也就是说,下文中用于插值处理的所述第一时钟计数器106a的数值(例如但不限于第一数值)和所述第二时钟计数器106b的对应数值(例如但不限于第二数值)可以利用所述第一时钟计数器106a与所述第二时钟计数器106b在相同时刻的数值差异进行补偿调节,以便消除诸如两个时钟计数器的起始计数值和/或频偏导致的在实际相同时刻的数值差异。该数值差异例如可以是time2-time1,也可以是time4-time3,参见以下实施例中的具体说明。
具体说来,参见图10,所述第一处理器102a和所述第二处理器102b中的至少一个可以配置为:在所述第一图像采集模块104a和所述第二图像采集模块104b的持续使用过程中,使得所述第一无线通信模块103a和所述第二无线通信模块103b中的一方向另一方发送无线信号S3(返回参见图2)。可以确定所述一方发送所述无线信号S3的第一时刻的时钟计时器的数值time1和所述另一方接收所述无线信号S3的第二时刻的时钟计数器的数值time2之间的差异time2-time1。双方彼此之间收发无线信号S3的空中时间通常可以忽略,则该数值差异time2-time1由第一时钟计时器与第二时钟计时器的初始值与初始计数时刻差异,以及第一时钟106a和第二时钟106b之间的频偏导致。当第一时钟106a和第二时钟106b同步后,time2-time1会是同一固定值或近似固定值。在一些实施例中,可以利用time2-time1的差值来调整第二时钟106b,使得第一时钟106a和第二时钟106b同步,从而使得time2-time1保持同一固定值或近似固定值。
在一些实施例中,所述第一处理器102a和所述第二处理器102b中的至少一个配置为:在所述第一图像采集模块104a和所述第二图像采集模块104b的持续使用过程中,使得所述第一无线通信模块103a和所述第二无线通信模块103b各自接收来自智能设备的无线信号S4(返回参见图2);确定所述第一无线通信模块103a接收所述无线信号S4的第三时刻的时钟计时器的数值time3和所述第二无线通信模块103b接收所述无线信号S4的第四时刻的时钟计数器的数值time4。双方接收来自智能设备的无线信号S4的空中时间偏差通常可以忽略,则该数值差异time4-time3也是由第一时钟计时器与第二时钟计时器的初始值与初始计数时刻差异,以及第一时钟106a和第二时钟106b之间的频偏导致。在一些实施例中,可以利用所述第一无线通信模块103a接收来自智能设备的无线信号S4实现第一时钟106a与智能设备无线时钟之间的同步;可以利用所述第一无线通信模块103b接收来自智能设备的无线信号S4实现第二时钟106b与智能设备无线时钟之间的同步;从而实现第一时钟106a与第二时钟106b的时间同步;从而也使得time3-time4保持同一固定值或近似固定值。
可以在第一时钟计数器106a与所述第二时钟计数器106b的计数值的基础上利用数值差异time2-time1或者time4-time3来进行补偿调整。具体说来,如果time2为第二无线通信模块103b发送所述无线信号S3的第一时刻的时钟计时器的数值,而time1则是第一无线通信模块103a接收所述无线信号S3的第二时刻的时钟计数器的数值,且time2>time1,则可以将第二时钟计数器106b的动态计数值减去(time2-time1),从而消除第一时钟计时器与第二时钟计时器的初始值与初始计数时刻差异以及第一时钟106a和第二时钟106b之间的频偏的不利影响。又例如,如果time4>time3,则可以将第二时钟计数器106b的动态计数值减去(time4-time3),从而消除第一时钟计时器与第二时钟计时器的初始值与初始计数时刻差异以及第一时钟106a和第二时钟106b之间的频偏的不利影响。
进行了计数值的补偿调整后,可以得到消除第一时钟计时器与第二时钟计时器的初始值与初始计数时刻差异以及第一时钟106a和第二时钟106b之间的频偏的不利影响后,接收到所述第一CSI接口的用于传输各帧第一图像的SOT或EOT信号时或者再经过预设时延后的时间,与接收到所述第二CSI接口的用于传输各帧第二图像的SOT或EOT信号时或者再经过所述预设时延
后的时间,两者之间真实的时间先后关系,以利于后续插值处理后的图像与参考图像的同步性更好。
如图9所示,示出了根据MIPI CSI-2协议定义的CSI接口连接在传输一个数据包时的时序图。CSI接口为单向传输,从摄像头向外传输,包括一个时钟lane(数据通道)和一到四个数据lane,四个数据lane即图9中所示的LANE 1、LANE 2、LANE 3和LANE 4。
SOT又称传输开始信号,EOT又称传输结束信号,是CSI接口传输来自摄像头的图像信息时的传输定时参考信号,由此,第一数值和第二数值实质上代表了第一图像和第二图像各自的拍摄和采集时间。如图9所示,在每个LANE上,有效负荷(字节0-字节N-1,N为正整数)在SOT与EOT信号之间传输。在各个数据包之间具有一个LPS状态(低功耗状态)。
所述第一处理器102a或者所述第二处理器102b进一步配置为,使得所述第一无线通信模块103a向所述第二无线通信模块103b传输所述第一数值,或者使得所述第一无线通信模块103a和所述第二无线通信模块103b向智能设备传输所述第一数值、所述第二数值和所述第二图像,以便所述第二处理器102b或者所述智能设备中的第三处理器102c(参见图2)获取到所述第一数值、所述第二数值连同所述第二图像。进一步地,获取到所述第一数值、所述第二数值连同所述第二图像的所述第二处理器102b或者所述智能设备中的第三处理器102c进一步配置为:基于利用所述第一时钟计数器106a与所述第二时钟计数器106b在相同时刻的数值差异进行补偿调节后的所述第一数值和所述第二数值,对各帧的第二图像进行插值处理,以得到与各帧的第一图像同步的各帧的第三图像。由此,利用CSI接口常规的信号交互方式,通过简单的插值运算,无需其他硬件接口的交互配置,就可以方便地实现第一部101a和第二部101b的各帧图像的精确动态同步。该精确动态同步的水平,使得各帧的第一图像和同步的各帧的第三图像可以用于生成全景视频或同步定位和地图构建。
在插值运算之前,也可以如上文中的其他实施例那样,先考虑第一时钟106a和第二时钟106b的时钟差异,并据此对所述第一数值和/或所述第二数值进行补偿调节,再用于对各帧的第二图像进行插值处理。本申请中各个实施例中记载的用所述第一时钟计数器106a与所述第二时钟计数器106b在相同时刻的数值差异对第一数值和/或所述第二数值进行补偿调节的方法可结
合于此。该数值差异例如可以是time2-time1,也可以是time4-time3,参见各个实施例中的具体说明。
例如,所述第一处理器102a和所述第二处理器102b中的至少一个进一步配置为:在所述第一图像采集模块104a和所述第二图像采集模块104b的持续使用过程中,使得所述第一无线通信模块103a和所述第二无线通信模块103b执行彼此的无线通信和/或两者与智能设备的无线通信;计算利用所述第一无线通信模块103a和所述第二无线通信模块103b执行所述无线通信时,所述第一时钟计数器106a与所述第二时钟计数器106b的数值差异;基于所述数值差异对所述第一数值和/或所述第二数值进行补偿调节,再用于对各帧的第二图像进行插值处理。上文中已经详细描述了如何利用所述第一无线通信模块103a和所述第二无线通信模块103b执行彼此的无线通信和/或两者与智能设备的无线通信来确定时钟差异(即所述第一时钟计数器106a与所述第二时钟计数器106b的数值差异),以及如何使用该数值差异来进行补偿调节,这些说明内容结合于此,在此不赘述。
在一些实施例中,所述第二处理器102b进一步配置为获取所述第二图像采集模块104b接收到所述第二CSI接口CSI2a-CSI2b的用于传输第N-1帧和第N帧的第二图像的SOT或EOT信号时,或者再经过所述预设时延后,所述第二时钟计数器106b的对应两个第二数值T0和T2。所述第二处理器102b可以获取第一图像采集模块104a接收到所述第一CSI接口CSI1a-CSI1b的用于传输第N帧的第一图像的SOT或EOT信号时,或者再经过所述预设时延后,所述第一时钟计数器的对应的第一数值T1,其中,T2大于或等于T1。也就是说,第N帧的第一图像对应的计时的第一数值T1可以位于第N-1帧和第N帧的第二图像对应的计时的两个数值T0与T2之间。
第二处理器102b可以获取第N-1帧的第二图像和第N帧的第二图像,并根据如下公式(1),对所获取的第N-1帧的第二图像和第N帧的第二图像进行插值处理,来得到与第N帧的第一图像同步的第N帧的第三图像:
New_image=[image_N_1*(T2-T1)+image_N*(T1-T0)]/(T2-T0),公式(1)
New_image=[image_N_1*(T2-T1)+image_N*(T1-T0)]/(T2-T0),公式(1)
其中,New_image表示与第N帧的第一图像同步的第N帧的第三图像,image_N_1表示第N-1帧的第二图像,image_N表示第N帧的第二图像。
返回参见图1,所述第一部101a和第二部101b各自可以包括亮度检测单元107a和107b,其配置为检测对应的第一摄像头105a和第二摄像头105b
的环境光的亮度。所述第一处理器102a和所述第二处理器102b中的至少一个进一步配置为:基于所述第一部101a和第二部101b各自检出的亮度,设置所述第一摄像头105a和所述第二摄像头105b的曝光时间,使得摄像头对应的检出亮度越低则曝光时间越长。
对第二图像采集模块104b接收到所述第二CSI接口CSI2a-CSI2b的用于传输第N帧的第二图像的SOT或EOT信号时,或者再经过所述预设时延后,所述第二时钟计数器106b的对应第二数值T2,可以根据所述第一摄像头105a和所述第二摄像头105b的曝光时间的差异进行调整。
具体说来,所述第二处理器102b可以进一步配置为:获取所述第一摄像头的第一曝光时间t00以及所述第二摄像头的第二曝光时间t11;根据如下公式(2),来计算T2经调整后的第二数值T2′:
T2′=T2-(t11-t00)/2,公式(2)。
T2′=T2-(t11-t00)/2,公式(2)。
在曝光时间段持续较长的情况下,可以将所采集的图像作为曝光时间段中的中间时间点的图像。例如,在第二摄像头105b相较第一摄像头105a的曝光时间的超出量为t11-t00的情况下,将T2提前所述曝光时间的超出量的大约一半t11-t00/2。如此,不仅可以弥补第二摄像头105b的亮度不足,还可以让第一摄像头105a和第二摄像头105b在彼此曝光时间段采集的图像保持同步,而不会受到曝光时间段的较长持续时间的影响。以上调整后的第二数值T2′可用于代入到公式(1)的T2,以插值得到与第N帧的第一图像同步的第N帧的第三图像。
图11示出根据本申请第十一实施例的无线智能可穿戴装置的图像采集方法的流程图。所述无线智能可穿戴装置包括彼此无线通信的第一部和第二部。所述第一部包括第一处理器、第一无线通信模块、第一摄像头、第一图像采集模块和第一时钟计数器,所述第二部包括第二处理器、第二无线通信模块、第二摄像头、第二图像采集模块和第二时钟计数器。所述第一时钟计数器和第二时钟计时器是彼此独立的,彼此可以存在动态的时钟偏差。所述无线智能可穿戴装置可以采用根据本申请各个实施例的构造,只要能够协同实现图像采集方法的流程即可,在此不赘述。
如图11所示,所述图像采集方法包括如下步骤。
在步骤1101,由第一图像采集模块通过第一CSI接口与所述第一摄像头互连,并获取由所述第一摄像头拍摄的各帧的第一图像作为参考图像。
在步骤1102,由第二图像采集模块通过第二CSI接口与所述第二摄像头互连,并获取由所述第二摄像头拍摄的各帧的第二图像。
在步骤1103,由所第一处理器,获取所述第一图像采集模块接收到所述第一CSI接口的用于传输各帧第一图像的SOT或EOT信号时,或者再经过预设时延后,所述第一时钟计数器的第一数值。
在步骤1104,由所述第二处理器,获取所述第二图像采集模块接收到所述第二CSI接口的用于传输各帧第二图像的SOT或EOT信号时,或者再经过所述预设时延后,所述第二时钟计数器的第二数值。
在步骤1105,由所述第一无线通信模块向所述第二无线通信模块传输所述第一数值,或者由所述第一无线通信模块和所述第二无线通信模块向智能设备传输所述第一数值、所述第二数值和所述第二图像,以便所述第二处理器或者所述智能设备中的第三处理器获取到所述第一数值、所述第二数值连同所述第二图像。
在步骤1106,由获取到所述第一数值、所述第二数值连同所述第二图像的所述第二处理器或者所述智能设备中的第三处理器,基于第一数值和所述第二数值,对各帧的第二图像进行插值处理,以得到与各帧的第一图像同步的各帧的第三图像。
上文中各个实施例中结合无线智能可穿戴装置的结构所描述的图像采集方法的处理步骤均可以结合于此,在此不赘述。
在一些实施例中,图像采集方法还可以包括:在所述第一图像采集模块和所述第二图像采集模块的持续使用过程中,由所述第一无线通信模块和所述第二无线通信模块执行彼此的无线通信和/或两者与智能设备的无线通信;计算利用所述第一无线通信模块和所述第二无线通信模块执行所述无线通信时,所述第一时钟计数器与所述第二时钟计数器的数值差异;以及基于所述数值差异对所述第一数值和/或所述第二数值进行补偿调节,再用于对各帧的第二图像进行插值处理。
在一些实施例中,图像采集方法还包括:利用各帧的第一图像和同步的各帧的第三图像用于生成全景视频或同步定位和地图构建。
在一些实施例中,所述图像采集方法还包括,由所述第二处理器:获取所述第二图像采集模块接收到所述第二CSI接口的用于传输第N-1帧和第N帧的第二图像的SOT或EOT信号时,或者再经过所述预设时延后,所述第二
时钟计数器的对应两个第二数值T0和T2;获取所述第一图像采集模块接收到所述第一CSI接口的用于传输第N帧的第一图像的SOT或EOT信号时,或者再经过所述预设时延后,所述第一时钟计数器的对应的第一数值T1,其中,T2大于或等于T1;获取第N-1帧的第二图像和第N帧的第二图像,并根据如下公式(1),对所获取的第N-1帧的第二图像和第N帧的第二图像进行插值处理,来得到与第N帧的第一图像同步的第N帧的第三图像:
New_image=[image_N_1*(T2-T1)+image_N*(T1-T0)]/(T2-T0),公式(1)
New_image=[image_N_1*(T2-T1)+image_N*(T1-T0)]/(T2-T0),公式(1)
其中,New_image表示与第N帧的第一图像同步的第N帧的第三图像,image_N_1表示第N-1帧的第二图像,image_N表示第N帧的第二图像。
在一些实施例中,所述图像采集方法还包括:检测第一摄像头和第二摄像头的环境光的亮度;设置所述第一摄像头和所述第二摄像头的曝光时间,使得环境光的亮度越低,对应的摄像头的曝光时间越长。
在一些实施例中,所述图像采集方法还包括,由所述第二处理器:获取所述第一摄像头的第一曝光时间t00以及所述第二摄像头的第二曝光时间t11;根据如下公式(2),来计算调整后的所述第二CSI接口的用于传输第N帧的第二图像的第二数值SOT或EOT信号时,或者再经过所述预设时延后,所述第二时钟计数器的对应第二数值T2′:
T2′=T2-(t11-t00)/2,公式(2)。
T2′=T2-(t11-t00)/2,公式(2)。
以上调整后的第二数值T2′可用于代入到公式(1)的T2,以插值得到与第N帧的第一图像同步的第N帧的第三图像。
以上各个实施例仅仅作为示例,而不作为对本发明的保护范围的限定。本发明的保护范围由权利要求书来定义,本领域技术人员在不偏离和超出权利要求书的保护范围的情况下,可以对各个实施例进行各种变型和修改。以上各个实施例中记载的技术要素的组合方式不限于各个实施例中记载的组合方式,而是不同实施例中的技术要素也可以彼此灵活组合。每个权利要求定义的技术方案都构成独立的实施例且可以彼此组合。
Claims (20)
- 一种无线智能可穿戴装置,包括可彼此无线通信的第一部和第二部,其特征在于,所述第一部包括第一处理器、第一无线通信模块、第一摄像头和第一图像采集模块且具有第一时钟,所述第二部包括第二处理器、第二无线通信模块、第二摄像头和第二图像采集模块且具有第二时钟,所述第一图像采集模块配置为:通过向所述第一摄像头发送基于所述第一时钟的第一硬件触发信号,以触发所述第一摄像头拍摄第一图像并获取第一图像;所述第二图像采集模块配置为:通过向所述第二摄像头发送基于所述第二时钟的第二硬件触发信号,以触发所述第二摄像头拍摄第二图像并获取第二图像,所述第一处理器和所述第二处理器中的至少一个配置为:在所述第一图像采集模块和所述第二图像采集模块的持续使用过程中,使得所述第一无线通信模块和所述第二无线通信模块执行彼此的无线通信和/或两者与智能设备的无线通信,并确定所述第一无线通信模块和所述第二无线通信模块各自执行所述无线通信的时钟差异,该时钟差异用于实现所述第一硬件触发信号和所述第二硬件触发信号的同步。
- 根据权利要求1所述的无线智能可穿戴装置,其特征在于,所述无线智能可穿戴装置包括无线智能眼镜装置,所述第一部和第二部中的一个为左眼镜部而另一个为右眼镜部。
- 根据权利要求1所述的无线智能可穿戴装置,其特征在于,所述第一图像采集模块和所述第一摄像头通过第一CSI接口和第一GPIO接口相连,所述第二图像采集模块和所述第二摄像头通过第二CSI接口和第二GPIO接口相连,所述第一图像采集模块进一步配置为:经由第一GPIO接口连接到所述第一摄像头以向所述第一摄像头发送所述第一硬件触发信号;所述第一摄像头进一步配置为:响应于接收到所述第一硬件触发信号,开启曝光和图像拍摄并经由所述第一CSI接口向所述第一图像采集模块传输图像;所述第二图像采集模块进一步配置为:经由第二GPIO接口连接到所述第二摄像头以向所述第二摄像头发送所述第二硬件触发信号;所述第二摄像头进一步配置为:响应于接收到所述第二硬件触发信号,开启曝光和图像拍摄并经由所述第二CSI接口向所述第二图像采集模块传输图像。
- 根据权利要求3所述的无线智能可穿戴装置,其特征在于,所述第一处理器和所述第二处理器中的至少一个配置为:在所述第一图像采集模块和所述第二图像采集模块的持续使用过程中,使得所述第一无线通信模块和所述第二无线通信模块中的一方向另一方发送无线信号;确定所述一方发送所述无线信号的第一时刻和所述另一方接收所述无线信号的第二时刻的时钟计数器的数值差异,所述数值差异用于实现所述第一硬件触发信号和所述第二硬件触发信号的同步。
- 根据权利要求3所述的无线智能可穿戴装置,其特征在于,所述第一处理器和所述第二处理器中的至少一个配置为:在所述第一图像采集模块和所述第二图像采集模块的持续使用过程中,使得所述第一无线通信模块和所述第二无线通信模块各自接收来自智能设备的无线信号;确定所述第一无线通信模块接收所述无线信号的第三时刻和所述第二无线通信模块接收所述无线信号的第四时刻的时钟计数器的数值差异,所述数值差异用于实现所述第一硬件触发信号和所述第二硬件触发信号的同步。
- 根据权利要求4或5所述的无线智能可穿戴装置,其特征在于,所述第一处理器进一步配置为:在第一部的时钟计数器的数值为第一预定数值时产生所述第一硬件触发信号;所述第二处理器进一步配置为:在第二部的时钟计数器的数值为第二预定数值时产生所述第二硬件触发信号,其中,所述第一预定数值和所述第二预定数值的差异基于所述数值差异来设定,使得表征同一时刻。
- 根据权利要求4或5所述的无线智能可穿戴装置,其特征在于,所述第一处理器进一步配置为:产生基准硬件触发信号并获取触发时刻第一部的时钟计数器的基准数值;使得所述第一无线通信模块将所述基准数值传输给所述第二无线通信模块;在所述触发时刻后经过预定时延产生所述第一硬件触发信号;所述第二处理器进一步配置为:基于所述基准数值、所述预定时延以及时钟计数器的数值差异,来确定用于产生所述第二硬件触发信号的时钟计数 器的数值;在第二部的时钟计数器的数值达到所确定的数值时,产生所述第二硬件触发信号,从而使得所述第一硬件触发信号和所述第二硬件触发信号在同一时刻产生。
- 根据权利要求3所述的无线智能可穿戴装置,其特征在于,所述第一部和第二部各自包括亮度检测单元,其配置为检测对应的摄像头的环境光的亮度;所述第一处理器和所述第二处理器中的至少一个进一步配置为:基于所述第一部和第二部各自检出的亮度,设置所述第一摄像头和所述第二摄像头的曝光时间,使得摄像头对应的检出亮度越低则曝光时间越长。
- 根据权利要求8所述的无线智能可穿戴装置,其特征在于,所述第一处理器和所述第二处理器中的至少一个进一步配置为:对于要设置更长的曝光时间的第一摄像头或第二摄像头,使其相连的图像采集模块相较另一摄像头相连的图像采集模块,提前预定时间产生和发送对应的硬件触发信号。
- 根据权利要求9所述的无线智能可穿戴装置,其特征在于,所提前的预定时间为所述曝光时间的超出量的大约一半。
- 根据权利要求2所述的无线智能可穿戴装置,其特征在于,所述第一处理器和所述第二处理器中的至少一个进一步配置为:利用所述第一图像和所述第二图像用于生成全景视频或同步定位和地图构建。
- 一种无线智能可穿戴装置的图像采集方法,所述无线智能可穿戴装置包括可彼此无线通信的第一部和第二部,其特征在于,所述第一部包括第一处理器、第一无线通信模块、第一摄像头和第一图像采集模块且具有第一时钟,所述第二部包括第二处理器、第二无线通信模块、第二摄像头和第二图像采集模块且具有第二时钟,所述图像采集方法包括:在所述第一图像采集模块和所述第二图像采集模块的持续使用过程中,使得所述第一无线通信模块和所述第二无线通信模块执行彼此的无线通信和/或两者与智能设备的无线通信,并确定所述第一无线通信模块和所述第二无线通信模块各自执行所述无线通信的时钟差异;由所述第一图像采集模块向所述第一摄像头发送基于所述第一时钟的第一硬件触发信号,以触发所述第一摄像头拍摄第一图像并获取第一图像;以及由所述第二图像采集模块向所述第二摄像头发送基于所述第二时钟的第二硬件触发信号,以触发所述第二摄像头拍摄第二图像并获取第二图像,其中,利用所述时钟差异以实现所述第一硬件触发信号和所述第二硬件触发信号的同步。
- 根据权利要求12所述的图像采集方法,其特征在于,还包括:将所述第一图像采集模块经由第一GPIO接口连接到所述第一摄像头以向所述第一摄像头发送所述第一硬件触发信号;响应于接收到所述第一硬件触发信号,由所述第一摄像头开启曝光和图像拍摄并经由第一CSI接口向所述第一图像采集模块传输图像;将所述第二图像采集模块经由第二GPIO接口连接到所述第二摄像头以向所述第二摄像头发送所述第二硬件触发信号;响应于接收到所述第二硬件触发信号,由所述第二摄像头开启曝光和图像拍摄并经由第二CSI接口向所述第二图像采集模块传输图像。
- 根据权利要求13所述的图像采集方法,其特征在于,进一步包括,通过如下步骤来确定所述第一无线通信模块和所述第二无线通信模块各自执行所述无线通信的时钟差异:在所述第一图像采集模块和所述第二图像采集模块的持续使用过程中,使得所述第一无线通信模块和所述第二无线通信模块中的一方向另一方发送无线信号;确定所述一方发送所述无线信号的第一时刻和所述另一方接收所述无线信号的第二时刻的时钟计数器的数值差异作为所述时钟差异。
- 根据权利要求13所述的图像采集方法,其特征在于,进一步包括,通过如下步骤来确定所述第一无线通信模块和所述第二无线通信模块各自执行所述无线通信的时钟差异:在所述第一图像采集模块和所述第二图像采集模块的持续使用过程中,使得所述第一无线通信模块和所述第二无线通信模块各自接收来自智能设备的无线信号;确定所述第一无线通信模块接收所述无线信号的第三时刻和所述第二无线通信模块接收所述无线信号的第四时刻的时钟计数器的数值差异作为所述时钟差异。
- 根据权利要求14或15所述的图像采集方法,其特征在于,进一步包括:在第一部的时钟计数器的数值为第一预定数值时产生所述第一硬件触发信号;在第二部的时钟计数器的数值为第二预定数值时产生所述第二硬件触发信号,其中,所述第一预定数值和所述第二预定数值的差异基于所述数值差异来设定,使得表征同一时刻。
- 根据权利要求14或15所述的图像采集方法,其特征在于,进一步包括:在第一部产生基准硬件触发信号,并获取触发时刻第一部的时钟计数器的基准数值;经由所述第一无线通信模块,将所述基准数值传输给所述第二无线通信模块;在所述触发时刻后经过预定时延产生所述第一硬件触发信号;在第二部,基于所述基准数值、所述预定时延以及时钟计数器的数值差异,来确定用于产生所述第二硬件触发信号的时钟计数器的数值;在第二部的时钟计数器的数值达到所确定的数值时,产生所述第二硬件触发信号,从而使得所述第一硬件触发信号和所述第二硬件触发信号在同一时刻产生。
- 根据权利要求13所述的图像采集方法,其特征在于,还包括:检测第一摄像头和第二摄像头的环境光的亮度;设置所述第一摄像头和所述第二摄像头的曝光时间,使得环境光的亮度越低,对应的摄像头的曝光时间越长。
- 根据权利要求18所述的图像采集方法,其特征在于,还包括:对于要设置更长的曝光时间的第一摄像头或第二摄像头,使其相连的图像采集模块相较另一摄像头相连的图像采集模块,提前预定时间产生和发送对应的硬件触发信号。
- 根据权利要求19所述的图像采集方法,其特征在于,所提前的预定时间为所述曝光时间的超出量的大约一半。
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202211159526.5 | 2022-09-22 | ||
CN202211159526.5A CN115604402A (zh) | 2022-09-22 | 2022-09-22 | 一种无线智能可穿戴装置及其图像采集方法 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2024060763A1 true WO2024060763A1 (zh) | 2024-03-28 |
Family
ID=84844614
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2023/103757 WO2024060763A1 (zh) | 2022-09-22 | 2023-06-29 | 一种无线智能可穿戴装置及其图像采集方法 |
Country Status (2)
Country | Link |
---|---|
CN (2) | CN115604402A (zh) |
WO (1) | WO2024060763A1 (zh) |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN115604402A (zh) * | 2022-09-22 | 2023-01-13 | 恒玄科技(上海)股份有限公司(Cn) | 一种无线智能可穿戴装置及其图像采集方法 |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20170078554A1 (en) * | 2015-09-11 | 2017-03-16 | Casio Computer Co., Ltd. | Imaging apparatus and imaging control apparatus having synchronous type wireless communication function |
CN110248111A (zh) * | 2018-04-28 | 2019-09-17 | Oppo广东移动通信有限公司 | 控制拍摄的方法、装置、电子设备及计算机可读存储介质 |
CN112399167A (zh) * | 2020-12-08 | 2021-02-23 | 恒玄科技(北京)有限公司 | 一种用于无线通信的智能眼镜 |
CN115604403A (zh) * | 2022-09-22 | 2023-01-13 | 恒玄科技(上海)股份有限公司(Cn) | 一种无线智能可穿戴装置及其图像采集方法 |
-
2022
- 2022-09-22 CN CN202211159526.5A patent/CN115604402A/zh active Pending
- 2022-09-22 CN CN202211201167.5A patent/CN115604403A/zh active Pending
-
2023
- 2023-06-29 WO PCT/CN2023/103757 patent/WO2024060763A1/zh unknown
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20170078554A1 (en) * | 2015-09-11 | 2017-03-16 | Casio Computer Co., Ltd. | Imaging apparatus and imaging control apparatus having synchronous type wireless communication function |
CN110248111A (zh) * | 2018-04-28 | 2019-09-17 | Oppo广东移动通信有限公司 | 控制拍摄的方法、装置、电子设备及计算机可读存储介质 |
CN112399167A (zh) * | 2020-12-08 | 2021-02-23 | 恒玄科技(北京)有限公司 | 一种用于无线通信的智能眼镜 |
CN115604403A (zh) * | 2022-09-22 | 2023-01-13 | 恒玄科技(上海)股份有限公司(Cn) | 一种无线智能可穿戴装置及其图像采集方法 |
CN115604402A (zh) * | 2022-09-22 | 2023-01-13 | 恒玄科技(上海)股份有限公司(Cn) | 一种无线智能可穿戴装置及其图像采集方法 |
Also Published As
Publication number | Publication date |
---|---|
CN115604402A (zh) | 2023-01-13 |
CN115604403A (zh) | 2023-01-13 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9654672B1 (en) | Synchronized capture of image and non-image sensor data | |
US9813783B2 (en) | Multi-camera dataset assembly and management with high precision timestamp requirements | |
WO2018228353A1 (zh) | 一种多相机系统同步曝光的控制方法、装置及终端设备 | |
US7511764B2 (en) | Digital camera synchronization | |
WO2018228352A1 (zh) | 一种同步曝光方法、装置及终端设备 | |
JP5855538B2 (ja) | 撮像装置、撮像システム、撮像方法及びプログラム | |
WO2024060763A1 (zh) | 一种无线智能可穿戴装置及其图像采集方法 | |
JP5817664B2 (ja) | 撮像装置、撮像方法及びプログラム | |
US20170013192A1 (en) | Electronic device and method for increasing a frame rate of a plurality of pictures photographed by an electronic device | |
US20140139740A1 (en) | Imaging device, information processing device, information processing method, and method for synchronizing frame data output | |
TW201224779A (en) | Motion initiated time synchronization | |
JP2011234360A (ja) | シャッターラグを回避する画像キャプチャモジュール及び画像キャプチャ方法 | |
CN105611144B (zh) | 摄像装置、摄像控制方法以及记录介质 | |
KR20230053711A (ko) | 듀얼 애퍼처 카메라 시스템에서의 프레임 동기화 | |
WO2018227329A1 (zh) | 一种同步曝光的方法、装置及终端设备 | |
WO2018056781A1 (ko) | 클라이언트 장치 및 그 로컬 클럭 스큐 보정 방법 | |
CN110177220B (zh) | 一种具有外部授时功能的相机及其控制方法 | |
JP2004297414A (ja) | 同期撮影機能を有するカメラおよび画像記録装置 | |
KR102617898B1 (ko) | 다수의 센서 장치들에서의 이미지 캡처 동기화 | |
CN113489880B (zh) | 电子设备、信息同步方法和计算机可读存储介质 | |
WO2023201822A1 (zh) | 多相机同步校正方法、装置及存储介质 | |
WO2018227327A1 (zh) | 一种同步曝光的方法、装置及终端设备 | |
CN112135007B (zh) | 一种流媒体视角切换方法及系统 | |
JP2011101218A (ja) | 撮像素子一体型交換レンズ式のカメラシステムとその動画生成方法 | |
CN117336419A (zh) | 拍摄方法、装置、电子设备及可读存储介质 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 23867058 Country of ref document: EP Kind code of ref document: A1 |