US20190158721A1 - Multi-camera synchronization through receiver hub back channel - Google Patents
Multi-camera synchronization through receiver hub back channel Download PDFInfo
- Publication number
- US20190158721A1 US20190158721A1 US15/910,567 US201815910567A US2019158721A1 US 20190158721 A1 US20190158721 A1 US 20190158721A1 US 201815910567 A US201815910567 A US 201815910567A US 2019158721 A1 US2019158721 A1 US 2019158721A1
- Authority
- US
- United States
- Prior art keywords
- interface circuit
- serial interface
- clock
- control information
- hub
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- H04N5/23203—
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/66—Remote control of cameras or camera parts, e.g. by remote control devices
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/10—Adaptations for transmission by electrical cable
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
Definitions
- Advanced Driver Assistance Systems are systems that are included in automobiles and other vehicles to enhance operational safety.
- An advanced driver assistance system may include a variety features can provide better visibility into conditions outside the vehicle.
- a multi-camera surround view systems a multiple cameras provide video streams that are used to generate a bird-eye view of the vehicle's surroundings in real time.
- Such systems can be used to monitor vehicle surroundings when the driving, parking, changing lanes, etc.
- Such a solution helps a driver park safely by providing a view of the entire 360 degree surrounding of the vehicle.
- a driver assistance system includes a first camera, a second camera, a first serial interface circuit, a second serial interface circuit, and a hub interface circuit.
- the first serial interface circuit is coupled to the first camera.
- the second serial interface circuit is coupled to the second camera.
- the hub interface circuit is coupled to the first serial interface circuit and the second serial interface circuit.
- the hub interface circuit is configured to receive transmissions from the first serial interface circuit and the second serial interface circuit, and to transmit control information to the first serial interface circuit and the second serial interface circuit.
- the hub interface is also configured to encode a clock signal in the control information.
- a video control system in another example, includes a hub interface circuit and a plurality of serial interface circuits.
- the hub interface circuit includes a clock generator and a plurality of transceivers.
- the clock generator is configured to synchronize a clock signal to a reference clock.
- Each of the transceivers is configured to communicate bidirectionally with a serial interface circuit, and to encode the clock signal in control information to be transmitted to a serial interface circuit.
- Each of the serial interface circuits is configured to bidirectionally communicate with the hub interface circuit.
- Each of the serial interface circuits includes a clock recovery circuit and a camera interface.
- the clock recovery circuit is configured to extract the clock signal from the control information transmitted by the hub interface circuit.
- the camera interface is configured to provide, to a camera, a synchronization signal generated from the clock signal extracted from the control information.
- a driver assistance system includes a hub interface circuit, a plurality of serial interface circuits, and a plurality of cameras.
- the hub interface circuit includes a clock generator, frame sync generation circuitry, and a plurality of transceivers.
- the clock generator is configured to synchronize a clock signal to a reference clock.
- the frame sync generation circuitry is configured to generate a frame sync signal.
- Each of the transceivers is configured to communicate bidirectionally with a serial interface circuit, and to encode the clock signal and the frame sync signal in control information to be transmitted to a serial interface circuit.
- Each of the serial interface circuits is configured to bidirectionally communicate with the hub interface circuit.
- Each of the serial interface circuits includes a clock recovery circuit, a clock generation circuitry, frame sync recovery circuitry, and a camera interface.
- the clock recovery circuit is configured to extract the clock signal from the control information transmitted by the hub interface circuit.
- the clock generation circuitry is configured to generate a clock based on the clock signal extracted from the control information.
- the frame sync recovery circuitry is configured to extract the frame sync signal from the control information.
- the camera interface is configured to provide the clock and the frame sync signal to a camera. Each of the cameras is coupled to one of the serial interface circuits.
- FIG. 1 shows a block diagram for an example of a surround view system of an Advanced Driver Assistance Systems in accordance with the present disclosure
- FIG. 2 shows a block diagram for an example of a hub interface circuit for use in a surround view system in accordance with the present disclosure
- FIG. 3 shows a block diagram for an example of a serial interface circuit for use in a surround view system in accordance with the present disclosure
- FIG. 4 shows a block diagram for an example of an image processor for use in a surround view system in accordance with the present disclosure.
- a bird's eye view image is stitched together from images captured by multiple cameras.
- the cameras operate asynchronously.
- An image processor buffers frames of the asynchronously acquired video and stiches frame data together to form the surround view image.
- Such systems include substantial frame memory and each camera includes independent timing circuitry.
- a hub interface circuit is connected to a number of serial interface circuits, each of serial interface circuits is interfaced to a camera.
- the hub interface circuit transmits timing information to the serial interface circuits.
- the timing information is encoded in control information transmitted to the serial interface circuits by the hub interface circuit, and includes a clock signal and a frame sync signal.
- the serial interface circuits recover the timing information from the transmitted control information and apply the clock and frame sync signals to synchronize image acquisition by the cameras and transfer of the image data to the hub interface circuit. Because the timing of the cameras and the serial interface circuits is derived from signals transmitted by the hub interface circuit, implementations of the serial interface circuits include no crystal timing reference. Because the image data acquired by the cameras and transferred to the hub interface circuit is synchronized using the timing information provided the hub interface circuit, the amount of frame memory included in an image processor, and latency in producing a surround view image is substantially reduced.
- FIG. 1 shows a block diagram for an example of a surround view system 100 of an Advanced Driver Assistance System in accordance with the present disclosure.
- the surround view system 100 includes a hub interface circuit 102 , a plurality of serial interface circuits 104 , a plurality of cameras 106 , a reference clock circuit 108 , an image processor 110 , and a display 112 .
- each of the cameras 106 is disposed to capture an image of a different side of a vehicle.
- a first camera 106 is disposed to capture an image of a view from a front side of the vehicle
- a second camera 106 is disposed to capture an image of a view from a back side of the vehicle
- a third camera 106 is disposed to capture an image of a view from a driver side of the vehicle
- a fourth camera 106 is disposed to capture an image of a view from a passenger side of the vehicle.
- Each of the cameras 106 is coupled to one of the serial interface circuits 104 .
- Each of the serial interface circuits 104 is coupled to the hub interface circuit 102 via a coaxial cable, a shielded twisted pair, or other signal conductor.
- the serial interface circuits 104 receive video data from the cameras 106 and transfer the video data to the hub interface circuit 102 .
- the serial interface circuits 104 also transfer control signals to the cameras 106 .
- the serial interface circuits 104 transfer synchronization signals, such as clock and frame sync signals to the cameras 106 .
- the hub interface circuit 102 is coupled to the image processor 110 , and transfers the video data received from the serial interface circuits 104 to the image processor 110 for processing and display. For example, some implementations of the hub interface circuit 102 interleave lines or frames of video data received from the serial interface circuits 104 and transfer the interleaved video data to the image processor 110 .
- the image processor 110 includes a processor, such as a general-purpose microprocessor, a digital signal processor, a graphics processing unit, or other device configured to process image data.
- the image processor 110 combines the video data received from the serial interface circuits 104 to form a composite image that simulates the view of a camera positioned above the vehicle.
- the image processor 110 is coupled to the display 112 .
- the composite image produced by the image processor 110 is displayed on the display 112 .
- the display 112 is a liquid crystal device, an organic light emitting diode device, or any other device suitable for producing a viewable image.
- the reference clock circuit 108 provides a reference clock to the hub interface circuit 102 .
- an implementation of the reference clock circuit 108 includes a crystal frequency reference or other reference frequency source that controls a frequency of the reference clock provided to the hub interface circuit 102 .
- the hub interface circuit 102 applies the reference clock signal to generate timing information that is transferred to the serial interface circuits 104 by the hub interface circuit 102 .
- the timing information includes frame synchronization information and clock timing information.
- the hub interface circuit 102 communicates with the serial interface circuits 104 via a forward channel and a back channel.
- the forward channel carries image data from the serial interface circuits 104 to the hub interface circuit 102 .
- the back channel carries control information from the hub interface circuit 102 to the serial interface circuits 104 , and carries status information from the serial interface circuits 104 to the hub interface circuit 102 .
- the hub interface circuit 102 encodes the frame synchronization information and the clock timing information in the control information carried on the back channel.
- the serial interface circuits 104 recovers the clock timing information from the control information received from the hub interface circuit 102 , and applies the clock timing information to synchronize the timing of the serial interface circuits 104 and the cameras 106 .
- the serial interface circuits 104 recover the frame sync information from the control information received from the hub interface circuit 102 , and apply the frame sync information to synchronize the acquisition of image frames in the cameras 106 , and synchronize the timing of image frame transfer from the serial interface circuits 104 to the hub interface circuit 102 .
- the serial interface circuits 104 apply the clock timing information and the frame sync information recovered from the control information received from the hub interface circuit 102 to synchronize image acquisition in the cameras 106 and image transfer to the hub interface circuit 102 , the amount of frame buffer memory included in the image processor 110 is reduced. Because the image data received from the serial interface circuits 104 is synchronized, the image processor 110 need not wait for asynchronous image data from the cameras 106 , and as a result, the time needed by the image processor 110 to produce a composite image based on images captured by the cameras 106 (e.g., a surround view image) is reduced.
- FIG. 2 shows a block diagram for an example of a hub interface circuit 102 in accordance with the present disclosure.
- the hub interface circuit 102 includes a clock generation circuit 202 , a frame sync generation circuit 204 , an image processor interface 206 , and transceivers 208 .
- Some implementations of the hub interface circuit 102 include a transceiver 208 for bidirectional communication with each of the serial interface circuits 104 .
- the hub interface circuit 102 includes four transceivers 208 .
- the transceivers 208 include circuitry for providing serial communication with the serial interface circuits 104 via a coaxial cable, a shielded twisted pair, or other signal conductor.
- a forward channel receiver of the transceiver 208 is to receive image data transmitted by a serial interface circuit 104
- a back channel transmitter of the transceiver 208 is to transmit control data, including the frame sync information and the clock timing information, to the serial interface circuit 104 .
- the clock generation circuit 202 includes circuitry to generate clock signals.
- an implementation of the clock generation circuit 202 includes phase locked loop circuitry or other frequency synthesis circuitry that produces a selected clock frequency based on the reference clock signal 114 provided by the reference clock circuit 108 .
- the clock generation circuit 202 generates a clock signal that is provided to the transceivers 208 for communication to the serial interface circuits 104 .
- the transceivers 208 encode the clock signal in the control information transmitted to the serial interface circuits 104 by the transceivers 208 .
- the some implementations of transceivers 208 provide transitions in the control information transmitted to the serial interface circuits 104 at a frequency related to the clock signal.
- the clock generation circuit 202 also provides a clock signal to the frame sync generation circuit 204 .
- the frame sync generation circuit 204 includes circuitry to generate a frame sync signal based on the clock signal received from the clock generation circuit 202 .
- the frame sync signal specifies the timing of image capture by the cameras 106 , and of image transfer from the serial interface circuits 104 to the hub interface circuit 102 .
- the frame sync signal generated by the frame sync generation circuit 204 is provided to the transceivers 208 for communication to the serial interface circuits 104 .
- the transceivers 208 encode the frame sync signal in the control information transmitted to the serial interface circuits 104 by the transceivers 208 . For example, some implementations of the transceivers 208 provide transitions in the control information transmitted to the serial interface circuits 104 at a frequency corresponding to the frame sync signal.
- the image processor interface 206 includes circuitry for communicating with the image processor 110 .
- the image processor interface 206 includes circuitry that transfers image data received from the serial interface circuits 104 to the image processor 110 .
- the image processor interface 206 implements a camera serial interface specified by the Mobile Industry Processor Interface Alliance (MIPI CSI-2).
- the hub interface circuit 102 may include additional circuitry that has been omitted from FIG. 2 in the interest of clarity.
- some implementations of the hub interface circuit 102 include a control interface (e.g., an inter-integrated circuit interface) via which the hub interface circuit 102 receives information, from a device external to the hub interface circuit 102 , to be transferred to the serial interface circuits 104 via the transceivers 208 , and via which status received from the serial interface circuits 104 is transferred to a device external to the hub interface circuit 102 .
- a control interface e.g., an inter-integrated circuit interface
- FIG. 3 shows a block diagram for an example of a serial interface circuit 104 in accordance with the present disclosure.
- the serial interface circuit 104 includes camera interface circuitry 302 , clock recovery circuitry 304 , clock generation circuitry 306 , frame sync recovery circuitry 308 , and a transceiver 310 .
- the transceiver 310 includes circuitry that bidirectionally communicates with the transceiver 208 .
- the transceiver 310 includes circuitry to communicate with the transceiver 208 via a coaxial cable, shielded twisted pair, or other signal conductor.
- a forward channel transmitter of the transceiver 310 is to transmit image data to the hub interface circuit 102
- a back channel receiver of the transceiver 310 is to receive control data, including the frame sync information and the clock timing information, transmitted by the hub interface circuit 102 .
- the clock recovery circuitry 304 is coupled to the transceiver 310 , and includes circuitry to recovery the clock timing information from the control information received from the serial interface circuits 104 .
- an implementation of the clock recovery circuitry 304 includes circuitry to extract transitions or pulses from the signal transmitted on the back channel by the hub interface circuit 102 .
- the clock recovery circuitry 304 provides signal indicative of the clock signal encoded by the hub interface circuit 102 in the back channel signal to the clock generation circuitry 306 .
- the clock generation circuitry 306 generates a clock signal that is synchronized to the signal extracted from the back channel signal by the clock recovery circuitry 304 .
- the clock generation circuitry 306 includes phase-locked loop circuitry or other frequency synthesis circuitry that produces a clock signal that is synchronized to the clock timing information encoded in the control information received from the hub interface circuit 102 .
- the clock signal generated by the clock generation circuitry 306 is provided to the camera interface circuitry 302 and the frame sync recovery circuitry 308 . Because the serial interface circuit 104 produces clock and other timing signals from the timing information received from the hub interface circuit 102 , some implementations of the serial interface circuit 104 include no crystal timing reference.
- the frame sync recovery circuitry 308 includes circuitry to recover the frame sync information from the control information received by the serial interface circuit 104 from the hub interface circuit 102 .
- a predetermined pattern of transitions in the control information received from the hub interface circuit 102 indicate frame sync timing.
- the frame sync recovery circuitry 308 identifies the pattern of transitions and provides the recovered frame sync signal to the camera interface circuitry 302 .
- the camera interface circuitry 302 includes circuitry for communication with a camera 106 .
- the camera interface circuitry 302 includes circuitry that receives image data from a camera 106 and provides synchronization signals, such as the clock generated by the clock generation circuitry 306 and the frame sync signal recovered by the frame sync recovery circuitry 308 , to the camera 106 .
- the camera interface circuitry 302 implements a camera serial interface specified by the Mobile Industry Processor Interface Alliance (MIPI CSI-2). Image data received from the camera 106 by the camera interface circuitry 302 is provided to the transceiver 310 for transmission to the hub interface circuit 102 .
- MIPI CSI-2 Mobile Industry Processor Interface Alliance
- FIG. 4 shows a block diagram for an example of an image processor 110 in accordance with the present disclosure.
- the image processor 110 includes hub communication circuitry 402 , a frame buffer 404 , a processor 406 , storage 408 , and a display interface 410 .
- the hub communication circuitry 402 includes circuitry that provides for communication with the hub interface circuit 102 .
- the hub communication circuitry 402 includes circuitry that is complementary to the circuitry of the transceivers 208 of the hub interface circuit 102 .
- the hub communication circuitry 402 implements a camera serial interface as specified by the Mobile Industry Processor Interface Alliance (MIPI CSI-2).
- MIPI CSI-2 Mobile Industry Processor Interface Alliance
- Image data received via the hub interface circuit 102 is stored in the frame buffer 404 .
- the frame buffer 404 includes sufficient memory to store a number of frames of video captured by each of the cameras 106 . Because the video frames captured by the cameras 106 and received by the frame buffer 404 are synchronized in time, the amount of memory included in the frame buffer 404 is reduced relative to implementations that do not synchronize image capture at the clock and frame level.
- the frame buffer 404 is coupled to the processor 406 .
- the processor 406 is a general-purpose microprocessor, a digital signal processor, a graphics processing unit, or other device configured to process image data.
- the processor 406 is coupled to the storage 408 .
- the storage 408 includes memory (e.g., volatile or non-volatile memory) that stores instructions to be executed by the processor 406 .
- the storage includes surround view processing instructions 412 .
- the processor 406 retrieves the surround view processing instructions 412 from the storage 408 .
- Execution of the surround view processing instructions 412 by the processor 406 causes the processor 406 to retrieve image data from the frame buffer 404 for each of the cameras 106 and to stitch the image data together to form a composite image (e.g., a surround view image) that includes image data provided by each of the cameras 106 . Because the image data from the cameras 106 is synchronized, the processor 406 need not wait for or search for time correspondent image data from each of the cameras 106 , and latency in producing a composite image is reduced relative to implementations that do not synchronize image capture at the clock and frame level.
- a composite image e.g., a surround view image
- the processor 406 is coupled to the display interface 410 , and the surround view processing instructions 412 cause the processor 406 to provide surround view images to the display 112 via the display interface 410 .
- the display interface 410 serialize the surround view image data provided by the processor 406 and transmit the surround view image data to the display 112 or circuitry coupled to the display 112 .
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Studio Devices (AREA)
Abstract
Description
- This application claims priority to U.S. Provisional Application No. 62/587,500, filed Nov. 17, 2017, which is hereby incorporated by reference.
- Advanced Driver Assistance Systems (ADAS) are systems that are included in automobiles and other vehicles to enhance operational safety. An advanced driver assistance system may include a variety features can provide better visibility into conditions outside the vehicle. In a multi-camera surround view systems, a multiple cameras provide video streams that are used to generate a bird-eye view of the vehicle's surroundings in real time. Such systems can be used to monitor vehicle surroundings when the driving, parking, changing lanes, etc. Such a solution helps a driver park safely by providing a view of the entire 360 degree surrounding of the vehicle.
- A multi-view driver assistance system and video control system suitable for use therein are disclosed herein. In one example, a driver assistance system includes a first camera, a second camera, a first serial interface circuit, a second serial interface circuit, and a hub interface circuit. The first serial interface circuit is coupled to the first camera. The second serial interface circuit is coupled to the second camera. The hub interface circuit is coupled to the first serial interface circuit and the second serial interface circuit. The hub interface circuit is configured to receive transmissions from the first serial interface circuit and the second serial interface circuit, and to transmit control information to the first serial interface circuit and the second serial interface circuit. The hub interface is also configured to encode a clock signal in the control information.
- In another example, a video control system includes a hub interface circuit and a plurality of serial interface circuits. The hub interface circuit includes a clock generator and a plurality of transceivers. The clock generator is configured to synchronize a clock signal to a reference clock. Each of the transceivers is configured to communicate bidirectionally with a serial interface circuit, and to encode the clock signal in control information to be transmitted to a serial interface circuit. Each of the serial interface circuits is configured to bidirectionally communicate with the hub interface circuit. Each of the serial interface circuits includes a clock recovery circuit and a camera interface. The clock recovery circuit is configured to extract the clock signal from the control information transmitted by the hub interface circuit. The camera interface is configured to provide, to a camera, a synchronization signal generated from the clock signal extracted from the control information.
- In a further example, a driver assistance system includes a hub interface circuit, a plurality of serial interface circuits, and a plurality of cameras. The hub interface circuit includes a clock generator, frame sync generation circuitry, and a plurality of transceivers. The clock generator is configured to synchronize a clock signal to a reference clock. The frame sync generation circuitry is configured to generate a frame sync signal. Each of the transceivers is configured to communicate bidirectionally with a serial interface circuit, and to encode the clock signal and the frame sync signal in control information to be transmitted to a serial interface circuit. Each of the serial interface circuits is configured to bidirectionally communicate with the hub interface circuit. Each of the serial interface circuits includes a clock recovery circuit, a clock generation circuitry, frame sync recovery circuitry, and a camera interface. The clock recovery circuit is configured to extract the clock signal from the control information transmitted by the hub interface circuit. The clock generation circuitry is configured to generate a clock based on the clock signal extracted from the control information. The frame sync recovery circuitry is configured to extract the frame sync signal from the control information. The camera interface is configured to provide the clock and the frame sync signal to a camera. Each of the cameras is coupled to one of the serial interface circuits.
- For a detailed description of various examples, reference will now be made to the accompanying drawings in which:
-
FIG. 1 shows a block diagram for an example of a surround view system of an Advanced Driver Assistance Systems in accordance with the present disclosure; -
FIG. 2 shows a block diagram for an example of a hub interface circuit for use in a surround view system in accordance with the present disclosure; -
FIG. 3 shows a block diagram for an example of a serial interface circuit for use in a surround view system in accordance with the present disclosure; and -
FIG. 4 shows a block diagram for an example of an image processor for use in a surround view system in accordance with the present disclosure. - Certain terms have been used throughout this description and claims to refer to particular system components. As one skilled in the art will appreciate, different parties may refer to a component by different names. This document does not intend to distinguish between components that differ in name but not function. In this disclosure and claims, the terms “including” and “comprising” are used in an open-ended fashion, and thus should be interpreted to mean “including, but not limited to . . . . ” Also, the term “couple” or “couples” is intended to mean either an indirect or direct wired or wireless connection. Thus, if a first device couples to a second device, that connection may be through a direct connection or through an indirect connection via other devices and connections. The recitation “based on” is intended to mean “based at least in part on.” Therefore, if X is based on Y, X may be a function of Y and any number of other factors.
- In a surround view system provided as part of an advanced driver assistance system, a bird's eye view image is stitched together from images captured by multiple cameras. In some surround view systems, the cameras operate asynchronously. An image processor buffers frames of the asynchronously acquired video and stiches frame data together to form the surround view image. Such systems include substantial frame memory and each camera includes independent timing circuitry.
- In the surround view systems disclosed herein, a hub interface circuit is connected to a number of serial interface circuits, each of serial interface circuits is interfaced to a camera. The hub interface circuit transmits timing information to the serial interface circuits. The timing information is encoded in control information transmitted to the serial interface circuits by the hub interface circuit, and includes a clock signal and a frame sync signal. The serial interface circuits recover the timing information from the transmitted control information and apply the clock and frame sync signals to synchronize image acquisition by the cameras and transfer of the image data to the hub interface circuit. Because the timing of the cameras and the serial interface circuits is derived from signals transmitted by the hub interface circuit, implementations of the serial interface circuits include no crystal timing reference. Because the image data acquired by the cameras and transferred to the hub interface circuit is synchronized using the timing information provided the hub interface circuit, the amount of frame memory included in an image processor, and latency in producing a surround view image is substantially reduced.
-
FIG. 1 shows a block diagram for an example of asurround view system 100 of an Advanced Driver Assistance System in accordance with the present disclosure. Thesurround view system 100 includes ahub interface circuit 102, a plurality ofserial interface circuits 104, a plurality ofcameras 106, areference clock circuit 108, animage processor 110, and adisplay 112. In an Advanced Driver Assistance System, each of thecameras 106 is disposed to capture an image of a different side of a vehicle. For example, afirst camera 106 is disposed to capture an image of a view from a front side of the vehicle, asecond camera 106 is disposed to capture an image of a view from a back side of the vehicle, athird camera 106 is disposed to capture an image of a view from a driver side of the vehicle, and afourth camera 106 is disposed to capture an image of a view from a passenger side of the vehicle. Each of thecameras 106 is coupled to one of theserial interface circuits 104. Each of theserial interface circuits 104 is coupled to thehub interface circuit 102 via a coaxial cable, a shielded twisted pair, or other signal conductor. Theserial interface circuits 104 receive video data from thecameras 106 and transfer the video data to thehub interface circuit 102. Theserial interface circuits 104 also transfer control signals to thecameras 106. For example, theserial interface circuits 104 transfer synchronization signals, such as clock and frame sync signals to thecameras 106. - The
hub interface circuit 102 is coupled to theimage processor 110, and transfers the video data received from theserial interface circuits 104 to theimage processor 110 for processing and display. For example, some implementations of thehub interface circuit 102 interleave lines or frames of video data received from theserial interface circuits 104 and transfer the interleaved video data to theimage processor 110. Theimage processor 110 includes a processor, such as a general-purpose microprocessor, a digital signal processor, a graphics processing unit, or other device configured to process image data. Theimage processor 110 combines the video data received from theserial interface circuits 104 to form a composite image that simulates the view of a camera positioned above the vehicle. Theimage processor 110 is coupled to thedisplay 112. The composite image produced by theimage processor 110 is displayed on thedisplay 112. Thedisplay 112 is a liquid crystal device, an organic light emitting diode device, or any other device suitable for producing a viewable image. - The
reference clock circuit 108 provides a reference clock to thehub interface circuit 102. For example, an implementation of thereference clock circuit 108 includes a crystal frequency reference or other reference frequency source that controls a frequency of the reference clock provided to thehub interface circuit 102. Thehub interface circuit 102 applies the reference clock signal to generate timing information that is transferred to theserial interface circuits 104 by thehub interface circuit 102. The timing information includes frame synchronization information and clock timing information. - The
hub interface circuit 102 communicates with theserial interface circuits 104 via a forward channel and a back channel. The forward channel carries image data from theserial interface circuits 104 to thehub interface circuit 102. The back channel carries control information from thehub interface circuit 102 to theserial interface circuits 104, and carries status information from theserial interface circuits 104 to thehub interface circuit 102. Thehub interface circuit 102 encodes the frame synchronization information and the clock timing information in the control information carried on the back channel. Theserial interface circuits 104 recovers the clock timing information from the control information received from thehub interface circuit 102, and applies the clock timing information to synchronize the timing of theserial interface circuits 104 and thecameras 106. Theserial interface circuits 104 recover the frame sync information from the control information received from thehub interface circuit 102, and apply the frame sync information to synchronize the acquisition of image frames in thecameras 106, and synchronize the timing of image frame transfer from theserial interface circuits 104 to thehub interface circuit 102. - Because the
serial interface circuits 104 apply the clock timing information and the frame sync information recovered from the control information received from thehub interface circuit 102 to synchronize image acquisition in thecameras 106 and image transfer to thehub interface circuit 102, the amount of frame buffer memory included in theimage processor 110 is reduced. Because the image data received from theserial interface circuits 104 is synchronized, theimage processor 110 need not wait for asynchronous image data from thecameras 106, and as a result, the time needed by theimage processor 110 to produce a composite image based on images captured by the cameras 106 (e.g., a surround view image) is reduced. -
FIG. 2 shows a block diagram for an example of ahub interface circuit 102 in accordance with the present disclosure. Thehub interface circuit 102 includes aclock generation circuit 202, a framesync generation circuit 204, animage processor interface 206, andtransceivers 208. Some implementations of thehub interface circuit 102 include atransceiver 208 for bidirectional communication with each of theserial interface circuits 104. For example, to interface with fourserial interface circuits 104, thehub interface circuit 102 includes fourtransceivers 208. In some implementations of thehub interface circuit 102, thetransceivers 208 include circuitry for providing serial communication with theserial interface circuits 104 via a coaxial cable, a shielded twisted pair, or other signal conductor. For example, a forward channel receiver of thetransceiver 208 is to receive image data transmitted by aserial interface circuit 104, and a back channel transmitter of thetransceiver 208 is to transmit control data, including the frame sync information and the clock timing information, to theserial interface circuit 104. - The
clock generation circuit 202 includes circuitry to generate clock signals. For example, an implementation of theclock generation circuit 202 includes phase locked loop circuitry or other frequency synthesis circuitry that produces a selected clock frequency based on thereference clock signal 114 provided by thereference clock circuit 108. Theclock generation circuit 202 generates a clock signal that is provided to thetransceivers 208 for communication to theserial interface circuits 104. Thetransceivers 208 encode the clock signal in the control information transmitted to theserial interface circuits 104 by thetransceivers 208. For example, the some implementations oftransceivers 208 provide transitions in the control information transmitted to theserial interface circuits 104 at a frequency related to the clock signal. Theclock generation circuit 202 also provides a clock signal to the framesync generation circuit 204. - The frame
sync generation circuit 204 includes circuitry to generate a frame sync signal based on the clock signal received from theclock generation circuit 202. The frame sync signal specifies the timing of image capture by thecameras 106, and of image transfer from theserial interface circuits 104 to thehub interface circuit 102. The frame sync signal generated by the framesync generation circuit 204 is provided to thetransceivers 208 for communication to theserial interface circuits 104. Thetransceivers 208 encode the frame sync signal in the control information transmitted to theserial interface circuits 104 by thetransceivers 208. For example, some implementations of thetransceivers 208 provide transitions in the control information transmitted to theserial interface circuits 104 at a frequency corresponding to the frame sync signal. - The
image processor interface 206 includes circuitry for communicating with theimage processor 110. For example, theimage processor interface 206 includes circuitry that transfers image data received from theserial interface circuits 104 to theimage processor 110. In some implementations, theimage processor interface 206 implements a camera serial interface specified by the Mobile Industry Processor Interface Alliance (MIPI CSI-2). - The
hub interface circuit 102 may include additional circuitry that has been omitted fromFIG. 2 in the interest of clarity. For example, some implementations of thehub interface circuit 102 include a control interface (e.g., an inter-integrated circuit interface) via which thehub interface circuit 102 receives information, from a device external to thehub interface circuit 102, to be transferred to theserial interface circuits 104 via thetransceivers 208, and via which status received from theserial interface circuits 104 is transferred to a device external to thehub interface circuit 102. -
FIG. 3 shows a block diagram for an example of aserial interface circuit 104 in accordance with the present disclosure. Theserial interface circuit 104 includescamera interface circuitry 302,clock recovery circuitry 304,clock generation circuitry 306, framesync recovery circuitry 308, and atransceiver 310. Thetransceiver 310 includes circuitry that bidirectionally communicates with thetransceiver 208. In some implementations, thetransceiver 310 includes circuitry to communicate with thetransceiver 208 via a coaxial cable, shielded twisted pair, or other signal conductor. For example, a forward channel transmitter of thetransceiver 310 is to transmit image data to thehub interface circuit 102, and a back channel receiver of thetransceiver 310 is to receive control data, including the frame sync information and the clock timing information, transmitted by thehub interface circuit 102. - The
clock recovery circuitry 304 is coupled to thetransceiver 310, and includes circuitry to recovery the clock timing information from the control information received from theserial interface circuits 104. For example, an implementation of theclock recovery circuitry 304 includes circuitry to extract transitions or pulses from the signal transmitted on the back channel by thehub interface circuit 102. Theclock recovery circuitry 304 provides signal indicative of the clock signal encoded by thehub interface circuit 102 in the back channel signal to theclock generation circuitry 306. - The
clock generation circuitry 306 generates a clock signal that is synchronized to the signal extracted from the back channel signal by theclock recovery circuitry 304. In some implementations, theclock generation circuitry 306 includes phase-locked loop circuitry or other frequency synthesis circuitry that produces a clock signal that is synchronized to the clock timing information encoded in the control information received from thehub interface circuit 102. The clock signal generated by theclock generation circuitry 306 is provided to thecamera interface circuitry 302 and the framesync recovery circuitry 308. Because theserial interface circuit 104 produces clock and other timing signals from the timing information received from thehub interface circuit 102, some implementations of theserial interface circuit 104 include no crystal timing reference. - The frame
sync recovery circuitry 308 includes circuitry to recover the frame sync information from the control information received by theserial interface circuit 104 from thehub interface circuit 102. For example, in some implementations a predetermined pattern of transitions in the control information received from thehub interface circuit 102 indicate frame sync timing. The framesync recovery circuitry 308 identifies the pattern of transitions and provides the recovered frame sync signal to thecamera interface circuitry 302. - The
camera interface circuitry 302 includes circuitry for communication with acamera 106. For example, thecamera interface circuitry 302 includes circuitry that receives image data from acamera 106 and provides synchronization signals, such as the clock generated by theclock generation circuitry 306 and the frame sync signal recovered by the framesync recovery circuitry 308, to thecamera 106. The In some implementations, thecamera interface circuitry 302 implements a camera serial interface specified by the Mobile Industry Processor Interface Alliance (MIPI CSI-2). Image data received from thecamera 106 by thecamera interface circuitry 302 is provided to thetransceiver 310 for transmission to thehub interface circuit 102. -
FIG. 4 shows a block diagram for an example of animage processor 110 in accordance with the present disclosure. Theimage processor 110 includeshub communication circuitry 402, aframe buffer 404, aprocessor 406,storage 408, and adisplay interface 410. Thehub communication circuitry 402 includes circuitry that provides for communication with thehub interface circuit 102. For example thehub communication circuitry 402 includes circuitry that is complementary to the circuitry of thetransceivers 208 of thehub interface circuit 102. In some implementations, thehub communication circuitry 402 implements a camera serial interface as specified by the Mobile Industry Processor Interface Alliance (MIPI CSI-2). - Image data received via the
hub interface circuit 102 is stored in theframe buffer 404. Theframe buffer 404 includes sufficient memory to store a number of frames of video captured by each of thecameras 106. Because the video frames captured by thecameras 106 and received by theframe buffer 404 are synchronized in time, the amount of memory included in theframe buffer 404 is reduced relative to implementations that do not synchronize image capture at the clock and frame level. - The
frame buffer 404 is coupled to theprocessor 406. In some implementations, theprocessor 406 is a general-purpose microprocessor, a digital signal processor, a graphics processing unit, or other device configured to process image data. Theprocessor 406 is coupled to thestorage 408. Thestorage 408 includes memory (e.g., volatile or non-volatile memory) that stores instructions to be executed by theprocessor 406. The storage includes surroundview processing instructions 412. Theprocessor 406 retrieves the surroundview processing instructions 412 from thestorage 408. Execution of the surroundview processing instructions 412 by theprocessor 406 causes theprocessor 406 to retrieve image data from theframe buffer 404 for each of thecameras 106 and to stitch the image data together to form a composite image (e.g., a surround view image) that includes image data provided by each of thecameras 106. Because the image data from thecameras 106 is synchronized, theprocessor 406 need not wait for or search for time correspondent image data from each of thecameras 106, and latency in producing a composite image is reduced relative to implementations that do not synchronize image capture at the clock and frame level. - The
processor 406 is coupled to thedisplay interface 410, and the surroundview processing instructions 412 cause theprocessor 406 to provide surround view images to thedisplay 112 via thedisplay interface 410. For example, some implementations of thedisplay interface 410 serialize the surround view image data provided by theprocessor 406 and transmit the surround view image data to thedisplay 112 or circuitry coupled to thedisplay 112. - The above discussion is meant to be illustrative of the principles and various embodiments of the present invention. Numerous variations and modifications will become apparent to those skilled in the art once the above disclosure is fully appreciated. It is intended that the following claims be interpreted to embrace all such variations and modifications.
Claims (20)
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/910,567 US20190158721A1 (en) | 2017-11-17 | 2018-03-02 | Multi-camera synchronization through receiver hub back channel |
PCT/US2018/061792 WO2019099991A1 (en) | 2017-11-17 | 2018-11-19 | Multi-camera synchronization through receiver hub back channel |
US17/145,740 US11470233B2 (en) | 2017-11-17 | 2021-01-11 | Multi-camera synchronization through receiver hub back channel |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201762587500P | 2017-11-17 | 2017-11-17 | |
US15/910,567 US20190158721A1 (en) | 2017-11-17 | 2018-03-02 | Multi-camera synchronization through receiver hub back channel |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/145,740 Continuation US11470233B2 (en) | 2017-11-17 | 2021-01-11 | Multi-camera synchronization through receiver hub back channel |
Publications (1)
Publication Number | Publication Date |
---|---|
US20190158721A1 true US20190158721A1 (en) | 2019-05-23 |
Family
ID=66533490
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/910,567 Abandoned US20190158721A1 (en) | 2017-11-17 | 2018-03-02 | Multi-camera synchronization through receiver hub back channel |
US17/145,740 Active 2038-05-31 US11470233B2 (en) | 2017-11-17 | 2021-01-11 | Multi-camera synchronization through receiver hub back channel |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/145,740 Active 2038-05-31 US11470233B2 (en) | 2017-11-17 | 2021-01-11 | Multi-camera synchronization through receiver hub back channel |
Country Status (2)
Country | Link |
---|---|
US (2) | US20190158721A1 (en) |
WO (1) | WO2019099991A1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20190222790A1 (en) * | 2018-01-12 | 2019-07-18 | Thine Electronics, Inc. | Video signal receiver and video signal transmission and reception system |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN117156071B (en) * | 2023-10-31 | 2024-02-23 | 深圳市国鼎科技有限公司 | Auxiliary driving image synchronization system and method |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6061089A (en) * | 1995-03-24 | 2000-05-09 | Ppt Vision, Inc. | High-speed digital video serial link |
US20050154736A1 (en) * | 2003-12-22 | 2005-07-14 | Meikleham Daniel M. | System and method for providing services across a network |
US20130287122A1 (en) * | 2011-03-09 | 2013-10-31 | Hitachi Consumer Electronics Co., Ltd. | Video transmission device, video transmission method, video receiving device, and video receiving method |
US20170048424A1 (en) * | 2012-12-05 | 2017-02-16 | Magna Electronics Inc. | Vehicle vision system utilizing camera synchronization |
Family Cites Families (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2001282714A (en) * | 2000-03-30 | 2001-10-12 | Olympus Optical Co Ltd | Multi-camera data transfer system and data transfer system |
-
2018
- 2018-03-02 US US15/910,567 patent/US20190158721A1/en not_active Abandoned
- 2018-11-19 WO PCT/US2018/061792 patent/WO2019099991A1/en active Application Filing
-
2021
- 2021-01-11 US US17/145,740 patent/US11470233B2/en active Active
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6061089A (en) * | 1995-03-24 | 2000-05-09 | Ppt Vision, Inc. | High-speed digital video serial link |
US20050154736A1 (en) * | 2003-12-22 | 2005-07-14 | Meikleham Daniel M. | System and method for providing services across a network |
US20130287122A1 (en) * | 2011-03-09 | 2013-10-31 | Hitachi Consumer Electronics Co., Ltd. | Video transmission device, video transmission method, video receiving device, and video receiving method |
US20170048424A1 (en) * | 2012-12-05 | 2017-02-16 | Magna Electronics Inc. | Vehicle vision system utilizing camera synchronization |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20190222790A1 (en) * | 2018-01-12 | 2019-07-18 | Thine Electronics, Inc. | Video signal receiver and video signal transmission and reception system |
US11825224B2 (en) * | 2018-01-12 | 2023-11-21 | Thine Electronics, Inc. | Video signal receiver and video signal transmission and reception system |
Also Published As
Publication number | Publication date |
---|---|
US11470233B2 (en) | 2022-10-11 |
WO2019099991A1 (en) | 2019-05-23 |
US20210136276A1 (en) | 2021-05-06 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11470233B2 (en) | Multi-camera synchronization through receiver hub back channel | |
US7986371B2 (en) | Video signal processing device, method of the same and vehicle-mounted camera system | |
US20140368609A1 (en) | Systems And Methods For Generating A Panoramic Image | |
JP5784382B2 (en) | Electronic endoscope device | |
US20130229404A1 (en) | Image data transmission system and electronic device | |
CN104918003A (en) | Multiple camera synchronization system | |
CN105430368A (en) | Two-viewpoint stereo image synthesizing method and system | |
US8314843B2 (en) | Method and apparatus for information reproduction | |
CN208956216U (en) | Based on the long large car viewing system away from transmission of low-light (level) camera synchronous acquisition | |
US10887512B2 (en) | Camera system | |
CN103814577A (en) | Sending device, sending method, receiving device, receiving method and sending/receiving system | |
CN102186035B (en) | Method for displaying screen display information | |
US20180054605A1 (en) | Method and apparatus for transmitting images captured by first and second image sensors | |
JP2012186574A (en) | On-vehicle camera device | |
JP2009147831A (en) | Image transmission device and wireless image reception device | |
JP2004312063A (en) | Imaging apparatus, image display system, and image displaying method | |
KR101469446B1 (en) | Synchronization composite system and method for the plural image | |
KR101809727B1 (en) | Surround View Monitoring System and Image Signal Processing Method thereof | |
JP2004208162A (en) | System and device for transmitting camera video | |
KR20080096160A (en) | Device and method for transferring data | |
CN108924455B (en) | Overlay transmission system based on FPD-Link bus | |
WO2015182083A1 (en) | Video signal transmission system and display device | |
JP4938699B2 (en) | Imaging apparatus and imaging system | |
Sauerwald | FPD-Link III–doing more with less | |
JPWO2016060268A1 (en) | Image display system for vehicles |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: TEXAS INSTRUMENTS INCORPORATED, TEXAS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:CEEKALA, VIJAYA;LIU, XIN;ZIAZADEH, RAMSIN;SIGNING DATES FROM 20180301 TO 20180302;REEL/FRAME:045091/0780 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STCV | Information on status: appeal procedure |
Free format text: APPEAL BRIEF (OR SUPPLEMENTAL BRIEF) ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: TC RETURN OF APPEAL |
|
STCV | Information on status: appeal procedure |
Free format text: APPEAL READY FOR REVIEW |
|
STCV | Information on status: appeal procedure |
Free format text: ON APPEAL -- AWAITING DECISION BY THE BOARD OF APPEALS |
|
STCV | Information on status: appeal procedure |
Free format text: BOARD OF APPEALS DECISION RENDERED |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- AFTER EXAMINER'S ANSWER OR BOARD OF APPEALS DECISION |