WO2017215331A1 - 全景摄像机及其拍摄方法 - Google Patents

全景摄像机及其拍摄方法 Download PDF

Info

Publication number
WO2017215331A1
WO2017215331A1 PCT/CN2017/080032 CN2017080032W WO2017215331A1 WO 2017215331 A1 WO2017215331 A1 WO 2017215331A1 CN 2017080032 W CN2017080032 W CN 2017080032W WO 2017215331 A1 WO2017215331 A1 WO 2017215331A1
Authority
WO
WIPO (PCT)
Prior art keywords
sub
image
captured
cameras
interpolation
Prior art date
Application number
PCT/CN2017/080032
Other languages
English (en)
French (fr)
Inventor
张铧铧
王晓江
柯常志
Original Assignee
杭州海康威视数字技术股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 杭州海康威视数字技术股份有限公司 filed Critical 杭州海康威视数字技术股份有限公司
Priority to US16/310,075 priority Critical patent/US10554882B2/en
Priority to EP17812443.4A priority patent/EP3471392B1/en
Publication of WO2017215331A1 publication Critical patent/WO2017215331A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/698—Control of cameras or camera modules for achieving an enlarged field of view, e.g. panoramic image capture
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/80—Geometric correction
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/10—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources

Definitions

  • the present application relates to the field of video surveillance, and in particular, to a panoramic camera and a method for photographing the same.
  • multiple cameras are generally used for shooting.
  • multiple network cameras are installed and arranged from various directions to form a 360-degree, dead-end monitoring.
  • each network camera independently transmits the video data to the monitoring center and displays it independently on the large screen of the monitoring center, which is not only costly, but also the independent picture scene brings the viewer's vision. The experience is also greatly affected.
  • the real-time video collected by the front-end multiple network cameras is first sent to the central server, and the back-end synthesis processing is performed to form a complete panoramic image.
  • the back-end synthesis processing is performed to form a complete panoramic image.
  • each network camera captures video images and transmits them over the network, different delays are generated. Due to different delays, there may be a mismatch in image stitching, resulting in problems such as misalignment or overlap of moving objects in the composite image.
  • the purpose of the application is to provide a panoramic camera and a shooting method thereof, which can make the processing efficiency better, the processing quality is better, and the system design is more compact.
  • a panoramic camera comprising:
  • N sub-cameras N is greater than 2; each sub-camera shoots at different angles; each sub-camera The captured image is partially coincident with the captured image captured by at least one other sub-camera in the panoramic camera;
  • a synchronization control module for controlling synchronous shooting of N sub-cameras
  • the splicing correction module is configured to buffer the continuous m rows of pixel values in the captured image in a sliding window manner, and complete the correction of the pixel values according to the preset correction table and the buffered m rows of pixel values, and sequentially output Corrected each row of pixel values, wherein m is greater than a preset maximum number of distortion lines +2, and m is less than the total number of lines of the captured image;
  • the splicing fusion module is configured to splicing the captured images synchronously captured by the sub-cameras under the control of the synchronization control module into a spliced image according to the overlapping portions of the captured images of the sub-cameras;
  • a transmission module configured to send, by using a communication network, a mosaic image output by the splicing fusion module.
  • the captured image of the N sub-cameras is an image in a Bayer format
  • the panoramic camera further includes: an interpolation module configured to perform RGB interpolation on the captured images captured by the N sub-cameras, and RGB The interpolated captured image is sent to the stitching correction module.
  • the captured image of the N sub-cameras is an image in a Bayer format
  • the panoramic camera further includes a G interpolation module and an RB interpolation module, where
  • the G interpolation module is configured to perform G interpolation on the image of the Bayer format captured by the N sub-cameras, and send the captured image after the G interpolation to the splicing correction module;
  • the splicing correction module is configured to correct the captured image after the G interpolation, and send the corrected captured image to the RB interpolation module;
  • the RB interpolation module is configured to perform R interpolation and B interpolation on the corrected captured image.
  • the panoramic camera further includes:
  • the bilinear interpolation module is configured to perform a bilinear interpolation on the captured image corrected by the splicing correction module according to the position of the target point on the original captured image, reference four pixels around the target point, and generate a second pixel value, and The bilinearly interpolated captured image is sent to the splicing fusion module.
  • the panoramic camera further includes:
  • the image rotation module is configured to perform 90° rotation on the captured image synchronously shot by the sub camera, and send the rotated captured image to the stitching correction module.
  • the splicing correction module is further configured to receive an image that is interpolated by the G interpolation module, and send the corrected captured image to the RB interpolation module.
  • each sub-camera includes a photo sensor and a lens, each sub-camera being at the same angle, and each sub-camera having the same angle as the ground plane.
  • a method for photographing a panoramic camera comprising N sub-cameras, N being greater than 2; each sub-camera shooting at a different angle; each sub-camera taking a captured image with the panoramic camera The captured images taken by at least one other sub-camera are partially coincident; the method includes the following steps:
  • Performing splicing correction on the captured images synchronously captured by each sub-camera includes the following sub-steps: buffering successive m rows of pixel values in the captured image in a sliding window manner, and according to a preset correction table and the cached m-line pixel values Completing the correction of the pixel value, and sequentially outputting the corrected pixel value of each line, wherein m is greater than a preset maximum number of distortion lines +2, and m is smaller than the total number of lines of the captured image;
  • the captured images synchronously captured by the respective sub-cameras are spliced into one spliced image
  • the stitched image is transmitted over a communication network.
  • the images captured by the N sub-cameras are in the Bayer format, and the method further includes the step of performing RGB interpolation on the captured images taken by the respective sub-cameras.
  • the images captured by the N sub-cameras are in the Bayer format
  • R-interpolation and B-interpolation are performed on the captured image after G interpolation and stitching correction.
  • the method further includes the following steps:
  • the captured image taken synchronously by the N sub-cameras is rotated by 90°.
  • the method before the step of transmitting the stitched image through the communication network, the method further includes the step of performing image compression on the stitched image.
  • the splicing correction module performs local buffering (line buffering) on the captured image, which effectively solves the problems in the prior art that the entire image frame needs to be read and corrected one by one, and the process of splicing correction can be made. There is no need to consume memory bandwidth, which greatly reduces the consumption of system resources.
  • the storage control of the splicing correction module can be reduced by half, and the resource consumption is significantly reduced.
  • the entire system is simple in design and is advantageous for design miniaturization.
  • the line buffer based stitching correction processing combined with the Bayer interpolation algorithm can ensure the stitching correction processing well while capturing the image quality.
  • a panoramic camera comprising: N sub-cameras, a housing, a processor, a memory, a circuit board, and a power supply circuit; wherein N is greater than 2, each of the sub-cameras The angles are different; the captured image captured by each of the sub-cameras partially coincides with the captured image captured by at least one other sub-camera in the panoramic camera; the circuit board is disposed inside the space enclosed by the housing, and the processor and the memory are disposed at Circuit board; power circuit for panoramic viewing The various circuits or devices of the camera are powered; the memory is for storing executable program code; the processor runs a program corresponding to the executable program code by reading executable program code stored in the memory for performing the above shooting method.
  • an executable program code for being executed to perform the above-described photographing method.
  • a storage medium for storing executable program code for being executed to perform the above-described photographing method.
  • FIG. 1 is a schematic structural diagram of a panoramic camera according to an embodiment of the present application.
  • FIG. 2 is a schematic structural diagram of a panoramic camera according to another embodiment of the present application.
  • FIG. 3 is a schematic structural diagram of a panoramic camera according to another embodiment of the present application.
  • FIG. 4 is a schematic diagram of a network structure of a panoramic camera in an embodiment of the present application.
  • FIG. 5 is a schematic diagram of a panoramic camera in an embodiment of the present application.
  • FIG. 6 is a schematic diagram of a stitching effect of a captured image of the panoramic camera of the present application.
  • FIG. 7 is a schematic diagram of main modules of a panoramic camera in an embodiment of the present application.
  • FIG. 8 is a schematic diagram of a data format of a Bayer format image of the present application.
  • FIG. 9 is a schematic diagram of an interpolation and correction process in an embodiment of the present application.
  • FIG. 10 is a schematic diagram of an interpolation process in an embodiment of the present application.
  • FIG. 11 is a schematic diagram of a storage architecture of an image before and after splicing correction in an embodiment of the present application
  • FIG. 13 is a schematic diagram of an interpolation and correction process in an embodiment of the present application.
  • FIG. 14 is a schematic diagram of image data after G interpolation of Bayer data in an embodiment
  • 15 is a schematic diagram of an image vertical stitching effect in an embodiment of the present application.
  • 16 is a schematic diagram of image rotation in an embodiment of the present application.
  • 17 is a schematic diagram of main modules of a panoramic camera in an embodiment of the present application.
  • FIG. 18 is a schematic flow chart of a method for photographing a panoramic camera according to an embodiment of the present application.
  • FIG. 19 is a schematic flow chart of a method for photographing a panoramic camera according to an embodiment of the present application.
  • 11 first sub-camera; 12: second sub-camera; 13: third sub-camera; 14: fourth sub-camera; 21: synchronous control module; 31: G interpolation module; 32: splicing correction Module; 33: RB interpolation module; 34: splicing fusion module; 35: transmission module; 322: interpolation module.
  • a first embodiment of the present application provides a panoramic camera, including:
  • each sub-camera captures a different angle, and the captured image captured by each sub-camera and at least one other sub-camera in the panoramic camera
  • the captured images have a partial overlap.
  • the synchronization control module 21 is configured to control the first sub-camera 11 , the second sub-camera 12 , the third sub-camera 13 and the fourth sub-camera 14 to simultaneously capture.
  • the splicing correction module 32 is configured to cache successive m rows of pixel values in the captured image in a sliding window manner, and complete the correction of the pixel values according to the preset correction table and the buffered m rows of pixel values, and sequentially output the corrections. Each subsequent row of pixel values, where m is greater than a preset maximum number of distortion lines +2, and m is less than the total number of lines of the captured image.
  • the splicing fusion module 34 is configured to synchronize the respective sub-cameras under the control of the synchronization control module 21 according to the overlapping portions of the captured images of the first sub-camera 11, the second sub-camera 12, the third sub-camera 13, and the fourth sub-camera 14.
  • the captured images are stitched together into a stitched image.
  • the transmission module 35 is configured to send the mosaic image output by the splicing fusion module 32 through the communication network.
  • each sub-camera includes a photosensitive sensor and a lens, each sub-camera is separated by the same angle, and each sub-camera has the same angle with the ground plane.
  • the network structure of the above panoramic camera can be seen in FIG. 4, wherein the panoramic camera 41 includes a first sub-camera 11, a second sub-camera 12, a third sub-camera 13, and a fourth sub-camera 14, and through the network and the network hard disk recorder 42, And the user client 43 is connected.
  • the angles of the sub-cameras of the panoramic camera are set differently, so that when the sub-cameras are simultaneously photographed, the obtained captured images are not completely identical.
  • the angles of view of the adjacent two sub-cameras partially overlap, so that the captured image captured by each sub-camera partially coincides with the captured image captured by at least one other sub-camera in the panoramic camera.
  • the size and direction of the overlap region may be determined by the installation parameters of each sub-camera.
  • each sub-camera of the panoramic camera can also be illustrated by another embodiment of the present application.
  • the panoramic camera has eight cameras, and each sub-camera includes one. Photosensitive sensors and a lens, each sub-camera is at the same angle, and each sub-camera has the same angle as the ground plane.
  • the angles of the sub-cameras can also be set differently, and the angle between each sub-camera and the ground plane can also be set differently.
  • the photosensitive sensor in each sub-camera of the panoramic camera may be a Complementary Metal Oxide Semiconductor (CMOS) sensor, a Charge-coupled Device (CCD) sensor, or the like.
  • CMOS Complementary Metal Oxide Semiconductor
  • CCD Charge-coupled Device
  • the synchronization control module 21 of the panoramic camera simultaneously transmits control signals to the first sub-camera 11, the second sub-camera 12, the third sub-camera 13, and the fourth sub-camera 14, so that synchronous imaging control of each sub-camera can be realized.
  • the synchronization control module 21 transmits a synchronous imaging electrical signal to the photosensitive sensors of the respective sub-cameras, thereby controlling the simultaneous shooting of the respective sub-cameras.
  • the synchronization control module 21 may also transmit a synchronous imaging electrical signal to the electronic shutters of the respective sub-cameras, thereby controlling the synchronization of the respective sub-cameras. It can be understood by those skilled in the art that the camera is photographed under the control of an external electrical signal, which is not described herein.
  • storing the continuous m-line pixel values in the captured image by sliding window means that the first to m rows are cached first, and then the new row is sequentially added to the cache. At the same time, the first cached row is discarded according to the principle of first in first out (FIFO), so that there are always only m rows in the cache until all the rows of an image have been added to the cache.
  • FIFO first in first out
  • first cache the first to m rows add the m+1th row and discard the first row, add m+2 rows and discard the second row, add the m+3th row and discard the third row, ..., This is done until the last line of the image is added to the cache (in this case the last m lines of the image in the cache).
  • the synchronization working module 21 controls the first sub-camera 11, the second sub-camera 12, the third sub-camera 13 and the fourth sub-camera 14 to perform synchronous shooting, and the obtained image is sent to the splicing correction module 32, and the splicing correction module 32 pairs
  • the image is captured for splicing correction, specifically, the continuous m-line pixel values in the captured image are buffered in a sliding window manner, and the pixel values are corrected according to the preset correction table and the buffered m-line pixel values, and The corrected pixel values of each row are sequentially output, wherein m is greater than a preset maximum number of distortion lines +2, and m is smaller than the total number of lines of the captured image, and the stitched corrected captured image is sent to the stitching fusion module 34.
  • the block 34 splices the captured images synchronously captured by the respective sub-cameras into one spliced image according to the overlapping portions of the captured images captured by the respective sub-cameras, and transmits them to the transmission module 35. Finally, the transmission module 35 transmits the stitched image over the communication network.
  • the splicing correction module performs local buffering (line buffering) on the captured image, which effectively solves the problems in the prior art that the entire image frame needs to be read and corrected one by one, and the process of splicing correction can be made. There is no need to consume memory bandwidth, which greatly reduces the consumption of system resources.
  • the design of the entire system facilitates miniaturization of the design.
  • a second embodiment of the present application provides a panoramic camera, which includes the components of the first embodiment.
  • the panoramic camera of this embodiment further includes:
  • the G interpolation module 31 is configured to perform G interpolation on the captured images captured by the first sub-camera 11 , the second sub-camera 12 , the third sub-camera 13 , and the fourth sub-camera 14 ;
  • the RB interpolation module 33 is configured to perform R interpolation and B interpolation on the captured image after the G interpolation.
  • connection relationship of each module in the panoramic camera is as follows:
  • the first sub-camera 11, the second sub-camera 12, the third sub-camera 13 and the fourth sub-camera 14 output the captured image to the G interpolation module 31;
  • the G interpolation module 31 outputs the processed captured image to the stitching correction module 32;
  • the splicing correction module 32 outputs the processed captured image to the RB interpolation module 33;
  • the RB interpolation module 33 outputs the processed image to the splicing fusion module 34;
  • the splicing fusion module 34 outputs the processed image to the transmission module 35;
  • V
  • H is the horizontal direction parameter and V is the vertical direction parameter
  • edgeDirection 0, where edgeDirection is the edge direction, then:
  • edgeDirection 1
  • R represents a red component
  • G represents a green component
  • B represents a blue component
  • i, j represents mutually perpendicular coordinate values
  • [i, j] is a green dot
  • [(i-1) , the position of j] and the position of [(i+1), j] are red dots
  • the position of [i, (j-1)] and the position of [i, (j+1)] are blue dots.
  • R ij G ij + ⁇ [R i(j-1) -G i(j-1) ]+[R i(j+1) -G i(j+1) ] ⁇ /2
  • B i(j+1) G i(j+1) + ⁇ [B (i-1)j -G i-1)j ]+[B (i-1)(j+2) -G i- 1)(j+2) ]+[B (i+1)j -G (i+1)j ]+[B (i+1)(j+2) -G (i+1)(j+2 ) ] ⁇ /4
  • R i(j+1) G (i+1)j + ⁇ [R i(j-1) -G i(j-1) ]+[R i(j+1) -G i(j+1 ) ]+[R (i+2)(j-1) -G (i+2)(j-1) ]+[R (i+2)(j+1) -G (i+2)(j +1) ] ⁇ /4
  • R (i+1)(j+1) G (i+1)(j+1) + ⁇ [R i(j+1) -G i(j+1) ]+[R (i+2) (j+1) -G (i+2)(j+1) ] ⁇ /2
  • R (i+1)(j+1) G (i+1)(j+1) + ⁇ [R (i+1)j -G (i+1)j ]+[R (i+1) (j+2) -G (i+1)(j+2) ] ⁇ /2
  • R represents a red component
  • G represents a green component
  • B represents a blue component
  • i, j represents mutually perpendicular coordinate values
  • [i, j] is a green dot
  • [(i-1) , the position of j] and the position of [(i+1), j] are red dots
  • the position of [i, (j-1)] and the position of [i, (j+1)] are blue dots.
  • image horizontal stitching or image vertical stitching can be arbitrarily selected.
  • the horizontal splicing and vertical splicing are described in more detail below.
  • the implementation system of the panoramic camera can refer to FIG. 7 , where the system includes a first sub camera 11 , a second sub camera 12 , a third sub camera 13 , and a fourth sub camera 14 .
  • the data acquisition module 72 is used for capturing the captured images captured by each sub-camera and converted into image data in parallel Bayer format.
  • the format of the parallel data can be seen in FIG. 8 .
  • the input 4 pairs of LVDS (Low Voltage Differential Signaling) serial data are collected and received, and converted into parallel Bayer format image data.
  • the parallel data format can be seen in FIG. 8.
  • splicing correction is performed on the parallel Bayer image, wherein the splicing correction input is a Bayer image, and a splicing correction table (referred to as a “correction table”) generated offline and placed in an external memory, and the output is a spliced corrected RGB image. See Figure 9.
  • the Bayer image needs to be interpolated first, and the Bayer image is generated by interpolation to generate an RGB image, wherein the interpolation process is as shown in FIG. Specifically, referring to Fig. 10, the four points "33", “35", “53” and “55” are red dots, "24", “26”, “44”, “46” are blue dots, and other dots are green.
  • the Bayer image (1001) which can be decomposed into G image (1011) and RB image (1012), generate a complete G image by G interpolation (1021), and RB interpolation to generate a complete R image (1022) ) and the B image (1023), finally, the R, G, and B primary colors are available at each pixel point, and the Bayer graph to RGB interpolation is realized.
  • Alpha1 and Beta1 represent the defined intermediate variables.
  • V
  • edgeDirection 0; (horizontal direction)
  • G35 (G34+G36)/2+(2*R35–R33–R37)/4;
  • G35 (G25+G45)/2+(2*R35–R15–R55)/4;
  • G35 (G34+G36+G25+G45)/4+(2*R35–R33–R37+2*R35–R15–R55)/8;
  • the interpolation scheme of G45 is the same as G35, and the images finally output by the above interpolation are all R, G, and B three-color images.
  • the interpolated RGB image is input into the splicing correction module 32, and the image is locally stored.
  • the storage space size of the splicing correction module 32 is related to the distortion of the image itself.
  • the current lens adopts a 5 mm-8 mm lens.
  • the maximum distortion number of the image is 69 lines, and the storage architecture can be seen in FIG. Specifically, the splicing correction module 32 shown in FIG.
  • each address stores one pixel RBG value
  • the designed maximum storage line number is the maximum distortion line number +3, in this embodiment
  • the splicing correction module 32 is implemented by the line_cache module, and in other embodiments of the present application, the splicing correction module 32 may also be implemented by other means.
  • the input captured image is stored in the splicing correction module 32 in a row structure, and when the splicing correction module 32 completes the area storage, the logic circuit is simultaneously externally Reading out the correction table in the memory, and parsing the information of the correction table, wherein the correction table description information is that the pixel moves from the original coordinate point to the target address coordinate, and the image is corrected, that is, the pixel value of each coordinate point of the original image is moved to the corresponding The coordinate position.
  • the coordinates of the stitching correction module 32 parsed in the correction table are: (X1, Y1); (X3, Y2); (X4, Y4); (X6, Y4); (X7, Y5); (X8, Y7)..
  • the corrected coordinates are (X1, Y1); (X2, Y1); (X3, Y1); (X4, Y1); (X5, Y1); (X6 , Y1)...; where the pixel values corresponding to the output coordinates are input coordinates (X1, Y1); (X3, Y2); (X4, Y4); (X6, Y4); (X7, Y5); (X8, Y7)
  • the RGB values corresponding to the coordinates are unchanged.
  • the scheme of locally buffering the splicing correction module 32 solves the conventional scheme that requires full frame buffering and then one-by-one reading correction of the image, so that the correction algorithm itself does not need to consume memory bandwidth.
  • the pixels of the desired position are taken out from the splicing correction module 32 one by one and are sequentially output.
  • the output image data is re-entered into the bilinear interpolation module (as shown in FIG. 7), and the captured image corrected by the splicing correction module 32 is obtained from the original captured image according to the target point.
  • the position on the captured image is referenced to four pixels around the target point to generate a second pixel value, the bilinear interpolation is completed, and the bilinearly interpolated captured image is sent to the splicing fusion module 34.
  • Px (1.1, 1.3) that is, the pixel value of the position coordinate at (1.1, 1.3) is:
  • the splicing and merging module 34 fuses the images corrected by the four sub-cameras to form a complete large image, as shown in FIG. 6.
  • the splicing correction module 32 since the splicing correction module 32 stores RGB data, the requirement for the internal buffer is high. Therefore, as shown in FIG. 13, the image interpolation image of the Bayer format is first interpolated by the G interpolation module 31, and then Send to the splicing correction module 32 for splicing correction, and then send The RB interpolation module 33 is sent to RB interpolation to output a captured image in RGB format.
  • the image data obtained by G-interpolating the Bayer data by the G interpolation module 31 can be seen in FIG. 14, that is, the image format of the output R/B, G, specifically, the G component of each image is complete, R/ The B component pixel adds another G component to the original R/B component. That is, the R/B, G format image is stored in the splicing correction module 32.
  • the calculation amount of the whole design is unchanged from the original scheme, except that the R and B component calculations are placed after the splicing correction module 32, that is, the values in the splicing correction module 32 are read, and the R and B that need to be filled are performed.
  • the interpolation that is, the correction algorithm and the interpolation are completed at the same time.
  • the multi-port splicing correction module 32 storage architecture is adopted, and the read bandwidth of the splicing correction module 32 is increased to satisfy the entire design. need.
  • the advantage of the image horizontal splicing scheme is that the sensor is horizontally spliced, and the complete image is wider after splicing.
  • the camera can adopt a lens with a relatively long focal length, and the solution adopts a 5-8 mm lens, so that the camera can be seen farther.
  • the front-end processing system used in this design adopts the no-video frame buffer mode as shown above, and the splicing correction module 32 is effectively controlled, making the overall scheme more effective.
  • the image vertical stitching scheme will be described. If the resolution of each sub-camera is 1920*1080, and the image is first transposed 90 degrees, and then spliced, the image resolution after the splicing is completed is: X*Y, wherein the theoretical resolution is less than 4480*1920.
  • the resolution is related to the lens and the specific installation method.
  • 4096*1800 is taken as an example, and the effect after splicing can be seen in FIG. 15 , and the main module structure can be seen in FIG. 17 .
  • the entire system scheme and the horizontal splicing scheme differ in that an image rotation module 73 connected to the external storage module 75 is added.
  • the image rotation module 73 is configured to perform 90° rotation on the captured image synchronously captured by the sub-camera, and send the rotated captured image to the splicing correction module 32. See Figure 16 for the rotation implementation flow.
  • step 1611 the input memory (1602) is prepared in block mode from the image (1601) input, and the memory storage format is 32*5.
  • the data format of the image in the memory is as shown in the figure.
  • the 2D block is expanded into one dimension for storage.
  • the block used in the scheme is 4*4. 8*8 or 4*8 and other block sizes can be used in the project.
  • step 1612 the memory (1602) is read out in one-dimensional order. Then, the read data is divided into 4 FIFOs (first in first out queue) cache, and the sorting method is indicated by arrows in the figure, and 4 FIFOs are cyclically placed in units of 4 pixels, and the format of the final data in the FIFO is shown in the figure. 16.
  • 4 FIFOs first in first out queue
  • step 1613 the formed image (1603) is cyclically read out from the four FIFOs, the format is shown in Figure 16, and the rotated image resolution is 8*20.
  • the rotated image is read out and organized into a frame image data stream, and then sent to the splicing correction module, wherein the splicing correction module and the bilinear interpolation and splicing fusion process are the same as the above one, and will not be described again.
  • the advantages of the image vertical stitching scheme vertical stitching of the sensor, the complete image is higher after stitching, the camera can adopt a relatively short focal length lens, and the scheme adopts a 4-5mm lens, so that the upper and lower field of view of the camera is larger.
  • the design also adopts the no video frame buffer mode, and the splicing correction module 32 is effectively controlled, so that the whole scheme is more effective.
  • each module mentioned in each device implementation manner of the present application is a logic module.
  • a logic module may be a physical module, a part of a physical module, or multiple physical entities.
  • the combined implementation of modules, the physical implementation of these logic modules themselves is not the most important, the combination of the functions implemented by these logic modules is the key to solving the technical problems raised by the present application.
  • the foregoing device implementations of the present application do not introduce modules that are not closely related to solving the technical problems raised by the present application, which does not indicate that the above device implementation does not exist in other embodiments. Module.
  • the localization buffer (line buffer) of the captured image by the splicing correction module effectively solves the existing In the technology, a problem arises in the process of performing full frame buffering and then reading and correcting the captured images one by one, so that the process of the stitching correction itself does not need to consume memory bandwidth, thereby greatly reducing the consumption of system resources.
  • the storage control of the splicing correction module can be reduced by half, and the resource consumption is significantly reduced.
  • the overall system design is simple, which is conducive to design miniaturization.
  • the line buffer based stitching correction processing combined with the Bayer interpolation algorithm can ensure the quality of the captured image while achieving the stitching correction processing.
  • a third embodiment of the present application provides a panoramic camera, which is different from the first embodiment and the second embodiment in that:
  • the captured images of the first sub-camera 11, the second sub-camera 12, the third sub-camera 13, and the fourth sub-camera 14 are images in the Bayer format, and the panoramic camera further includes an interpolation module 322 for photographing each sub-camera. Take an image for RGB interpolation.
  • the captured image of the Bayer format is subjected to RGB interpolation by the interpolation module 322, and then sent to the splicing correction module 32, so that the format of the captured image is RGB format.
  • the input of the stitching correction module 32 is an image of the Bayer format, and a stitching correction table stored in the external memory, and the output is a stitched corrected RGB image.
  • FIG. 10 See Figure 10 for the interpolation process.
  • the left side of FIG. 10 is a captured image in the Bayer format, and a complete G image is generated by G interpolation, and a complete R, B image is generated by RB interpolation, and finally, the R, G, and B primary colors of each pixel point have Thus, interpolation from the captured image in the baye format to the RGB format is realized.
  • the direction judgment is first performed, and the edge direction edgeDirection is calculated.
  • the calculation process of the edge direction is exemplified below:
  • V
  • edgeDirection 0; (horizontal direction)
  • G35 (G34+G36)/2+(2*R35–R33–R37)/4;
  • G35 (G25+G45)/2+(2*R35–R15–R55)/4;
  • G35 (G34+G36+G25+G45)/4+(2*R35–R33–R37+2*R35–R15–R55)/8;
  • the interpolation scheme of G45 is the same as G35.
  • the final output images are all R, G, and B images.
  • the interpolated RGB image input splicing correction module 32 performs local storage on the image, wherein the storage space size of the splicing correction module 32 is related to the distortion of the image itself.
  • the lens adopts a 5 mm-8 mm lens.
  • the input captured image is 1920*1080
  • the maximum distortion line number of the captured image is 72 lines.
  • the memory architecture can be seen in FIG.
  • the input captured image is stored in the splicing correction module 32 in a row structure.
  • the logic circuit simultaneously reads out the correction table from the external memory and parses the information of the correction table.
  • the description information of the correction table is that the pixel moves from the original coordinate point to the target address coordinate.
  • the correction of the captured image is completed, that is, the pixel value of each coordinate point of the original captured image is completed and moved to the corresponding coordinate position.
  • the pixel coordinates currently to be output are found out from the splicing correction module 32 according to the information of the correction table, and are read out from the splicing correction module 32, and sent to the subsequent module for subsequent processing, as shown in the figure, the correction table is shown.
  • the coordinates in the intermediate parse correction module 32 are: (X1, Y1); (X3, Y2); (X4, Y4); (X6, Y4); (X7, Y5); (X8, Y7)....
  • the value sequence output forms the first line of complete data, the corrected coordinates are (X1, Y1); (X2, Y1); (X3, Y1); (X4, Y1); (X5, Y1); (X6, Y1) )...; where the pixel value corresponding to the output coordinate is the input coordinate (X1, Y1); (X3, Y2); (X4, Y4); (X6, Y4); (X7, Y5); (X8, Y7)
  • the RGB values corresponding to the coordinates are unchanged.
  • the advantage of the scheme of using the above-mentioned splicing correction module 32 for local buffering is that the conventional scheme of performing full frame buffering and then reading correction one by one is solved, so that the correction algorithm itself does not need to consume memory bandwidth.
  • the output captured image of the RGB format is input to the bilinear interpolation module 74 to generate a final pixel value with reference to the surrounding four pixels in accordance with the position of the target point on the original image.
  • Px (1.1, 1.3) that is, the pixel value of the position coordinate at (1.1, 1.3) is as follows:
  • R component values of P0; P1; P2; P3 are: 100, 120, 140, 160;
  • the splicing and merging module 34 is used to fuse the captured images of the splicing correction described above to form a complete large captured image. See FIG. 6.
  • the splicing correction module performs local buffering (line buffering) on the captured image, which effectively solves the problems in the prior art that the entire image frame needs to be read and corrected one by one, and the process of splicing correction can be made. There is no need to consume memory bandwidth, which greatly reduces the consumption of system resources.
  • the line buffer based stitching correction processing combined with the Bayer interpolation algorithm can ensure the quality of the captured image while achieving the stitching correction processing.
  • the overall system design is simple, which is conducive to design miniaturization.
  • the panoramic camera includes four sub-cameras, each of which is photographed at different angles; the captured image taken by each sub-camera and the captured image taken by at least one other sub-camera in the panoramic camera There are some overlaps.
  • the shooting method of this embodiment includes the following steps:
  • Step 101 Control 4 sub cameras to shoot synchronously.
  • Step 102 Perform splicing correction on the captured images synchronously captured by each sub-camera, including the following sub-steps: buffering consecutive m-line pixel values in the captured image in a sliding window manner, and according to a preset correction table and cached m rows The pixel value is completed, and the pixel value is corrected, and the corrected pixel value of each line is sequentially output, wherein m is greater than a preset maximum number of distortion lines +2, and m is smaller than the total number of lines of the captured image.
  • Step 103 splicing the captured images synchronously captured by the sub-cameras into a spliced image according to the overlapping portions of the captured images captured by the respective sub-cameras;
  • Step 104 Send the stitched image through a communication network.
  • the above embodiments are device embodiments corresponding to the embodiments of the present method, and the present embodiment can be implemented in cooperation with the above embodiments.
  • the related technical details mentioned in the above embodiments are still effective in the present embodiment, and are not described herein again in order to reduce repetition. Accordingly, the related technical details mentioned in the present embodiment can also be applied to the above embodiments.
  • the local cache (line buffer) of the captured image effectively solves the problems in the prior art that the entire image frame needs to be read and corrected one by one, and the process of the stitching correction itself does not need to be consumed.
  • Memory bandwidth which greatly reduces the consumption of system resources.
  • the overall system design is simple, which is conducive to design miniaturization.
  • Embodiment Shooting method of panoramic camera 2
  • the shooting method of the panoramic camera of the present embodiment is basically the same as that of the above embodiment, and will not be described again. The differences between them are as follows:
  • the captured image captured by each sub-camera is in the Bayer format, and the photographing method of the present embodiment further includes the following steps:
  • RGB interpolation is performed on the captured images taken by the respective sub-cameras.
  • the above embodiments are device embodiments corresponding to the embodiments of the present method, and the present embodiment can be implemented in cooperation with the above embodiments.
  • the related technical details mentioned in the above embodiments are still effective in the present embodiment, and are not described herein again in order to reduce repetition. Accordingly, the related technical details mentioned in the present embodiment can also be applied to the above embodiments.
  • the local cache (line buffer) of the captured image effectively solves the problems in the prior art that the entire image frame needs to be read and corrected one by one, and the process of the stitching correction itself does not need to be consumed.
  • Memory bandwidth which greatly reduces the consumption of system resources.
  • the line buffer based stitching correction processing combined with the Bayer interpolation algorithm can ensure the quality of the captured image while achieving the stitching correction processing.
  • the overall system design is simple, which is conducive to design miniaturization.
  • Embodiment Shooting method of panoramic camera 3
  • the photographing method of the panoramic camera of the present embodiment is basically the same as the above embodiment, and the differences between them are as follows:
  • the image captured by each sub-camera is in the Bayer format, and the photographing method of the present embodiment further includes the following steps:
  • R-interpolation and B-interpolation are performed on the captured image after G interpolation.
  • the photographing method of the panoramic camera of the present embodiment includes the following steps:
  • Step 201 Control 4 sub cameras to shoot synchronously.
  • Step 202 Perform G interpolation on the captured image of each sub-camera
  • the G interpolation is implemented by the following technical solutions:
  • V
  • H is the horizontal direction parameter and V is the vertical direction parameter
  • edgeDirection 0, where edgeDirection is the edge direction, then:
  • edgeDirection 1
  • R represents a red component
  • G represents a green component
  • B represents a blue component
  • i, j represents mutually perpendicular coordinate values
  • [i, j] is a green dot
  • [(i-1) , the position of j] and the position of [(i+1), j] are red dots
  • the position of [i, (j-1)] and the position of [i, (j+1)] are blue dots.
  • Step 203 Perform splicing correction on the captured image synchronously captured by each sub-camera, including: buffering consecutive m-row pixel values in the captured image in a sliding window manner, and according to a preset correction table and a buffered m-line pixel value, The correction of the pixel values is completed, and the corrected pixel values of each line are sequentially output, wherein m is greater than a preset maximum number of distortion lines +2, and m is smaller than the total number of lines of the captured image.
  • the buffer output Since the number of corrected image lines must be smaller than before the correction, in the image output process, the buffer output has more than one line updated, but the output has only one line;
  • the above sub-step 203 is repeatedly performed until all the lines of one image are corrected, where m is a preset maximum number of distortion lines +3.
  • step 203 In order to better understand the stitching correction process in step 203, further explanation and examples are given below.
  • the correction table stores the mapping relationship information of the pre-corrected images, that is, for the pixels in the corrected image, the coordinate information of the pixels of the pre-correction image corresponding thereto can be found in the correction table.
  • the calibration header is included in the calibration table.
  • the correction header describes which lines of the corrected image come from which lines in the pre-correction image, namely the start line and the end line. According to the correction header, if the line between the start line and the end line of the pre-correction image corresponding to one line (referred to as the current line) of the corrected image to be output is already in the buffer, the current output from the buffer can be output. Row.
  • the image before correction is 1920 lines
  • the number of lines to be output after correction is 1800 lines.
  • the selected splicing correction module 32 ie, the buffer in rows
  • the data sequence of the image before correction is sequentially written to the splicing correction module 32.
  • the first line of the corrected image is corrected according to the information in the correction header (ie, The output of the current line needs to use lines 1-16 of the image before correction, and at this time, the data already buffered in the splicing correction module 32 has satisfied the requirement, and then the corrected image is output according to the mapping information in the correction table. Line 1 of the line. Then, the 17th row of the pre-corrected image is input to the splicing correction module 32. At this time, the splicing correction module 32 has data 1-17 lines, and according to the correction header information, it can be known that the 2-17 lines of the image before correction can be output.
  • the pre-correction image is input into the splicing correction module 32 row by row (if the splicing correction module 32 is full, the earliest input line is discarded according to the principle of the first-in first-out FIFO), and the current splicing is judged according to the correction header information.
  • the data in the correction module 32 can output the current 1 line of the corrected image, if possible, outputs the current 1 line of the corrected image, otherwise continues to input the data of the 1 line of the pre-correction image to the splicing correction module 32.
  • the entire process is as described above. After the 1837th line of the pre-correction image is input to the splicing correction module 32, the data in the splicing correction module 32 satisfies the condition of the 1800th line output of the corrected image, and the 1800th line image of the corrected image. The output, for the subsequent 1837--1920 lines of image data in the pre-correction image, is no longer written into the stitching correction module 32. When the new frame starts, the entire line_cache is cleared to the next cycle.
  • the rows and columns of the image referred to in the various embodiments of the present application may be user-defined.
  • 1920 may be defined as a row, 1080 as a column, or 1080 as a row.
  • 1920 is defined as a column.
  • the maximum number of distortion lines of the image is obtained by counting the test image, that is, the standard test image is taken by each sub-camera, and the captured image is compared with a standard test image, that is, The position change of each pixel in the image before and after the shooting can be obtained, and then the number of rows spanned by the maximum distortion can be obtained, that is, the maximum number of distortion lines of the image.
  • the maximum number of distortion lines of the image can be set based on empirical values.
  • step 204 R-interpolation and B-interpolation are performed on the captured image after the G interpolation.
  • R interpolation and B interpolation are implemented by the following technical solutions:
  • R ij G ij + ⁇ [R i(j-1) -G i(j-1) ]+[R i(j+1) -G i(j+1) ] ⁇ /2
  • B i(j+1) G i(j+1) + ⁇ [B (i-1)j -G i-1)j ]+[B (i-1)(j+2) -G i- 1)(j+2) ]+[B (i+1)j -G (i+1)j ]+[B (i+1)(j+2) -G (i+1)(j+2 ) ] ⁇ /4
  • R i(j+1) G (i+1)j + ⁇ [R i(j-1) -G i(j-1) ]+[R i(j+1) -G i(j+1 ) ]+[R (i+2)(j-1) -G (i+2)(j-1) ]+[R (i+2)(j+1) -G (i+2)(j +1) ] ⁇ /4
  • R (i+1)(j+1) G (i+1)(j+1) + ⁇ [R i(j+1) -G i(j+1) ]+[R (i+2) (j+1) -G (i+2)(j+1) ] ⁇ /2
  • R (i+1)(j+1) G (i+1)(j+1) + ⁇ [R (i+1)j -G (i+1)j ]+[R (i+1) (j+2) -G (i+1)(j+2) ] ⁇ /2
  • R represents a red component
  • G represents a green component
  • B represents a blue component
  • i, j represents mutually perpendicular coordinate values
  • [i, j] is a green dot
  • [(i-1) , the position of j] and the position of [(i+1), j] are red dots
  • the position of [i, (j-1)] and the position of [i, (j+1)] are blue dots.
  • Step 205 splicing the captured images synchronously captured by the sub-cameras into one stitched image according to the overlapping portions of the captured images captured by the respective sub-cameras;
  • Step 206 Perform image compression on the stitched image.
  • Step 207 Send the stitched image through the communication network.
  • each pixel is composed of three colors of RGB, and one RGB image can be divided into three sub-images of three colors of R, G, and B.
  • G interpolation is to interpolate the green part of the Bayer image to get a complete green sub-image (G image).
  • the R interpolation is to interpolate the red portion of the Bayer image to obtain a complete red sub-image (R image).
  • B interpolation is to interpolate the blue part of the Bayer image to get a complete blue sub-image (B image).
  • the step of performing the stitching correction is performed after the step of performing the G interpolation, and the step of performing the stitching correction is performed before the step of performing the R interpolation and the B interpolation.
  • G interpolation and RB interpolation may be performed first to obtain a complete RGB image, and then splicing correction is performed.
  • the spliced image when the transmission bandwidth is sufficient, may be directly transmitted without image compression.
  • For the corrected captured image refer to the target according to the position of the target point on the original captured image.
  • Four pixels around the punctuation generate a second pixel value, and complete bilinear interpolation.
  • the step of performing G interpolation on the captured image of each sub-camera is performed before the step of splicing correction of the captured image; and, the image after the interpolation by G is performed.
  • the steps of interpolation and B interpolation are performed after the step of splicing correction of the captured image.
  • the above embodiments are device embodiments corresponding to the embodiments of the present method, and the present embodiment can be implemented in cooperation with the above embodiments.
  • the related technical details mentioned in the above embodiments are still effective in the present embodiment, and are not described herein again in order to reduce repetition. Accordingly, the related technical details mentioned in the present embodiment can also be applied to the above embodiments.
  • the local cache (line buffer) of the captured image effectively solves the problems in the prior art that the entire image frame needs to be read and corrected one by one, and the process of the stitching correction itself does not need to be consumed.
  • Memory bandwidth which greatly reduces the consumption of system resources.
  • the storage control of the splicing correction module 32 (splicing correction module) can be reduced by half, and the resource consumption is remarkably reduced.
  • the overall system design is simple, which is conducive to design miniaturization.
  • the line buffer based stitching correction processing combined with the Bayer interpolation algorithm can ensure the quality of the captured image while achieving the stitching correction processing.
  • the storage control of the splicing correction module can be reduced by half, and the resource consumption is significantly reduced.
  • the line buffer based stitching correction processing combined with the Bayer interpolation algorithm can ensure the quality of the captured image while achieving the stitching correction processing.
  • the embodiment of the present application further provides a panoramic camera, including: N sub-cameras, a casing, a processor, a memory, a circuit board, and a power supply circuit; wherein, N is greater than 2, and each of the sub-cameras captures an angle differently;
  • the captured image captured by the sub-camera partially coincides with the captured image captured by at least one other sub-camera in the panoramic camera;
  • the circuit board is disposed inside the space enclosed by the housing, and the processor and the memory are disposed on the circuit board; a power circuit for powering various circuits or devices of the panoramic camera; a memory for storing executable program code; the processor running a program corresponding to the executable program code by reading executable program code stored in the memory to use
  • the shooting method is performed.
  • Methods include:
  • Performing splicing correction on the captured images synchronously captured by the respective sub-cameras including the following sub-steps: buffering successive m rows of pixel values in the captured image in a sliding window manner, and according to a preset correction table and the cache m rows of pixel values, complete the correction of the pixel values, and sequentially output the corrected pixel values of each row, wherein m is greater than a preset maximum number of distortion lines +2, and m is smaller than the captured image Total number of lines;
  • the captured images synchronously captured by the respective sub-cameras are spliced into one spliced image
  • the stitched image is transmitted over a communication network.
  • the embodiment of the present application further provides an executable program code, where the executable program code is used to execute the shooting method, and the method includes:
  • Performing splicing correction on the captured images synchronously captured by the respective sub-cameras including the following sub-steps: buffering successive m rows of pixel values in the captured image in a sliding window manner, and according to a preset correction table and the cache m rows of pixel values, complete the correction of the pixel values, and sequentially output the corrected pixel values of each row, wherein m is greater than a preset maximum number of distortion lines +2, and m is smaller than the captured image Total number of lines;
  • the captured images synchronously captured by the respective sub-cameras are spliced into one spliced image
  • the stitched image is transmitted over a communication network.
  • the embodiment of the present application further provides a storage medium for storing executable program code, where the executable program code is used to execute the shooting method, and the method includes:
  • Performing splicing correction on the captured images synchronously captured by the respective sub-cameras including the following sub-steps: buffering successive m rows of pixel values in the captured image in a sliding window manner, and according to a preset correction table and the cache m rows of pixel values, complete the correction of the pixel values, and sequentially output the corrected pixel values of each row, wherein m is greater than a preset maximum number of distortion lines +2, and m is smaller than the captured image Total number of lines;
  • the stitched image is transmitted over a communication network.
  • the instruction code can be stored in any type of computer-accessible memory (eg, permanent or modifiable, volatile or non-volatile, solid state Or non-solid, fixed or replaceable media, etc.).
  • the memory may be, for example, Programmable Array Logic ("PAL"), Random Access Memory (RAM), or Programmable Read Only Memory (PROM). "), Read-Only Memory (“ROM”), Electrically Erasable Programmable ROM (“EEPROM”), Disk, CD, Digital Versatile Disc , referred to as "DVD”) and so on.
  • PAL Programmable Array Logic
  • RAM Random Access Memory
  • PROM Programmable Read Only Memory
  • ROM Read-Only Memory
  • EEPROM Electrically Erasable Programmable ROM
  • Disk CD
  • DVD Digital Versatile Disc
  • a logical unit may be a physical unit, or may be a part of a physical unit, or may be implemented by a combination of multiple physical units.
  • the physical implementation of these logical units is not the most important, and the combination of the functions implemented by these logical units is the key to solving the technical problems raised by the present application.
  • the foregoing device implementation manners of the present application do not introduce units that are not closely related to solving the technical problems raised by the present application, which does not indicate that the above device implementation manner does not exist in other embodiments. unit.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Studio Devices (AREA)
  • Image Processing (AREA)

Abstract

本申请公开了一种全景摄像机及其拍摄方法，本申请的全景摄像机包括：N个子摄像头；同步控制模块；拼接校正模块，用于以滑动窗口的方式缓存拍摄图像中连续的m行像素值，并根据预先设定的校正表和所述缓存的m行像素值，完成对所述像素值的校正，并依次输出校正后的每一行像素值，其中，m大于预先设定的图像最大畸变行数+2，并且m小于所述拍摄图像的总行数；拼接融合模块，用于根据所述各子摄像头的拍摄图像的重合部分，将所述各个子摄像头在所述同步控制模块控制下同步拍摄的拍摄图像拼接成一个拼接图像；以及传输模块。本申请的全景摄像机处理效率更好，处理质量更好，并且更有利于系统设计小型化。

Description

全景摄像机及其拍摄方法
本申请要求于2016年6月14日提交中国专利局、申请号为201610416973.2、发明名称为“全景摄像机及其拍摄方法”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及视频监控领域，特别涉及一种全景摄像机及其拍摄方法。
背景技术
为了实现重点监控区域的全覆盖，一般采用多个相机进行拍摄。举例来说，对于一条城市主干道的十字路口的视频监控，从各个方向安装布置多个网络摄像机，从而形成360度、无死角的监控。
然而，上述技术方案中，每一台网络摄像机独立地将视频数据传输到监控中心，并在监控中心的大屏上独立地显示，这样做不但成本高，而且独立画面场景带给观看者的视觉体验也受到较大影响。
为了解决上述问题，通常先将前端多台网络摄像机采集的实时视频分别发送给中心端服务器，进行后端合成处理后，形成完整的全景图像。但是，由于每一个网络摄像机采集视频图像，并通过网络传输后，会导致分别产生不同的时延。由于不同的时延，在图像拼接时有可能存在不匹配的现象，导致合成图像中的运动物体错位或重叠等问题。
另一方面，虽然也有通过将前端多台网络摄像机或网络摄像机的多个摄像头采集的实时视频在前端直接进行拼接，获得全景图像的解决方案，但存在对系统资源的消耗大、拼接效果不好的问题。
发明内容
本申请的目的就是提供一种全景摄像机及其拍摄方法，能够使得处理效率更好，处理质量更好，并且更有利于系统设计小型化。
在本申请的第一个方面，提供了一种全景摄像机，包括：
N个子摄像头，N大于2；每个子摄像头拍摄的角度不同；每个子摄像头 拍摄的拍摄图像与该全景摄像机中至少一个其他子摄像头拍摄的拍摄图像有部分重合；
同步控制模块，用于控制N个子摄像头同步拍摄；
拼接校正模块，用于以滑动窗口的方式缓存拍摄图像中连续的m行像素值，并根据预先设定的校正表和该缓存的m行像素值，完成对该像素值的校正，并依次输出校正后的每一行像素值，其中，m大于预先设定的图像最大畸变行数+2，并且m小于该拍摄图像的总行数；
拼接融合模块，用于根据各子摄像头的拍摄图像的重合部分，将各个子摄像头在该同步控制模块控制下同步拍摄的拍摄图像拼接成一个拼接图像；
传输模块，用于通过通信网络发送该拼接融合模块输出的拼接图像。
在本申请的另一实施方式中，N个子摄像头的拍摄图像为Bayer格式的图像，并且，该全景摄像机还包括：插值模块，用于对N个子摄像头拍摄的拍摄图像进行RGB插值，并将RGB插值后的拍摄图像发送给所述拼接校正模块。
在本申请的另一实施方式中，N个子摄像头的拍摄图像为Bayer格式的图像，并且，该全景摄像机还包括包括G插值模块和RB插值模块，其中，
G插值模块用于对N个子摄像头拍摄的Bayer格式的图像进行G插值，并将经G插值后的拍摄图像发送给拼接校正模块；
拼接校正模块用于对经G插值后的拍摄图像进行校正，并将校正后的拍摄图像发送给RB插值模块；
RB插值模块用于对校正后的拍摄图像进行R插值和B插值。
在本申请的另一实施方式中，全景摄像机还包含：
双线性插值模块，用于对由拼接校正模块校正后的拍摄图像，根据目标点在原拍摄图像上的位置，参考目标点周边四个像素，生成第二像素值，完成双线性插值，并将经双线性插值后的拍摄图像发送给拼接融合模块。
在本申请的另一实施方式中，全景摄像机还包含：
图像旋转模块，用于对子摄像头同步拍摄的拍摄图像进行90°旋转，并将经旋转的拍摄图像发送给拼接校正模块。
在本申请的另一实施方式中，该拼接校正模块还用于接收该G插值模块进行插值后的图像，并将校正后的拍摄图像发送给该RB插值模块。
在本申请的另一实施方式中，每个子摄像头包括一个感光传感器和一个透镜，每个子摄像头间隔相同的角度，且每个子摄像头与地平面夹角相同。
在本申请的第二个方面，提供了一种全景摄像机的拍摄方法，该全景摄像机包括N个子摄像头，N大于2；每个子摄像头拍摄的角度不同；每个子摄像头拍摄的拍摄图像与该全景摄像机中至少一个其他子摄像头拍摄的拍摄图像有部分重合；该方法包括以下步骤：
控制N个子摄像头同步拍摄；
对各子摄像头同步拍摄的拍摄图像进行拼接校正，包含以下子步骤：以滑动窗口的方式缓存拍摄图像中连续的m行像素值，并根据预先设定的校正表和该缓存的m行像素值，完成对该像素值的校正，并依次输出校正后的每一行像素值，其中，m大于预先设定的图像最大畸变行数+2，并且m小于该拍摄图像的总行数；
根据各子摄像头拍摄的拍摄图像的重合部分，将各子摄像头同步拍摄的拍摄图像拼接成一个拼接图像；
通过通信网络发送该拼接图像。
在本申请的另一实施方式中，N个子摄像头拍摄的图像为Bayer格式，并且，该方法还包含以下步骤：对各个子摄像头拍摄的拍摄图像进行RGB插值。
在本申请的另一实施方式中，N个子摄像头拍摄的图像为Bayer格式，并且，
在对各子摄像头同步拍摄的拍摄图像进行拼接校正的步骤之前，还包含以下步骤：
对各子摄像头的拍摄图像进行G插值；
在对各子摄像头同步拍摄的拍摄图像进行拼接校正的步骤之后，还包含以下步骤：
对经G插值和拼接校正之后的拍摄图像进行R插值和B插值。在本申请的另一实施方式中，在所述控制所述N个子摄像头同步拍摄的步骤之后，还包含以下步骤：
对所述N个子摄像头同步拍摄的拍摄图像进行90°旋转。
在本申请的另一实施方式中，通过通信网络发送该拼接图像的步骤之前，还包括以下步骤：对该拼接图像进行图像压缩。
本申请实施方式与现有技术相比，至少具有以下区别和效果：
通过拼接校正模块对拍摄图像进行局部缓存(行缓存)，有效解决了现有技术中需要对拍摄图像进行整帧缓存再逐一读取并校正的过程中产生的问题，能够使拼接校正的过程本身不需要消耗内存带宽，从而极大地减小了对系统资源的消耗。
进一步的，通过对拍摄图像先进行G插值处理，然后进行拼接校正，最后进行RB插值处理，能够使拼接校正模块的存储控制减少一半，明显降低了对资源的消耗。
进一步的，整个系统设计简单，有利于设计小型化。
进一步的，基于行缓存的拼接校正处理与Bayer插值算法结合，能够保证拍摄图像质量的同时很好地实现拼接校正处理。
在本申请的第三个方面，提供了一种全景摄像机，包括：N个子摄像头、壳体、处理器、存储器、电路板和电源电路；其中，N大于2，每个所述子摄像头拍摄的角度不同；每个所述子摄像头拍摄的拍摄图像与所述全景摄像机中至少一个其他子摄像头拍摄的拍摄图像有部分重合；电路板安置在壳体围成的空间内部，处理器和存储器设置在电路板上；电源电路，用于为全景摄 像机的各个电路或器件供电；存储器用于存储可执行程序代码；处理器通过读取存储器中存储的可执行程序代码来运行与可执行程序代码对应的程序，以用于执行上述拍摄方法。
在本申请的第四个方面，提供了一种可执行程序代码，所述可执行程序代码用于被运行以执行上述拍摄方法。
在本申请的第五个方面，提供了一种存储介质，所述存储介质用于存储可执行程序代码，所述可执行程序代码用于被运行以执行上述拍摄方法。
应理解，在本申请范围内中，本申请的上述各技术特征和在下文(如实施例)中具体描述的各技术特征之间都可以互相组合，从而构成新的或优选的技术方案。限于篇幅，在此不再一一累述。
附图说明
为了更清楚地说明本申请实施例和现有技术的技术方案，下面对实施例和现有技术中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1是本申请一个实施方式中一种全景摄像机的结构示意图；
图2是本申请另一个实施方式中一种全景摄像机的结构示意图；
图3是本申请另一个实施方式中一种全景摄像机的结构示意图；
图4是本申请一个实施方式中全景摄像机的网络结构示意图；
图5是本申请一个实施方式中全景摄像机的示意图；
图6是本申请的全景摄像机的拍摄图像拼接效果示意图；
图7是本申请一个实施方式中一种全景摄像机的主要模块示意图；
图8是本申请的Bayer格式图像的数据格式示意图；
图9是本申请一个实施方式中插值与校正过程示意图；
图10是本申请一个实施方式中插值过程示意图；
图11是本申请一个实施方式中图像在拼接校正前后的存储架构示意图；
图12是本申请一个实施方式中双线性插值示意图；
图13是本申请一个实施方式中插值与校正过程示意图；
图14是一个实施方式中对Bayer数据进行G插值后的图像数据示意图；
图15是本申请一个实施方式中图像垂直拼接效果示意图；
图16是本申请一个实施方式中图像旋转示意图；
图17是本申请一个实施方式中全景摄像机的主要模块示意图；
图18是本申请一个实施方式中一种全景摄像机的拍摄方法的流程示意图；
图19是本申请一个实施方式中一种全景摄像机的拍摄方法的流程示意图。
在各附图中，11：第一子摄像头；12：第二子摄像头；13：第三子摄像头；14：第四子摄像头；21：同步控制模块；31：G插值模块；32：拼接校正模块；33：RB插值模块；34：拼接融合模块；35：传输模块；322：插值模块。
具体实施方式
为使本申请的目的、技术方案、及优点更加清楚明白，以下参照附图并举实施例，对本申请进一步详细说明。显然，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
在以下的叙述中，为了使读者更好地理解本申请而提出了许多技术细节。但是，本领域的普通技术人员可以理解，即使没有这些技术细节和基于以下各实施方式的种种变化和修改，也可以实现本申请各权利要求所要求保护的技术方案。
为使本申请的目的、技术方案和优点更加清楚，下面将结合附图对本申请的实施方式作进一步地详细描述。
参见图1，本申请第一实施方式提供一种全景摄像机，包括：
第一子摄像头11、第二子摄像头12、第三子摄像头13和第四子摄像头14，其中，每个子摄像头拍摄的角度不同，每个子摄像头拍摄的拍摄图像与全景摄像机中至少一个其他子摄像头拍摄的拍摄图像有部分重合。
同步控制模块21，用于控制第一子摄像头11、第二子摄像头12、第三子摄像头13和第四子摄像头14同步拍摄。
拼接校正模块32，用于以滑动窗口的方式缓存拍摄图像中连续的m行像素值，并根据预先设定的校正表和缓存的m行像素值，完成对像素值的校正，并依次输出校正后的每一行像素值，其中，m大于预先设定的图像最大畸变行数+2，并且m小于所述拍摄图像的总行数。
拼接融合模块34，用于根据第一子摄像头11、第二子摄像头12、第三子摄像头13和第四子摄像头14的拍摄图像的重合部分，将各个子摄像头在同步控制模块21控制下同步拍摄的拍摄图像拼接成一个拼接图像。以及
传输模块35，用于通过通信网络发送拼接融合模块32输出的拼接图像。
在上述实施方式或其他实施方式中，每个子摄像头包括一个感光传感器和一个透镜，每个子摄像头间隔相同的角度，且每个子摄像头与地平面夹角相同。
上述全景摄像机的网络结构可参见图4，其中全景摄像机41包括第一子摄像头11、第二子摄像头12、第三子摄像头13和第四子摄像头14，并且，通过网络与网络硬盘录像机42，以及用户客户端43连接。
下面进一步说明本实施方式中的全景摄像机的技术细节。
本实施方式中，全景摄像机的各子摄像头拍摄的角度设置得各不相同，使各子摄像头在同时拍摄时，其得到的拍摄图像不完全相同。同时，相邻的两个子摄像头的视场角又存在部分重叠，使每个子摄像头拍摄的拍摄图像与全景摄像机中至少一个其他子摄像头拍摄的拍摄图像有部分重合。在本申请的实施方式中，该重叠区域的大小和方向可由每个子摄像头的安装参数确定。
全景摄像机的各子摄像头的设置方式也可以通过本申请的另一个实施方式予以说明，如图5所示，全景摄像机有8个摄像头，每个子摄像头包括一 个感光传感器和一个透镜，每个子摄像头间隔相同的角度，且每个子摄像头与地平面夹角相同。但本领域技术人员可以理解，在本申请的其他实施方式中，各子摄像头间隔的角度也可以设置得不同，每个子摄像头与地平面的夹角也可以设置得不同。
进一步地说，全景摄像机的各子摄像头中的感光传感器可以是互补金属氧化物半导体(Complementary Metal Oxide Semiconductor，CMOS)传感器、电荷耦合元件(Charge-coupled Device，CCD)传感器，等等。
全景摄像机的同步控制模块21同时向第一子摄像头11、第二子摄像头12、第三子摄像头13和第四子摄像头14发送控制信号，可实现对各子摄像头的同步拍摄控制。举例来说，在本申请的一个实施方式中，同步控制模块21向各子摄像头的感光传感器发送同步拍摄电信号，以此控制各子摄像头同步拍摄。在另一实施方式中，同步控制模块21也可以向各子摄像头的电子快门发送同步拍摄电信号，从而控制各子摄像头同步拍摄。本领域技术人员可以理解，摄像头在外部电信号的控制下进行拍摄是现有技术，在此不做赘述。
需要指出的是，在全景摄像机的拼接校正模块32中，“以滑动窗口的方式缓存拍摄图像中连续的m行像素值”是指先缓存第1至m行，再不断地依次将新行加入缓存，同时根据先入先出(FIFO)的原则丢弃最先缓存的行，使缓存中始终只有m行，直到一个图像的所有行都先后加入过缓存。具体地说，先缓存第1至m行，加入第m+1行同时丢弃第1行，加入m+2行同时丢弃第2行，加入第m+3行同时丢弃第3行，……，如此直到将图像的最后一行加入到缓存(此时缓存中是该图像的最后m行)。
下面简要说明本实施方式的全景摄像机的工作过程：
首先，同步工作模块21控制第一子摄像头11、第二子摄像头12、第三子摄像头13和第四子摄像头14进行同步拍摄，得到的图像被发送给拼接校正模块32，拼接校正模块32对拍摄图像进行拼接校正，具体地说，以滑动窗口的方式缓存拍摄图像中连续的m行像素值，并根据预先设定的校正表和缓存的m行像素值，完成对像素值的校正，并依次输出校正后的每一行像素值，其中，m大于预先设定的图像最大畸变行数+2，并且m小于拍摄图像的总行数，并将拼接校正后的拍摄图像发送给拼接融合模块34。此后，拼接融合模 块34根据各子摄像头拍摄的拍摄图像的重合部分，将各子摄像头同步拍摄的拍摄图像拼接成一个拼接图像，并发送给传输模块35。最后，传输模块35通过通信网络发送拼接图像。
本实施方式的优点如下：
通过拼接校正模块对拍摄图像进行局部缓存(行缓存)，有效解决了现有技术中需要对拍摄图像进行整帧缓存再逐一读取并校正的过程中产生的问题，能够使拼接校正的过程本身不需要消耗内存带宽，从而极大地减小了对系统资源的消耗。整个系统的设计有利于设计小型化。
参见图3，本申请的第二实施方式提供一种全景摄像机，包含第一实施方式的各组件，此外，本实施例的全景摄像机还包括：
G插值模块31，用于对第一子摄像头11、第二子摄像头12、第三子摄像头13和第四子摄像头14拍摄的拍摄图像进行G插值；
RB插值模块33，用于对经G插值之后的拍摄图像进行R插值和B插值。
如图3所示，全景摄像机中各模块的连接关系如下：
第一子摄像头11、第二子摄像头12、第三子摄像头13和第四子摄像头14输出拍摄图像到G插值模块31；
G插值模块31输出处理后的拍摄图像到拼接校正模块32；
拼接校正模块32输出处理后的拍摄图像到RB插值模块33；
RB插值模块33输出处理后的图像到拼接融合模块34；
拼接融合模块34输出处理后的图像到传输模块35；
下面具体说明本实施方式中G插值模块31对第一子摄像头11、第二子摄像头12、第三子摄像头13和第四子摄像头14拍摄的拍摄图像进行G插值的方案：
H＝|Gij-Gi(j+2)|+|2*Bi(j+1)-Bi(j-1)-Bi(j+3)|
V＝|G(i-1)(j+1)-G(i+1)(j+1)|+|2*Bi(j+1)-B(i-2)(j+1)-B(i+2)(j+1)|
其中，H是水平方向参数，V是垂直方向参数
如果H<V，则edgeDirection＝0，其中，edgeDirection是边缘方向，则：
【公式1】
Figure PCTCN2017080032-appb-000001
如果H>V，则edgeDirection＝1，则：
【公式2】
Figure PCTCN2017080032-appb-000002
如果H＝V，则edgeDirection＝2，则：
【公式3】
Figure PCTCN2017080032-appb-000003
其中，R代表红色分量，G代表绿色分量，B代表蓝色分量，并且，i，j代表相互垂直的坐标值，[i，j]的位置上是绿色点，并且，[(i-1)，j]的位置和[(i+1)，j]的位置上是红色点，[i，(j-1)]的位置和[i，(j+1)]的位子上是蓝色点。
下面具体说明本实施方式中RB插值模块33对经G插值之后的图像进行R插值和B插值的方案：
【公式4】
Rij＝Gij+{[Ri(j-1)-Gi(j-1)]+[Ri(j+1)-Gi(j+1)]}/2
【公式5】
Bij＝Gij+{[R(i-1)j-Gi-1)j]+[R(i+1)j-G(i+1)j]}/2
【公式6】
Bi(j+1)＝Gi(j+1)+{[B(i-1)j-Gi-1)j]+[B(i-1)(j+2)-Gi-1)(j+2)]+[B(i+1)j-G(i+1)j]+[B(i+1)(j+2)-G(i+1)(j+2)]}/4
【公式7】
Ri(j+1)＝G(i+1)j+{[Ri(j-1)-Gi(j-1)]+[Ri(j+1)-Gi(j+1)]+[R(i+2)(j-1)-G(i+2)(j-1)]+[R(i+2)(j+1)-G(i+2)(j+1)]}/4
【公式8】
R(i+1)(j+1)＝G(i+1)(j+1)+{[Ri(j+1)-Gi(j+1)]+[R(i+2)(j+1)-G(i+2)(j+1)]}/2
【公式9】
R(i+1)(j+1)＝G(i+1)(j+1)+{[R(i+1)j-G(i+1)j]+[R(i+1)(j+2)-G(i+1)(j+2)]}/2
其中，R代表红色分量，G代表绿色分量，B代表蓝色分量，并且，i，j代表相互垂直的坐标值，[i，j]的位置上是绿色点，并且，[(i-1)，j]的位置和[(i+1)，j]的位置上是红色点，[i，(j-1)]的位置和[i，(j+1)]的位置上是蓝色点。
此外，可以理解，对于Bayer图像进行R插值、G插值、B插值的技术方案有多种，本申请不限于使用上述技术方案对Bayer图像进行R插值、G插值和B插值。
在本实施方式中，可以任意选择图像水平拼接或图像垂直拼接。下面分别对水平拼接与垂直拼接进行更详细的说明。
参见图6-图14，其中，该全景摄像的实现系统可以参见图7，该系统中包含全景摄像机的第一子摄像头11、第二子摄像头12、第三子摄像头13和第四子摄像头14，图像采集模块72，拼接校正模块32，双线性插值模块74，拼接融合模块34，同步控制模块21，校正表存储模块78，以及主控芯片79。
其中，数据采集模块72用于各子摄像头拍摄的拍摄图像进行采集，并转化为并行的Bayer格式的图像数据，并行数据的格式可参见图8。
拼接前和拼接后的效果可以参见图6，其中左侧第一张图像对应第一子摄像头11的拍摄图像，第二张图像对应第二子摄像头12的拍摄图像，以此类推。
在图像水平拼接方案中，假设图7中的各子摄像头的分辨率为1920*1080，按水平方案拼接四个子摄像头的图像，则拼接完成后图像分辨率为：X*Y，其中，理论分辨率小于7680*1080，最终分辨率和镜头及结构安装相关。在本文中，为了便于说明，以7168*1024进行说明。
具体地说，首先对输入的4对LVDS(Low Voltage Differential Signaling，低压差分信号)串行数据进行采集接收，并转化为并行的Bayer格式图像数据，并行数据格式可参见图8。接着，对并行的Bayer图像进行拼接校正，其中，拼接校正输入为Bayer图像，以及离线生成并放置在外部存储器中的拼接校正表(简称“校正表”)，输出为拼接校正后的RGB图像，可参见图9。
进一步说，首先需要对Bayer图像进行插值，插值后Bayer图像就生成RGB图像，其中插值过程如图10所示。具体地说，参见图10，“33”“35”“53”“55”这4个点为红色点，“24”“26”“44”“46”为蓝色点，其它的点为绿色点，并且，最左侧为Bayer图像(1001)，可以分解为G图像(1011)和RB图像(1012)，通过G插值生成完整的G图像(1021)，RB插值生成完整的R图像(1022)和B图像(1023)，最终每个像素点上R，G，B三原色都具有了，实现了Bayer图到RGB的插值。
其中R，B插值过程：
Figure PCTCN2017080032-appb-000004
其中，Alpha1、Beta1表示定义的中间变量。
对于G的插值首先进行方向判断，计算边缘方向edgeDirection，计算边缘方向的方法：
以插值G35分量为例，计算边缘方向：
H＝|G34–G36|+|2*B35–B33–B37|；
V＝|G25–G45|+|2*B35–B15–B55|；
如果H<V；则edgeDirection＝＝0；(水平方向)
如果H>V；则edgeDirection＝＝1；(垂直方向)
如果H＝V；则edgeDirection＝＝2；(不区分方向)
如果edgeDirection＝＝0
G35＝(G34+G36)/2+(2*R35–R33–R37)/4；
如果edgeDirection＝＝1
G35＝(G25+G45)/2+(2*R35–R15–R55)/4；
否则
edgeDirection＝＝2
G35＝(G34+G36+G25+G45)/4+(2*R35–R33–R37+2*R35–R15–R55)/8；
同理G45的插值方案同G35，通过上述插值最终输出的图像都是R、G、B三色图像。
此后，将完成插值的RGB图像输入拼接校正模块32，对图像进行局部存储，其中拼接校正模块32的存储空间大小和图像本身畸变大小有关，在本实施方式中，目前镜头采用5mm-8mm镜头，例如，对输入图像为1920*1080，图像最大畸变行数为69行，存储架构可参见图11。具体地说，图11中所示拼接校正模块32内部按行结构对图像按行存储，每个地址存储一个像素RBG值，设计的最大储存行数为最大畸变行数+3，在本实施方式中是72行，则由计算得到存储空间：1920*72*36bits/8＝622KB，其中，1920为每行像素点的个数，R像素点、G像素点、B像素点各占用12bits，和为36bits，8bits为1Byte。
需要指出的是，在本实施方式中，拼接校正模块32是通过line_cache模块实现的，而在本申请的其他实施方式中，也可以通过其他方式实现拼接校正模块32。
下面说明一下本实施方式中的拼接校正模块32(line_cache模块)的工作机制：输入的拍摄图像按行结构存入拼接校正模块32中，当拼接校正模块32完成区域存储后，逻辑电路同时从外部存储器中读出校正表，并解析校正表的信息，其中校正表描述信息为像素从原始坐标点移动到目标地址坐标，完成图像的校正，即完成原图像每一个坐标点的像素值搬移到对应的坐标位置。
根据校正表的信息从拼接校正模块32中找出当前要输出的像素坐标，并从拼接校正模块32中读出，送往后续模块进行后续处理，如图11中所示， 校正表中解析的拼接校正模块32中坐标为：(X1，Y1)；(X3，Y2)；(X4，Y4)；(X6，Y4)；(X7，Y5)；(X8，Y7)....的值顺序输出形成第一行完整数据，校正后的坐标为(X1，Y1)；(X2，Y1)；(X3，Y1)；(X4，Y1)；(X5，Y1)；(X6，Y1)...；其中输出坐标对应的像素值为输入坐标(X1，Y1)；(X3，Y2)；(X4，Y4)；(X6，Y4)；(X7，Y5)；(X8，Y7)坐标对应的RGB值不变。
采用上述拼接校正模块32局部缓存的方案解决了需要对图像进行整帧缓存再逐一读取校正的传统方案，使得校正算法本身不需要消耗内存带宽。
如图12所示，逐个解析校正表从拼接校正模块32中取出所需位置的像素并按顺序输出。
此后，输出的图像数据再输入到双线性插值模块(如图7所示)，对由拼接校正模块32校正后的拍摄图像，根据目标点在原拍摄图像(即由上述子摄像头同步拍摄得到的拍摄图像)上的位置，参考目标点周边四个像素，生成第二像素值，完成双线性插值，并将经双线性插值后的拍摄图像发送给拼接融合模块34。
具体地说，如图12所示，Px(1.1，1.3)即位置坐标在(1.1，1.3)的像素值为：
假设P0；P1；P2；P3的R分量值分别为：100，120，140，160；设定这四个像素点的权重，具体的，离Px越近的点，权重可以越大；利用双线性插值算法：
Px的R分量＝(1-0.1)*(1-0.3)P0+0.1*(1-0.3)P1+(1-0.1)*0.3P2+0.1*0.3P3＝114；
同理得到G，B分量的值，最终输出经过双线性插值后的图像。
接着，由拼接融合模块34对4个子摄像头拼接校正后的图像进行融合，形成完整的大图，如图6所示。
需要说明的是，由于拼接校正模块32中存储为RGB数据，对内部缓存的要求较高，因此，可以如图13所示，先由G插值模块31对Bayer格式的拍摄图像进行B插值，然后发送给拼接校正模块32进行拼接校正，然后再发 送给RB插值模块33进行RB插值，从而输出RGB格式的拍摄图像。具体地说，由G插值模块31对Bayer数据进行G插值后得到的图像数据可参见图14，即输出R/B，G的图像格式，具体地说，每个像数G分量完整，R/B分量像素点在原来R/B分量上再增加G分量。即R/B，G格式图像存储在拼接校正模块32中，通过上述改进，具有明显的优点，拼接校正模块32存储空间为比原来减少了二分之一。
具体地说，原方案中，拼接校正模块32存储空间＝1920*72*36bits/8＝622KB，经过本方案的改进，拼接校正模块32存储空间＝1920*72*(12+24)/2/8bits＝311KB。整个设计计算量和原方案相比没有变化，只是把R，B分量计算放在了拼接校正模块32之后，即在读出拼接校正模块32中的值同时，对需要补齐的R，B进行插值，即校正算法和插值同时完成，在设计中为了实现和原方案同样的处理速度，采用了多端口的拼接校正模块32存储架构，增加拼接校正模块32的读取带宽，满足了整个设计的需要。
可以理解，通过上述方案，整个流程中没有进行图像缓存，所需的外部存储器的带宽只要满足校正表的读写带宽，改进方案对内部拼接校正模块32进行了优化，使得内部缓存减小有利于设计芯片小型化。
进一步的说，图像水平拼接方案的优点在于，对sensor进行水平拼接，拼接后完整图像比较宽，相机可以采用相对长焦距的镜头，方案中采用5-8mm镜头，使得相机可以看得更远，本设计采用的前端处理系统如上所示采用无视频帧缓存模式，且拼接校正模块32得到有效控制，使得整个方案更为有效。
接下来说明图像垂直拼接方案。如果每个子摄像头的分辨率为1920*1080，并先对图像做90度转置后，再进行拼接，则拼接完成后图像分辨率为：X*Y，其中，理论分辨率小于4480*1920，分辨率和镜头及具体安装方式相关。为了便于对本申请进行说明，本实施方式中以4096*1800为例，拼接后的效果可参见图15，主要模块结构可参见图17。
进一步说，如图17所示，整个系统方案和水平拼接方案(参见图7)的差别在于，增加了与外部存储模块75相连的图像旋转模块73。图像旋转模块73用于对子摄像头同步拍摄的拍摄图像进行90°旋转，并将经旋转的拍摄图像发送给拼接校正模块32。旋转实现流程可参见图16。
下面以输入图像(1601)为20*8分辨率为例，简要说明转置方案的流程：
在步骤1611中，从图像(1601)输入按块方式准备写入内存(1602)，内存存储格式为32*5。图像在内存中的数据格式如图，采用把2维块展开成一维进行存储，方案中采用的块为4*4，工程中可以根据需要采用8*8或者4*8及其他块大小。
在步骤1612中，从内存(1602)中按一维顺序读出。接着，对读出的数据分4个FIFO(先入先出队列)缓存整理，整理方式在图中以箭头表示，按4个像素为单位循环放置4个FIFO，最终数据在FIFO中的格式见图16。
在步骤1613中，从4个FIFO中循环读出形成完成的图像(1603)，格式见图16，旋转后图像分辨率为8*20。
此后，读出旋转后的图像，并整理成为一帧图像数据流，再发送给拼接校正模块，其中拼接校正模块和双线性插值以及拼接融合过程和以上方案一相同，不再赘述。
图像垂直拼接方案的优点：对sensor进行垂直拼接，拼接后完整图像比较高，相机可以采用相对短焦距的镜头，方案中采用4-5mm镜头，使得相机上下视场角更大。本设计同样采用无视频帧缓存模式，且拼接校正模块32得到有效控制，使得整个方案较为有效。
需要说明的是，本申请各设备实施方式中提到的各模块都是逻辑模块，在物理上，一个逻辑模块可以是一个物理模块，也可以是一个物理模块的一部分，还可以以多个物理模块的组合实现，这些逻辑模块本身的物理实现方式并不是最重要的，这些逻辑模块所实现的功能的组合才是解决本申请所提出的技术问题的关键。此外，为了突出本申请的创新部分，本申请上述各设备实施方式并没有将与解决本申请所提出的技术问题关系不太密切的模块引入，这并不表明上述设备实施方式并不存在其它的模块。
本实施方式的优点如下：
通过拼接校正模块对拍摄图像进行局部缓存(行缓存)，有效解决了现有 技术中需要对拍摄图像进行整帧缓存再逐一读取并校正的过程中产生的问题，能够使拼接校正的过程本身不需要消耗内存带宽，从而极大地减小了对系统资源的消耗。
通过对拍摄图像先进行G插值处理，然后进行拼接校正，最后进行RB插值处理，能够使拼接校正模块的存储控制减少一半，明显降低了对资源的消耗。
整个系统设计简单，有利于设计小型化。
基于行缓存的拼接校正处理与Bayer插值算法结合，能够保证拍摄图像质量的同时很好地实现拼接校正处理。
参见图2，本申请第三实施方式提供一种全景摄像机，与第一实施方式、第二实施方式不同的是：
第一子摄像头11、第二子摄像头12、第三子摄像头13和第四子摄像头14的拍摄图像为Bayer格式的图像，并且，全景摄像机还包括插值模块322，用于对各子摄像头拍摄的拍摄图像进行RGB插值。如图9所示，Bayer格式的拍摄图像经过插值模块322进行RGB插值后，发送给拼接校正模块32，从而输出的拍摄图像的格式为RGB格式。参见图9可以理解，在本实施方式中，拼接校正模块32的输入是Bayer格式的图像，以及存储在外部存储器中的拼接校正表，而输出是拼接校正后的RGB图像。插值过程可参见图10。具体地说，图10的左侧是Bayer格式的拍摄图像，通过G插值生成完整的G图像，通过RB插值生成完整的R，B图像，最终每个像素点上R、G、B三原色都具有了，从而实现从baye格式的拍摄图像到RGB格式的插值。
其中，R、B插值过程如下：
Figure PCTCN2017080032-appb-000005
另一方面，对于G的插值，首先进行方向判断，计算边缘方向edgeDirection，下面举例说明边缘方向的计算过程：
以插值G35分量为例，计算边缘方向：
H＝|G34–G36|+|2*B35–B33–B37|；
V＝|G25–G45|+|2*B35–B15–B55|；
如果H<V；则edgeDirection＝＝0；(水平方向)
如果H>V；则edgeDirection＝＝1；(垂直方向)
如果H＝V；则edgeDirection＝＝2；(不区分方向)
如果edgeDirection＝＝0
G35＝(G34+G36)/2+(2*R35–R33–R37)/4；
如果edgeDirection＝＝1
G35＝(G25+G45)/2+(2*R35–R15–R55)/4；
否则
edgeDirection＝＝2
G35＝(G34+G36+G25+G45)/4+(2*R35–R33–R37+2*R35–R15–R55)/8；
同理G45的插值方案同G35，通过上述插值，最终输出的图像都为R、G、B三色图像。
此后，插值完成的RGB图像输入拼接校正模块32，对图像进行局部存储，其中拼接校正模块32的存储空间大小和图像本身畸变大小有关，在本实施方式中，镜头采用5mm-8mm镜头。在这种情况下，输入的拍摄图像为1920*1080，拍摄图像的最大畸变行数为72行，存储架构可参见图11。
具体地说，图11中的拼接校正模块32内部按行结构对图像按行存储，每个地址存储一个像素RBG值，设计的最大储存行数为最大畸变行数，即，在本实施方式中，是72行，在这种情况下，由计算得到存储空间：1920*72*36bits/8＝622KB。
下面简要说明line_cache的工作原理：
输入的拍摄图像按行结构存入拼接校正模块32中，当拼接校正模块32完成区域存储后，逻辑电路会同时从外部存储器中读出校正表，并解析校正表的信息。需要指出的是，校正表的描述信息是像素从原始坐标点移动到目标地址坐标。完成对拍摄图像的校正，也就是完成将原拍摄图像的每一个坐标点的像素值搬移到对应的坐标位置。
具体地说，根据校正表的信息从拼接校正模块32中找出当前要输出的像素坐标，并从拼接校正模块32中读出，送往后续模块进行后续处理，如图中所示，校正表中解析的拼接校正模块32中坐标为：(X1，Y1)；(X3，Y2)；(X4，Y4)；(X6，Y4)；(X7，Y5)；(X8，Y7)....的值顺序输出形成第一行完整数据，校正后的坐标为(X1，Y1)；(X2，Y1)；(X3，Y1)；(X4，Y1)；(X5，Y1)；(X6，Y1)...；其中输出坐标对应的像素值为输入坐标(X1，Y1)；(X3， Y2)；(X4，Y4)；(X6，Y4)；(X7，Y5)；(X8，Y7)坐标对应的RGB值不变。
采用上述拼接校正模块32局部缓存的方案的优点在于，解决了需要对图像进行整帧缓存再逐一读取校正的传统方案，使得校正算法本身不需要消耗内存带宽。
接着，如图7所示，输出的RGB格式的拍摄图像被输入到双线性插值模块74，以根据目标点在原图像上的位置，参考周边四个像素生成最终像素值。
具体地说，如图12所示，Px(1.1，1.3)，即位置坐标在(1.1，1.3)的像素值为例：
假设P0；P1；P2；P3的R分量值分别为：100，120，140，160；则：
Px的R分量＝(1-0.1)*(1-0.3)P0+0.1*(1-0.3)P1+(1-0.1)*0.3P2+0.1*0.3P3＝114；
同理得到G，B分量的值，最终输出经过双线性插值后的拍摄图像。
如图7所示，拼接融合模块34用于对上述完成拼接校正的拍摄图像进行融合，从而形成完整的大的拍摄图像，可参见图6。
本实施方式的全景摄像机的基本工作原理与上述实施方式类似，在此不做赘述。
本实施方式的优点如下：
通过拼接校正模块对拍摄图像进行局部缓存(行缓存)，有效解决了现有技术中需要对拍摄图像进行整帧缓存再逐一读取并校正的过程中产生的问题，能够使拼接校正的过程本身不需要消耗内存带宽，从而极大地减小了对系统资源的消耗。
基于行缓存的拼接校正处理与Bayer插值算法结合，能够保证拍摄图像质量的同时很好地实现拼接校正处理。
整个系统设计简单，有利于设计小型化。
参见图18，本实施方式的全景摄像机的拍摄方法中，全景摄像机包括4个子摄像头，每个子摄像头拍摄的角度不同；每个子摄像头拍摄的拍摄图像与全景摄像机中至少一个其他子摄像头拍摄的拍摄图像有部分重合，本实施方式的拍摄方法包括以下步骤：
步骤101：控制4个子摄像头同步拍摄。
步骤102：对各子摄像头同步拍摄的拍摄图像进行拼接校正，包含以下子步骤：以滑动窗口的方式缓存拍摄图像中连续的m行像素值，并根据预先设定的校正表和缓存的m行像素值，完成对像素值的校正，并依次输出校正后的每一行像素值，其中，m大于预先设定的图像最大畸变行数+2，并且m小于拍摄图像的总行数。
步骤103：根据各子摄像头拍摄的拍摄图像的重合部分，将各子摄像头同步拍摄的拍摄图像拼接成一个拼接图像；
步骤104：通过通信网络发送所述拼接图像。
需要在指出的是，上文中的实施方式是与本方法实施方式相对应的装置实施方式，本实施方式可与上述实施方式互相配合实施。上述实施方式中提到的相关技术细节在本实施方式中依然有效，为了减少重复，这里不再赘述。相应地，本实施方式中提到的相关技术细节也可应用在上述实施方式中。
本实施方式的优点如下：
通过对拍摄图像进行局部缓存(行缓存)，有效解决了现有技术中需要对拍摄图像进行整帧缓存再逐一读取并校正的过程中产生的问题，能够使拼接校正的过程本身不需要消耗内存带宽，从而极大地减小了对系统资源的消耗。
整个系统设计简单，有利于设计小型化。
实施例：全景摄像机的拍摄方法2
本实施方式的全景摄像机的拍摄方法与上述实施方式基本相同，不再赘述，它们之间的区别如下：
各个子摄像头拍摄的拍摄图像为Bayer格式，并且，本实施方式的拍摄方法还进一步包含以下步骤：
对各个子摄像头拍摄的拍摄图像进行RGB插值。
另外，在本申请的另一个实施方式中，在对各子摄像头同步拍摄的拍摄图像进行拼接校正的步骤之后，还包含以下步骤：
对校正后的拍摄图像，根据目标点在原拍摄图像上的位置，参考目标点周边四个像素，生成第二像素值，完成双线性插值。
同样地，上文中的实施方式是与本方法实施方式相对应的装置实施方式，本实施方式可与上述实施方式互相配合实施。上述实施方式中提到的相关技术细节在本实施方式中依然有效，为了减少重复，这里不再赘述。相应地，本实施方式中提到的相关技术细节也可应用在上述实施方式中。
本实施方式的优点如下：
通过对拍摄图像进行局部缓存(行缓存)，有效解决了现有技术中需要对拍摄图像进行整帧缓存再逐一读取并校正的过程中产生的问题，能够使拼接校正的过程本身不需要消耗内存带宽，从而极大地减小了对系统资源的消耗。
基于行缓存的拼接校正处理与Bayer插值算法结合，能够保证拍摄图像质量的同时很好地实现拼接校正处理。
整个系统设计简单，有利于设计小型化。
实施例：全景摄像机的拍摄方法3
本实施方式的全景摄像机的拍摄方法与上述实施方式基本相同，它们之间的区别如下：
各子摄像头拍摄的图像为Bayer格式，并且，本实施方式的拍摄方法还包含以下步骤：
对各子摄像头的拍摄图像进行G插值；
对经G插值之后的拍摄图像进行R插值和B插值。
具体地说，如图19所示，本实施方式的全景摄像机的拍摄方法包含以下步骤：
步骤201：控制4个子摄像头同步拍摄。
步骤202：对各子摄像头的拍摄图像进行G插值；
在本实施方式中，通过以下技术方案实现G插值：
H＝|Gij-Gi(j+2)|+|2*Bi(j+1)-Bi(j-1)-Bi(j+3)|
V＝|G(i-1)(j+1)-G(i+1)(j+1)|+|2*Bi(j+1)-B(i-2)(j+1)-B(i+2)(j+1)|
其中，H是水平方向参数，V是垂直方向参数
如果H<V，则edgeDirection＝0，其中，edgeDirection是边缘方向，则：
【公式1】
Figure PCTCN2017080032-appb-000006
如果H>V，则edgeDirection＝1，则：
【公式2】
Figure PCTCN2017080032-appb-000007
如果H＝V，则edgeDirection＝2，则：
【公式3】
Figure PCTCN2017080032-appb-000008
其中，R代表红色分量，G代表绿色分量，B代表蓝色分量，并且，i，j代表相互垂直的坐标值，[i，j]的位置上是绿色点，并且，[(i-1)，j]的位置和[(i+1)，j]的位置上是红色点，[i，(j-1)]的位置和[i，(j+1)]的位子上是蓝色点。
步骤203：对各子摄像头同步拍摄的拍摄图像进行拼接校正，包括：以滑动窗口的方式缓存拍摄图像中连续的m行像素值，并根据预先设定的校正表和缓存的m行像素值，完成对像素值的校正，并依次输出校正后的每一行像素值，其中，m大于预先设定的图像最大畸变行数+2，并且m小于拍摄图像的总行数。
由于校正后的图像行数一定比校正前小，在图像输出过程中，缓存区输出存在更新了多行，但输出只有一行的情况；
假设校正后图像为n行，对于n从1至，重复执行上述子步骤203，直至一个图像的所有行均被校正输出，其中，m为预先设定的图像最大畸变行数+3。
为了能更好地理解步骤203中的拼接校正过程，下面作进一步的解释和举例。
校正表中存储有校正前、后图像的对映关系信息，即，对于校正后图像中的像素，都可以在校正表中找到与之对应的校正前图像的像素的坐标信息。
校正表中包括校正表头。输出一行图像时需要查看当前行的校正表头。校正表头描述了校正后图像的一行来自于校正前图像中哪些行，即开始行和结束行。根据校正表头，如果正要输出的校正后图像的一行(称为当前行)所对应的校正前图像的开始行和结束行之间的行都已在缓存中，则可以从缓存中输出当前行。
下面举一个具体的例子：
假设校正前图像为1920行，校正后需要输出行数为1800行。通过校正表分析，要输出1800行的校正后图像的话，只需要用到校正前图像中的1-1837行，其中最大畸变小于64行，选定的拼接校正模块32(即以行为单位的缓存)为72行。每一帧图像开始，逐行将校正前图像的数据顺序往拼接校正模块32中写入，当写到16行时，根据校正表头中的信息，校正后图像的第1行(即 当前行)的输出需要用到校正前图像的第1－16行，而此时拼接校正模块32中已经缓存的数据已满足了要求，于是就根据校正表中的对映关系信息输出校正后图像的第1行。接着校正前图像的第17行像素输入拼接校正模块32，此时拼接校正模块32中有数据1-17行，同时根据校正表头信息可以知道需要校正前图像的第2－17行就可以输出校正后图像的第2行，而此条件此时已满足，于是根据校正表中的对映关系信息输出校正后图像的第2行。……以此类推，将校正前图像逐行输入拼接校正模块32(如果拼接校正模块32已满则根据先入先出FIFO的原则将最早输入的行丢弃)，并根据校正表头信息判断当前拼接校正模块32中的数据是否能够输出校正后图像的当前1行，如果能够则输出校正后图像的当前1行，否则继续向拼接校正模块32输入1行校正前图像的数据。这个过程中可能会多处存在以下两种特殊情况：
1.写入校正前图像的1行到拼接校正模块32，根据校正表头中的信息判定要输出的校正后图像的当前1行所需要的校正前图像中的行并没有全在拼接校正模块32中，则需要继续输入校正前图像的下1行到拼接校正模块32中才能满足输出条件，即输入2行甚至多行才能输出1行；
2.在输出校正后图像的1行之后，根据校正表头中的信息判定，要输出的校正后图像的下1行所需参考的校正前图像的相关行都已经在拼接校正模块32中了，此时不用再向拼接校正模块32输入新的校正前图像的行就可以继续输出校正后的下1行，即此时输入1行数据输出2行。
整个过程如上描述，当校正前图像的第1837行输入到拼接校正模块32中后，在拼接校正模块32中的数据满足校正后图像的第1800行输出的条件，校正后图像的第1800行图像输出，对于校正前图像中后续1837--1920行图像数据不再写入拼接校正模块32中。当新的一帧开始时整个line_cache清空进入下个循环。
本申请各实施方式中所称的图像的行和列是可以由用户自定义的，例如对于1920*1080的图像，可以将1920定义为行、1080定义为列，也可以将1080定义为行、1920定义为列。
在一个优选例中，图像最大畸变行数由测试图像统计得到，即将标准的测试图像供各子摄像头拍摄，将拍摄所得图像与标准的测试图像相比较，即 可得到各像素点在拍摄前后图像中的位置变化，再进行统计得到最大的畸变所跨越的行数，即图像最大畸变行数。在另一例子中，可以根据经验值设定图像最大畸变行数。
在步骤204：对经G插值之后的拍摄图像进行R插值和B插值。
在一个优选例中，通过以下技术方案实现R插值和B插值：
【公式4】
Rij＝Gij+{[Ri(j-1)-Gi(j-1)]+[Ri(j+1)-Gi(j+1)]}/2
【公式5】
Bij＝Gij+{[R(i-1)j-Gi-1)j]+[R(i+1)j-G(++1)j]}/2
【公式6】
Bi(j+1)＝Gi(j+1)+{[B(i-1)j-Gi-1)j]+[B(i-1)(j+2)-Gi-1)(j+2)]+[B(i+1)j-G(i+1)j]+[B(i+1)(j+2)-G(i+1)(j+2)]}/4
【公式7】
Ri(j+1)＝G(i+1)j+{[Ri(j-1)-Gi(j-1)]+[Ri(j+1)-Gi(j+1)]+[R(i+2)(j-1)-G(i+2)(j-1)]+[R(i+2)(j+1)-G(i+2)(j+1)]}/4
【公式8】
R(i+1)(j+1)＝G(i+1)(j+1)+{[Ri(j+1)-Gi(j+1)]+[R(i+2)(j+1)-G(i+2)(j+1)]}/2
【公式9】
R(i+1)(j+1)＝G(i+1)(j+1)+{[R(i+1)j-G(i+1)j]+[R(i+1)(j+2)-G(i+1)(j+2)]}/2
其中，R代表红色分量，G代表绿色分量，B代表蓝色分量，并且，i，j代表相互垂直的坐标值，[i，j]的位置上是绿色点，并且，[(i-1)，j]的位置和[(i+1)，j]的位置上是红色点，[i，(j-1)]的位置和[i，(j+1)]的位子上是蓝色点。
步骤205：根据各子摄像头拍摄的拍摄图像的重合部分，将各子摄像头同步拍摄的拍摄图像拼接成一个拼接图像；
步骤206：对经拼接图像进行图像压缩；
步骤207：通过通信网络发送拼接图像。
本领域的技术人员可以理解，对于一个完整的RGB图像来说，每一个像素都是由RGB三个颜色组成的，可以将一个RGB图像分为R、G、B三个颜色的三个子图像。
G插值就是对Bayer图像中的绿色部分进行插值，得到完整的绿色子图像(G图像)。
R插值就是对Bayer图像中的红色部分进行插值，得到完整的红色子图像(R图像)。
B插值就是对Bayer图像中的蓝色部分进行插值，得到完整的蓝色子图像(B图像)。
进行拼接校正的步骤在进行G插值的步骤之后，且进行拼接校正的步骤在进行R插值和B插值的步骤之前。
在本申请的另一个实施方式中，也可以先进行G插值和RB插值，得到完整的RGB图像后，再进行拼接校正。
在本申请的另一个实施方式中，传输带宽足够时，也可以不进行图像压缩，直接将拼接图像传输出去。
在本申请的另一个实施方式中，在对各子摄像头同步拍摄的拍摄图像进行拼接校正和RB插值的步骤之后，还包含以下步骤：
对校正后的拍摄图像，根据目标点在原拍摄图像上的位置，参考所述目 标点周边四个像素，生成第二像素值，完成双线性插值。
由此可见，在本实施方式的拍摄方法中，对各子摄像头的拍摄图像进行G插值的步骤，在对拍摄图像进行拼接校正的步骤之前；并且，对经所述G插值之后的图像进行R插值和B插值的步骤，在对拍摄图像进行拼接校正的步骤之后。
同样地，上文中的实施方式是与本方法实施方式相对应的装置实施方式，本实施方式可与上述实施方式互相配合实施。上述实施方式中提到的相关技术细节在本实施方式中依然有效，为了减少重复，这里不再赘述。相应地，本实施方式中提到的相关技术细节也可应用在上述实施方式中。
本实施方式的优点如下：
通过对拍摄图像进行局部缓存(行缓存)，有效解决了现有技术中需要对拍摄图像进行整帧缓存再逐一读取并校正的过程中产生的问题，能够使拼接校正的过程本身不需要消耗内存带宽，从而极大地减小了对系统资源的消耗。
通过对拍摄图像先进行G插值处理，然后进行拼接校正，最后进行RB插值处理，能够使拼接校正模块32(拼接校正模块)的存储控制减少一半，明显降低了对资源的消耗。
整个系统设计简单，有利于设计小型化。
基于行缓存的拼接校正处理与Bayer插值算法结合，能够保证拍摄图像质量的同时很好地实现拼接校正处理。
综上所述，本申请的主要优点包括：
1)通过拼接校正模块32对拍摄图像进行局部缓存，有效解决了现有技术中需要对拍摄图像进行整帧缓存再逐一读取并校正的过程中产生的问题， 能够使拼接校正的过程本身不需要消耗内存带宽，从而极大地减小了对系统资源的消耗。
2)通过对拍摄图像先进行G插值处理，然后进行拼接校正，最后进行RB插值处理，能够使拼接校正模块的存储控制减少一半，明显降低了对资源的消耗。
3)整个系统设计简单，有利于设计小型化。
4)基于行缓存的拼接校正处理与Bayer插值算法结合，能够保证拍摄图像质量的同时很好地实现拼接校正处理。
本申请实施例还提供了一种全景摄像机，包括：N个子摄像头、壳体、处理器、存储器、电路板和电源电路；其中，N大于2，每个所述子摄像头拍摄的角度不同；每个所述子摄像头拍摄的拍摄图像与所述全景摄像机中至少一个其他子摄像头拍摄的拍摄图像有部分重合；电路板安置在壳体围成的空间内部，处理器和存储器设置在电路板上；电源电路，用于为全景摄像机的各个电路或器件供电；存储器用于存储可执行程序代码；处理器通过读取存储器中存储的可执行程序代码来运行与可执行程序代码对应的程序，以用于执行所述拍摄方法。方法包括：
控制所述N个子摄像头同步拍摄；
对所述各子摄像头同步拍摄的拍摄图像进行拼接校正，包含以下子步骤：以滑动窗口的方式缓存所述拍摄图像中连续的m行像素值，并根据预先设定的校正表和所述缓存的m行像素值，完成对所述像素值的校正，并依次输出校正后的每一行像素值，其中，m大于预先设定的图像最大畸变行数+2，并且m小于所述拍摄图像的总行数；
根据各子摄像头拍摄的拍摄图像的重合部分，将所述各子摄像头同步拍摄的拍摄图像拼接成一个拼接图像；
通过通信网络发送所述拼接图像。
应用本申请所示实施例，通过对拍摄图像进行局部缓存(行缓存)，有效 解决了现有技术中需要对拍摄图像进行整帧缓存再逐一读取并校正的过程中产生的问题，能够使拼接校正的过程本身不需要消耗内存带宽，从而极大地减小了对系统资源的消耗。
本申请实施例还提供了一种可执行程序代码，所述可执行程序代码用于被运行以执行所述拍摄方法，方法包括：
控制所述N个子摄像头同步拍摄；
对所述各子摄像头同步拍摄的拍摄图像进行拼接校正，包含以下子步骤：以滑动窗口的方式缓存所述拍摄图像中连续的m行像素值，并根据预先设定的校正表和所述缓存的m行像素值，完成对所述像素值的校正，并依次输出校正后的每一行像素值，其中，m大于预先设定的图像最大畸变行数+2，并且m小于所述拍摄图像的总行数；
根据各子摄像头拍摄的拍摄图像的重合部分，将所述各子摄像头同步拍摄的拍摄图像拼接成一个拼接图像；
通过通信网络发送所述拼接图像。
应用本申请所示实施例，通过对拍摄图像进行局部缓存(行缓存)，有效解决了现有技术中需要对拍摄图像进行整帧缓存再逐一读取并校正的过程中产生的问题，能够使拼接校正的过程本身不需要消耗内存带宽，从而极大地减小了对系统资源的消耗。
本申请实施例还提供了一种存储介质，所述存储介质用于存储可执行程序代码，所述可执行程序代码用于被运行以执行所述拍摄方法，方法包括：
控制所述N个子摄像头同步拍摄；
对所述各子摄像头同步拍摄的拍摄图像进行拼接校正，包含以下子步骤：以滑动窗口的方式缓存所述拍摄图像中连续的m行像素值，并根据预先设定的校正表和所述缓存的m行像素值，完成对所述像素值的校正，并依次输出校正后的每一行像素值，其中，m大于预先设定的图像最大畸变行数+2，并且m小于所述拍摄图像的总行数；
根据各子摄像头拍摄的拍摄图像的重合部分，将所述各子摄像头同步拍 摄的拍摄图像拼接成一个拼接图像；
通过通信网络发送所述拼接图像。
应用本申请所示实施例，通过对拍摄图像进行局部缓存(行缓存)，有效解决了现有技术中需要对拍摄图像进行整帧缓存再逐一读取并校正的过程中产生的问题，能够使拼接校正的过程本身不需要消耗内存带宽，从而极大地减小了对系统资源的消耗。
本申请的各方法实施方式均可以以软件、硬件、固件等方式实现。不管本申请是以软件、硬件、还是固件方式实现，指令代码都可以存储在任何类型的计算机可访问的存储器中(例如永久的或者可修改的，易失性的或者非易失性的，固态的或者非固态的，固定的或者可更换的介质等等)。同样，存储器可以例如是可编程阵列逻辑(Programmable Array Logic，简称“PAL”)、随机存取存储器(Random Access Memory，简称“RAM”)、可编程只读存储器(Programmable Read Only Memory，简称“PROM”)、只读存储器(Read-Only Memory，简称“ROM”)、电可擦除可编程只读存储器(Electrically Erasable Programmable ROM，简称“EEPROM”)、磁盘、光盘、数字通用光盘(Digital Versatile Disc，简称“DVD”)等等。
本申请各设备实施方式中提到的各单元都是逻辑单元，在物理上，一个逻辑单元可以是一个物理单元，也可以是一个物理单元的一部分，还可以以多个物理单元的组合实现，这些逻辑单元本身的物理实现方式并不是最重要的，这些逻辑单元所实现的功能的组合才是解决本申请所提出的技术问题的关键。此外，为了突出本申请的创新部分，本申请上述各设备实施方式并没有将与解决本申请所提出的技术问题关系不太密切的单元引入，这并不表明上述设备实施方式并不存在其它的单元。
在本申请提及的所有文献都在本申请中引用作为参考，就如同每一篇文献被单独引用作为参考那样。此外应理解，在阅读了本申请的上述讲授内容之后，本领域技术人员可以对本申请作各种改动或修改，这些等价形式同样落于本申请所附权利要求书所限定的范围。
并且，在本专利的权利要求书和说明书中，诸如第一和第二等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。而且，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括一个”限定的要素，并不排除在包括所述要素的过程、方法、物品或者设备中还存在另外的相同要素。本专利的权利要求书和说明书中，如果提到根据某要素执行某行为，则是指至少根据该要素执行该行为的意思，其中包括了两种情况：仅根据该要素执行该行为、和根据该要素和其它要素执行该行为。
虽然通过参照本申请的某些优选实施例，已经对本申请进行了图示和描述，但本领域的普通技术人员应该明白，可以在形式上和细节上对其作各种改变，而不偏离本申请的精神和范围。

Claims (15)

  1. 一种全景摄像机，其特征在于，包括：
    N个子摄像头，N大于2；每个所述子摄像头拍摄的角度不同；每个所述子摄像头拍摄的拍摄图像与所述全景摄像机中至少一个其他子摄像头拍摄的拍摄图像有部分重合；
    同步控制模块，用于控制所述N个子摄像头进行同步拍摄；
    拼接校正模块，用于以滑动窗口的方式缓存所述拍摄图像中连续的m行像素值，并根据预先设定的校正表和所述缓存的m行像素值，对所述像素值进行校正，并依次输出校正后的每一行像素值，得到校正后的拍摄图像，其中，m大于预先设定的图像最大畸变行数+2，并且m小于所述拍摄图像的总行数；
    拼接融合模块，用于根据所述各子摄像头的拍摄图像的重合部分，将所述各个子摄像头在所述同步控制模块控制下同步拍摄的校正后的拍摄图像拼接成一个拼接图像；
    传输模块，用于通过通信网络发送所述拼接融合模块输出的拼接图像。
  2. 如权利要求1所述的全景摄像机，其特征在于，所述N个子摄像头的拍摄图像为Bayer格式的图像，并且，所述全景摄像机还包括：
    插值模块，用于对所述N个子摄像头拍摄的Bayer格式的图像进行RGB插值，并将RGB插值后的拍摄图像发送给所述拼接校正模块。
  3. 如权利要求1所述的全景摄像机，其特征在于，所述N个子摄像头的拍摄图像为Bayer格式的图像，并且，所述全景摄像机还包括G插值模块和RB插值模块，其中，
    所述G插值模块用于对所述N个子摄像头拍摄的Bayer格式的图像进行G插值，并将经G插值后的拍摄图像发送给所述拼接校正模块；
    所述拼接校正模块用于对所述经G插值后的拍摄图像进行校正，并将校正后的拍摄图像发送给所述RB插值模块；
    所述RB插值模块用于对所述校正后的拍摄图像进行R插值和B插值。
  4. 如权利要求1所述的全景摄像机，其特征在于，还包含
    双线性插值模块，用于对由所述拼接校正模块校正后的拍摄图像，根据目标点在原拍摄图像上的位置，参考所述目标点周边四个像素，生成第二像素值，完成双线性插值，并将经双线性插值后的拍摄图像发送给所述拼接融合模块。
  5. 如权利要求1所述的全景摄像机，其特征在于，还包含
    图像旋转模块，用于对所述子摄像头同步拍摄的拍摄图像进行90°旋转，并将经旋转的拍摄图像发送给所述拼接校正模块。
  6. 根据权利要求1所述的全景摄像机，其特征在于，每个所述子摄像头包括一个感光传感器和一个透镜，每个所述子摄像头间隔相同的角度，且每个所述子摄像头与地平面夹角相同。
  7. 一种全景摄像机的拍摄方法，其特征在于，所述全景摄像机包括N个子摄像头，N大于2；每个所述子摄像头拍摄的角度不同；每个所述子摄像头拍摄的拍摄图像与所述全景摄像机中至少一个其他子摄像头拍摄的拍摄图像有部分重合；
    所述方法包括以下步骤：
    控制所述N个子摄像头同步拍摄；
    对所述各子摄像头同步拍摄的拍摄图像进行拼接校正，包含以下子步骤：以滑动窗口的方式缓存所述拍摄图像中连续的m行像素值，并根据预先设定 的校正表和所述缓存的m行像素值，完成对所述像素值的校正，并依次输出校正后的每一行像素值，其中，m大于预先设定的图像最大畸变行数+2，并且m小于所述拍摄图像的总行数；
    根据各子摄像头拍摄的拍摄图像的重合部分，将所述各子摄像头同步拍摄的拍摄图像拼接成一个拼接图像；
    通过通信网络发送所述拼接图像。
  8. 如权利要求7所述的全景摄像机的拍摄方法，其特征在于，所述N个子摄像头拍摄的图像为Bayer格式，并且，所述方法还包含以下步骤：
    对所述各个子摄像头拍摄的拍摄图像进行RGB插值。
  9. 如权利要求7所述的全景摄像机的拍摄方法，其特征在于，所述N个子摄像头拍摄的图像为Bayer格式，并且，
    在所述对所述各子摄像头同步拍摄的拍摄图像进行拼接校正的步骤之前，还包含以下步骤：
    对所述各子摄像头的拍摄图像进行G插值；
    在所述对所述各子摄像头同步拍摄的拍摄图像进行拼接校正的步骤之后，还包含以下步骤：
    对经所述G插值和拼接校正之后的拍摄图像进行R插值和B插值。
  10. 如权利要求7所述的全景摄像机的拍摄方法，其特征在于，
    在所述对所述各子摄像头同步拍摄的拍摄图像进行拼接校正的步骤之后，还包含以下步骤：
    对校正后的拍摄图像，根据目标点在原拍摄图像上的位置，参考所述目标点周边四个像素，生成第二像素值，完成双线性插值。
  11. 如权利要求7所述的全景摄像机的拍摄方法，其特征在于，
    在所述控制所述N个子摄像头同步拍摄的步骤之后，还包含以下步骤：
    对所述N个子摄像头同步拍摄的拍摄图像进行90°旋转。
  12. 如权利要求7所述的全景摄像机的拍摄方法，其特征在于，在所述通过通信网络发送所述拼接图像的步骤之前，还包括以下步骤：
    对所述拼接图像进行图像压缩。
  13. 一种全景摄像机，其特征在于，包括：N个子摄像头、壳体、处理器、存储器、电路板和电源电路；其中，N大于2，每个所述子摄像头拍摄的角度不同；每个所述子摄像头拍摄的拍摄图像与所述全景摄像机中至少一个其他子摄像头拍摄的拍摄图像有部分重合；电路板安置在壳体围成的空间内部，处理器和存储器设置在电路板上；电源电路，用于为全景摄像机的各个电路或器件供电；存储器用于存储可执行程序代码；处理器通过读取存储器中存储的可执行程序代码来运行与可执行程序代码对应的程序，以用于执行权利要求7-12任一项所述的拍摄方法。
  14. 一种可执行程序代码，其特征在于，所述可执行程序代码用于被运行以执行权利要求7-12任一项所述的拍摄方法。
  15. 一种存储介质，其特征在于，所述存储介质用于存储可执行程序代码，所述可执行程序代码用于被运行以执行权利要求7-12任一项所述的拍摄方法。
PCT/CN2017/080032 2016-06-14 2017-04-11 全景摄像机及其拍摄方法 WO2017215331A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US16/310,075 US10554882B2 (en) 2016-06-14 2017-04-11 Panoramic camera and photographing method thereof
EP17812443.4A EP3471392B1 (en) 2016-06-14 2017-04-11 Panoramic camera and photographing method thereof

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201610416973.2 2016-06-14
CN201610416973.2A CN107249096B (zh) 2016-06-14 2016-06-14 全景摄像机及其拍摄方法

Publications (1)

Publication Number Publication Date
WO2017215331A1 true WO2017215331A1 (zh) 2017-12-21

Family

ID=60016144

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/080032 WO2017215331A1 (zh) 2016-06-14 2017-04-11 全景摄像机及其拍摄方法

Country Status (4)

Country Link
US (1) US10554882B2 (zh)
EP (1) EP3471392B1 (zh)
CN (1) CN107249096B (zh)
WO (1) WO2017215331A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117953470A (zh) * 2024-03-26 2024-04-30 杭州感想科技有限公司 全景拼接相机的高速公路事件识别方法及装置

Families Citing this family (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107948505B (zh) * 2017-11-14 2020-06-23 维沃移动通信有限公司 一种全景拍摄方法及移动终端
JP6845173B2 (ja) * 2018-03-15 2021-03-17 株式会社東芝 画像処理装置及び画像処理方法
CN108513107A (zh) * 2018-06-15 2018-09-07 苏州思在科技有限公司 一种摄像装置
CN108932695B (zh) * 2018-07-02 2022-10-14 安徽理工大学 一种基于几何结构的井壁弧形图校正方法
CN109600584A (zh) * 2018-12-11 2019-04-09 中联重科股份有限公司 观察塔机的方法和装置、塔机及机器可读存储介质
CN110087032A (zh) * 2019-04-24 2019-08-02 招商局重庆公路工程检测中心有限公司 一种全景式隧道视频监视装置及方法
CN110217271A (zh) * 2019-05-30 2019-09-10 成都希格玛光电科技有限公司 基于图像视觉的快速轨道侵限识别监测系统及方法
CN110087054B (zh) * 2019-06-06 2021-06-18 北京七鑫易维科技有限公司 图像的处理方法、装置及系统
CN110944129B (zh) * 2019-12-11 2021-10-22 中国电子科技集团公司第四十四研究所 一种复用型像素控制电路
WO2021168690A1 (en) * 2020-02-26 2021-09-02 Shenzhen Xpectvision Technology Co., Ltd. Image sensors and methods of operating the same
CN113724141B (zh) * 2020-05-26 2023-09-05 杭州海康威视数字技术股份有限公司 一种图像校正方法、装置及电子设备
CN111741259B (zh) * 2020-06-11 2022-05-06 北京三快在线科技有限公司 无人驾驶设备的控制方法及装置
CN111738923B (zh) * 2020-06-19 2024-05-10 京东方科技集团股份有限公司 图像处理方法、设备及存储介质
CN111917984A (zh) * 2020-08-13 2020-11-10 上海航天测控通信研究所 一种虚拟云台及控制方法
CN112004023A (zh) * 2020-08-31 2020-11-27 深圳创维数字技术有限公司 拍摄方法、多摄像头模组以及存储介质
CN112163442B (zh) * 2020-09-29 2022-05-06 杭州海康威视数字技术股份有限公司 一种图形码识别系统、方法及装置
CN112581369A (zh) * 2020-12-24 2021-03-30 中国银联股份有限公司 图像拼接方法以及装置
CN112598577B (zh) * 2020-12-24 2022-02-11 暨南大学 基于错位采样的图像插值方法、系统及存储介质
US11665330B2 (en) 2021-01-27 2023-05-30 Dell Products L.P. Dynamic-baseline imaging array with real-time spatial data capture and fusion
CN113112404B (zh) * 2021-04-02 2023-04-28 广州图匠数据科技有限公司 一种基于滑动窗口的图像拼接方法及装置
CN114119410B (zh) * 2021-11-19 2022-04-22 航天宏康智能科技(北京)有限公司 校正畸变表格图像中的单元格的方法及装置
CN117438056B (zh) * 2023-12-20 2024-03-12 达州市中心医院(达州市人民医院) 用于消化内镜影像数据的编辑筛选与存储控制方法和系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102355545A (zh) * 2011-09-20 2012-02-15 中国科学院宁波材料技术与工程研究所 一种360度实时全景摄像机
CN102857739A (zh) * 2012-08-20 2013-01-02 上海光亮光电科技有限公司 分布式全景监控系统及其方法
CN103338343A (zh) * 2013-05-29 2013-10-02 山西绿色光电产业科学技术研究院（有限公司） 以全景图像为基准的多路图像无缝拼接方法及装置
US20140098185A1 (en) * 2012-10-09 2014-04-10 Shahram Davari Interactive user selected video/audio views by real time stitching and selective delivery of multiple video/audio sources
US20150138311A1 (en) * 2013-11-21 2015-05-21 Panavision International, L.P. 360-degree panoramic camera systems

Family Cites Families (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH08191440A (ja) * 1995-01-10 1996-07-23 Fukuda Denshi Co Ltd 内視画像補正方法及び装置
US6522325B1 (en) * 1998-04-02 2003-02-18 Kewazinga Corp. Navigable telepresence method and system utilizing an array of cameras
US7084904B2 (en) * 2002-09-30 2006-08-01 Microsoft Corporation Foveated wide-angle imaging system and method for capturing and viewing wide-angle images in real time
US7834910B2 (en) * 2006-03-01 2010-11-16 David M. DeLorme Method and apparatus for panoramic imaging
KR100725053B1 (ko) * 2006-06-22 2007-06-08 삼성전자주식회사 휴대용 단말기의 파노라마 촬영 장치 및 방법
US8253815B2 (en) * 2008-09-16 2012-08-28 Altia Systems Inc. Synchronized multiple imager system and method
JP5558973B2 (ja) * 2010-08-31 2014-07-23 株式会社日立情報通信エンジニアリング 画像補正装置、補正画像生成方法、補正テーブル生成装置、補正テーブル生成方法、補正テーブル生成プログラムおよび補正画像生成プログラム
CN102592124B (zh) * 2011-01-13 2013-11-27 汉王科技股份有限公司 文本图像的几何校正方法、装置和双目立体视觉系统
US9405795B2 (en) * 2011-07-20 2016-08-02 Hitachi, Ltd. Stream data processing server and a non-transitory computer-readable storage medium storing a stream data processing program
CN102497209B (zh) * 2011-12-06 2014-01-22 南京信息工程大学 滑动窗口式数据采样方法及装置
CN102611822B (zh) * 2012-03-14 2015-07-01 海信集团有限公司 投影仪及其投影图像校正方法
US9563105B1 (en) * 2013-04-10 2017-02-07 Ic Real Tech Inc. Screw coupler enabling direct secure fastening between communicating electronic components
TWI545388B (zh) * 2013-06-14 2016-08-11 豪威科技股份有限公司 產生全景圖像的系統和方法
WO2014208378A1 (ja) * 2013-06-28 2014-12-31 ソニー株式会社 情報処理装置、情報処理方法、プログラム、及び、情報処理システム
CN103646385A (zh) * 2013-11-21 2014-03-19 江西好帮手电子科技有限公司 一种全景泊车图像的自动拼接方法及其系统
US9524426B2 (en) * 2014-03-19 2016-12-20 GM Global Technology Operations LLC Multi-view human detection using semi-exhaustive search
US9420176B2 (en) * 2014-06-19 2016-08-16 Omnivision Technologies, Inc. 360 degree multi-camera system
US20160295108A1 (en) * 2015-04-01 2016-10-06 Cheng Cao System and method for panoramic imaging
US20170126985A1 (en) * 2015-10-30 2017-05-04 Gopro, Inc. Enumeration of Cameras in an Array
US9842624B2 (en) * 2015-11-12 2017-12-12 Intel Corporation Multiple camera video image stitching by placing seams for scene objects
US9811946B1 (en) * 2016-05-30 2017-11-07 Hong Kong Applied Science and Technology Research Institute Company, Limited High resolution (HR) panorama generation without ghosting artifacts using multiple HR images mapped to a low resolution 360-degree image

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102355545A (zh) * 2011-09-20 2012-02-15 中国科学院宁波材料技术与工程研究所 一种360度实时全景摄像机
CN102857739A (zh) * 2012-08-20 2013-01-02 上海光亮光电科技有限公司 分布式全景监控系统及其方法
US20140098185A1 (en) * 2012-10-09 2014-04-10 Shahram Davari Interactive user selected video/audio views by real time stitching and selective delivery of multiple video/audio sources
CN103338343A (zh) * 2013-05-29 2013-10-02 山西绿色光电产业科学技术研究院（有限公司） 以全景图像为基准的多路图像无缝拼接方法及装置
US20150138311A1 (en) * 2013-11-21 2015-05-21 Panavision International, L.P. 360-degree panoramic camera systems

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3471392A4 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117953470A (zh) * 2024-03-26 2024-04-30 杭州感想科技有限公司 全景拼接相机的高速公路事件识别方法及装置

Also Published As

Publication number Publication date
US10554882B2 (en) 2020-02-04
US20190335099A1 (en) 2019-10-31
CN107249096A (zh) 2017-10-13
EP3471392A4 (en) 2019-05-22
CN107249096B (zh) 2021-02-26
EP3471392B1 (en) 2021-07-28
EP3471392A1 (en) 2019-04-17

Similar Documents

Publication Publication Date Title
WO2017215331A1 (zh) 全景摄像机及其拍摄方法
US11064110B2 (en) Warp processing for image capture
US11457157B2 (en) High dynamic range processing based on angular rate measurements
US9386287B2 (en) Image processor which rearranges color information, image processing method, and digital camera
US20050007478A1 (en) Multiple-view processing in wide-angle video camera
US11871105B2 (en) Field of view adjustment
JP6129119B2 (ja) 画像処理装置、画像処理システム、撮像装置、および画像処理方法
US20170345129A1 (en) In loop stitching for multi-camera arrays
CN104023169A (zh) 成像设备、控制该成像设备的方法和计算机程序
JP2007129573A (ja) 撮像装置
TWI386038B (zh) 視訊信號擷取裝置，信號處理及控制裝置，和視訊信號擷取，視訊信號處理及轉換系統及方法
WO2019128274A1 (zh) 摄像装置的控制方法、多光谱摄像设备、无人机及介质
WO2017205597A1 (en) Image signal processing-based encoding hints for motion estimation
Wilburn High-performance imaging using arrays of inexpensive cameras
TWI475875B (zh) 三維影像產生裝置
US20180211413A1 (en) Image signal processing using sub-three-dimensional look-up tables
JP5187316B2 (ja) 画像撮影装置、エンコード方法およびプログラム
JP6012167B2 (ja) 画像処理装置、画像処理方法およびコンピュータプログラム
JP2013219682A (ja) 撮像装置
CN117939201A (zh) 基于人体识别的图像采集测量现场修正系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17812443

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2017812443

Country of ref document: EP

Effective date: 20190114