TW201410015A - A multiple camera system and method therefore - Google Patents

Abstract

A multiple camera system and method therefore. The multiple camera system comprises of a first camera, a second camera and a control module. The first camera takes a first image, the second takes a second image. The control module couples to the first camera and the second camera. The control module receives the first image and the second image. The control module selects a target object in the first image and at least three reference point. The control module calculates the coordinate parameters according to the reference points. The control module seeks the response view in the second image according to the coordinate parameters. The control module adjusts the view of the second camera. The second camera takes the target object.

Description

Multi-camera integrated processing system and method thereof

An integrated system and method for multiple electronic devices, in particular, relates to an integrated processing system for a multi-camera and a method thereof.

With the popularization of monitoring equipment, more and more people will install monitoring equipment to protect their property and personal safety. In order to be able to monitor without a blind spot, there are manufacturers that propose a camera that can be panned. The implementation of the panoramic camera can be achieved by joining multiple sets of lenses, or by using a single fisheye lens. The advantage of panoramic shooting is that the captured digital image can be viewed in all directions and without dead ends.

Although the panoramic camera has the advantage of wide-angle shooting, the panoramic camera cannot perform optical zoom processing. Therefore, the viewer cannot perform scaling of the image from any of the viewing targets in the panoramic image. Although the panoramic camera can zoom in and out of the viewing target by digital zoom. However, such a processing method is only using interpolation processing of pixels, so that the image resolution after digital zooming is greatly reduced, which causes a problem that the digital image has poor image quality. If an accident occurs far away from the panoramic camera, it is more difficult for the viewer to view the clear viewing target from the panoramic image.

Compared to a panoramic camera, a pan-tilt-zoom (PTZ) can perform optical zoom processing. Therefore, the swivel zoom camera can zoom in the shooting field so that the distant viewing target can be enlarged. But rotary zoom photography The shooting field of the machine is smaller than the shooting field of the panoramic camera. Therefore, US 7,990,422 and US 7,479,979 propose to use a panoramic camera with another rotary zoom camera to shoot the same scene.

However, since the viewing angles of cameras at different locations may not be exactly the same, if the other rotary zoom camera in the environment is driven from the viewpoint of a panoramic camera in the environment, an error occurs in the field of view. Therefore, the conventional photographic frame setting method needs to erect two cameras as close as possible to each other to alleviate the above problems.

In view of the above problems, the present invention provides an integrated processing system for a multi-camera, which integrates cameras of different shooting fields and performs interlocking processing on the same viewing target.

The integrated processing system of the multi-camera disclosed by the present invention comprises a first camera, a second camera and a control module. The first camera captures the first image; the second camera captures the second image, the shooting range of the first image at least partially covers the shooting range of the second image; and the second camera performs panoramic mapping on the second image at different viewing angles, and obtains a panoramic image; the control module receives the first image and the panoramic image; the control module selects at least three reference points of the corresponding positions from the first image and the panoramic image; the control module selects a viewing target from the first image; The group selects three reference points adjacent to the viewing target to perform an affine transformation parameter calculation operation of the first image on the second image and obtains a conversion parameter; the control module performs affine transformation processing according to the conversion parameter to calculate a position of the viewing target in the panoramic image. The control module adjusts the shooting range of the second camera so that the second camera captures the viewing target.

The invention further provides a processing method for integrating a multi-camera, comprising the steps of: respectively capturing a first image by a first camera, and capturing a second image by a second camera, and the field of view of the first image at least partially covering a field of view of the second image; Performing a panoramic connection on the second image at different viewing angles and obtaining a panoramic image; selecting at least three reference points of the corresponding positions from the first image and the panoramic image; selecting a viewing target from the first image; and selecting a neighboring viewing target The three reference points perform the affine transformation parameter calculation operation of the first image on the second image and obtain the conversion parameter; perform affine transformation processing according to the conversion parameter to calculate the position of the viewing target in the panoramic image; adjust the field of view of the second camera , so that the second camera shoots the target.

The integrated processing system and method of the multi-camera proposed by the invention are applied to a plurality of cameras with different shooting fields, so that multiple cameras can synchronously shoot the same viewing target.

The features and implementations of the present invention are described in detail below with reference to the drawings.

Please refer to FIG. 1 , which is a schematic diagram of the architecture of the present invention. The integrated processing system of the multi-camera of the present invention includes a first camera 110, a second camera 120, and a control module 130. The first camera 110 captures the first image 111 and the second camera 120 captures the second image 121. The first camera 110 is a fisheye camera or a panoramic camera or a wide-angle camera. In this embodiment, a fisheye camera is taken as an example. Therefore, the first image 111 is a panoramic image (or a fisheye image, please refer to FIG. 2A). ). In general, a panoramic image can cover a 360-degree field of view. Second camera 120 It is a rotary zoom camera or a camera with digital zoom zoom, and the second camera 120 captures the second image 121. For the imaging of the second image 121, please refer to FIG. 2B. The second camera 120 can perform optical zoom (or digital zoom) and a process of adjusting the field of view. The second image 121 in different fields of view can be connected into the panoramic image 122 by using a conventional algorithm (such as panoramic stitching), as shown in FIG. 2C. Basically, the captured field of view of the first image 111 covers at least the captured field of view of the partial panoramic image 122. In the case where the photographing field of view of the first image 111 does not cover the photographing field of the panoramic image 122, the corresponding processing is not performed in the present invention.

The first camera 110 and the second camera 120 may be arranged in such a manner that the two cameras can be placed at adjacent positions, and the first camera 110 and the second camera 120 can be erected at a predetermined distance. The field of view captured by the first camera 110 needs to cover at least part of the field of view captured by the second camera 120. Taking the same room as an example, the first camera 110 can be placed adjacent to the second camera 120 in an adjacent position, as shown in FIG. 2D. Further, the first camera 110 and the second camera 120 may be disposed at different positions in the room as shown in FIG. 2E.

The control module 130 can be implemented by a personal computer, a network video recorder (DVR), an embedded system, or an electronic device with computing power. The control module 130 has a processing unit 131, a storage unit 132, a first signal input unit 133, a second signal input unit 134, a signal output unit 135, and a display unit 136.

The processing unit 131 is connected to the storage unit 132, the first signal input unit 133, the second signal input unit 134, the signal output unit 135, and the display unit 136. Store The unit 132 records the first image 111, the second image 121, the panoramic image 122, the affine transformation processing program 141 and the linkage integration program 142, and the storage unit 132 can also selectively store a motion detection processing program or object. Object detection program. The first signal input unit 133 is connected to the first camera 110 and the second camera 120 for receiving the first image 111, the second image 121 and the panoramic image 122. The number of connection terminals of the first signal input unit 133 is not limited, and only one single one is not limited by the number in the first drawing.

The second signal input unit 134 receives an operation command for selecting a viewing target. In the present invention, an object to be photographed is clearly defined, so it is defined as a viewing target. The operation command is manually issued by the user or automatically issued by the control module itself according to the result of motion detection or object detection or by other external electronic computing devices. The signal output unit 135 is connected to the second camera 120, and the signal output unit 135 issues a drive command for adjusting the shooting field to the second camera 120. Although the signal output unit 135 and the first signal input unit 133 are divided into two independent elements in the present invention, they can be combined according to the content of the present invention. Similarly, although the signal output unit 135 and the second signal input unit 134 are divided into two separate elements in the present invention, they can be combined in accordance with the teachings of the present invention. The display unit 136 is configured to play the first image 111 and the second image 121 to provide user viewing.

After the first camera 110 and the second camera 120 are connected to the control module 130, the processing unit 131 performs a linkage integration program 142 of the camera according to the first image 111 and the second image 121. Please refer to Figure 3A, which is the operational flow of the present invention. Schematic diagram. The multi-camera linkage integration program of the present invention includes the following steps: Step S310: respectively capturing a first image by the first camera, and capturing a second image by the second camera; Step S320: performing a panoramic connection on the second image of different viewing angles, and Obtaining a panoramic image; step S330: respectively selecting at least three reference points of corresponding positions from the first image and the panoramic image; step S340: selecting a viewing target from the first image; and step S350: selecting three closest to the viewing target Performing an affine transformation parameter calculation operation of the first image on the second image by the reference point and obtaining a conversion parameter; Step S360: performing affine transformation processing according to the conversion parameter to calculate a position of the viewing target in the panoramic image; and step S370: adjusting the second The field of view taken by the camera causes the second camera to capture the viewing target.

First, the first image 111 is captured by the first camera 110, and the second camera 121 is captured by the second camera 121. The second camera 120 performs a panoramic connection on the second image 121 at different viewing angles by using a conventional algorithm (such as panoramic stitching) to obtain the panoramic image 122. Please refer to the 3B, 3C, and 3D drawings. It is a schematic view of the image joint of a part of the invention. The second images 121 of different viewing angles are respectively taken in the 3B and 3C images, and the viewing angles covered by the second images 121 are adjacent backgrounds. In order to highlight the plurality of second images 121 taken, it is phase The background of the neighbors, therefore, some of the regions in the second image 121 are overlapped. On the left side in FIGS. 3B and 3C are the captured second images 121, that is, images that have not been subjected to deformation processing. The right side in Figures 3B and 3C is a distorted image. The first image 111 is an image taken by a fisheye lens or a wide-angle lens. Therefore, the second image 121 needs to be twisted to perform a panoramic image processing.

The panorama is used to splicing the plurality of second images 121 captured by the second camera 120. In general, the manner of the panorama connection can be performed according to a plurality of sets of feature points in the plurality of second images 121, and the two second images 121 can be docked according to the scene or the object when merging. The second image 121 is subjected to splicing after being twisted, as shown in FIG. 3D. After the aforementioned splicing process, a partial panorama connection can be obtained. Similarly, for other areas in the first image 111, the second camera 120 can take a picture and perform corresponding processing. Finally, the other second image 121 is also subjected to the aforementioned splicing process to obtain the panoramic image 122 of the 3E image.

The control module 130 receives the first image 111 and the panoramic image 122. A reference point of the same position is found in the first image 111 and the panoramic image 122. The method of selecting the reference point can be selected by the user, and the same point position in the first image 111 and the panoramic image 122 can be found through the feature point extraction and matching. Please refer to Figure 4. Therefore, the field of view of the first image 111 of the present invention needs to cover at least part of the field of view of the panoramic image 122. Next, the present invention can select a plurality of reference points from the two images for the simultaneously covered regions from the two images. In general, the number of reference points for the present invention The amount needs at least three or more.

To clearly illustrate the reference points selected in the different images, the set of reference points selected in the first image 111 is defined as the first set of reference points 410; and the set of reference points selected in the panoramic image 122 is the first Two sets of reference points 420. The number of first set of reference points 410 is equal to the number of second set of reference points 420.

Next, the viewing target is selected from the first image. The system can automatically select the viewing target through motion detection or the like or manually select it by the user. The processing unit 131 performs an affine conversion processing program 141 according to the first group of reference points 410 and the second group of reference points 420 for calculating coordinate conversion parameters between the first image 111 and the second image 121. The affine transformation processing of the present invention operates according to the following formula:

Where (X _i , Y _i ) is the coordinate of the reference point of the first image 111, and (X _o , Y _o ) is the coordinate of the reference point of the second image 121, (a ₁ , a ₂ , a ₃ , b ₁ , b ₂ , b ₃ ) are the conversion parameters.

The processing unit 131 selects three reference points adjacent to the viewing target, which are the three reference points having the shortest distance from the viewing target in the embodiment, but are not limited thereto. The closer the reference point is selected, the more fully the rendered image will be rendered. Conversely, if the selection distance of the reference point is farther, the presentation of the picture may have a large error. The processing unit 131 brings the three sets of reference points 410 and the three vertex coordinates of the second set of reference points 420 into Equation 1 of the affine transformation. In this way, the processing unit 131 can obtain six yuan once. equation set. The processing unit 131 solves all (six) conversion parameters according to the equation. The processing unit 131 calculates a viewing target position mapping of the first image to a corresponding position of the panoramic image 122 according to the conversion parameters. And the first image in Fig. 4 selects the viewing target (indicated by a star symbol). Please refer to FIG. 4, which is a schematic diagram of reference point selection of the first image and the panoramic image of the present invention. The processing unit 131 selects three reference points from the first image 111 and the panoramic image 122 (circled in FIG. 4, and the corresponding pixel positions are indicated by broken line segments).

The processing unit 131 calculates a corresponding position of the viewing target in the second movie according to the conversion information. Next, the processing unit 131 adjusts the photographing field of view viewed by the second camera 120 based on the aforementioned calculation result. In other words, the processing unit 131 adjusts the shooting field of the second camera 120 to be centered on the viewing target.

The viewer only needs to select any viewing target from the first image 111. If the target area has corresponding conversion information, the control module 130 will adjust the second camera 120 to capture the viewing target. Alternatively, when the motion detection program is executed by the processing unit 131, if any image object movement in the first image 111 is detected, the processing unit 131 determines that the detected object is located at the position of the first image 111 and determines Whether the location of the object has the corresponding conversion information. If there is conversion information in the position of the object, the processing unit 131 drives the second camera 120 to view.

The integrated processing system and method of the multi-camera proposed by the invention are applied to a plurality of cameras with different shooting fields, so that multiple cameras can synchronously shoot the same viewing target.

Although the present invention has been disclosed above in the foregoing preferred embodiments, it is not intended to be limiting. In the present invention, the scope of the patent protection of the present invention is defined by the scope of the patent application attached to the present specification, without departing from the spirit and scope of the present invention. quasi.

110‧‧‧First camera

111‧‧‧ first image

120‧‧‧Second camera

121‧‧‧Second image

122‧‧‧ panoramic image

130‧‧‧Control Module

131‧‧‧Processing unit

132‧‧‧ storage unit

133‧‧‧first signal input unit

134‧‧‧second signal input unit

135‧‧‧Signal output unit

136‧‧‧Display unit

141‧‧‧Affine conversion processing program

142‧‧‧ linkage integration procedure

410‧‧‧First set of reference points

420‧‧‧Second reference point

Figure 1 is a schematic diagram of the architecture of the present invention.

Figure 2A is a schematic diagram of the first image of the present invention.

Figure 2B is a schematic diagram of the second image of the present invention.

2C is a schematic diagram showing the result of the second image of the present invention.

The 2D drawing is a schematic diagram of the erection of the first camera and the second camera of the present invention.

Fig. 2E is a schematic view showing another erection of the first camera and the second camera of the present invention.

Figure 3A is a schematic diagram of the operational flow of the present invention.

Figure 3B is a schematic diagram of the connection process of a portion of the second image of the present invention.

Figure 3C is a schematic diagram showing the result of the connection of the second image of the present invention.

The 3D figure is a schematic diagram showing the result of the connection of the second image of the present invention.

Figure 3E is a schematic diagram showing the result of the second image of the present invention.

Figure 4 is a schematic diagram of reference point selection for the first image and the second image of the present invention.

110‧‧‧First camera

120‧‧‧Second camera

130‧‧‧Control Module

131‧‧‧Processing unit

132‧‧‧ storage unit

133‧‧‧first signal input unit

134‧‧‧second signal input unit

135‧‧‧Signal output unit

136‧‧‧Display unit

141‧‧‧Affine conversion processing program

142‧‧‧ linkage integration procedure

Claims (11)

A multi-camera integrated processing system includes: a first camera for capturing a first image; a second camera for capturing a second image, the first camera being electrically coupled to the second camera The shooting field of the first image at least partially covers the field of view of the second image; and a control module, the control module receives the first image and the second image, and the control module receives the first image from the first image Selecting a reference point of at least three corresponding positions in the second image, the control module selects a viewing target from the first image, and the control module calculates a set of coordinates according to the viewing target and the plurality of the reference points. Converting parameters, performing a target conversion according to the set of coordinate conversion parameters and calculating a position of the viewing target in the second image, the control module adjusting a field of view taken by the second camera to capture the viewing target. The multi-camera integrated processing system of claim 1, wherein the coordinate conversion is an affine transformation, and the operation is performed according to the following formula: X _i , Y _{i are} coordinates of the reference point of the first image, and X _o , Y _{o are} coordinates of the reference point of the second image, a ₁ , a ₂ , a ₃ , b ₁ , b ₂ , b _{3 is} the coordinate conversion parameter. The multi-camera integrated processing system of claim 1, wherein the first camera is a fisheye camera or a panoramic camera or a wide-angle camera. The multi-camera integrated processing system of claim 1, wherein the second camera is a pan-tilt-zoom (PTZ) or a camera having a digital zoom function. The multi-camera integrated processing system of claim 1, wherein the second camera performs panoramic mapping on the second image of the plurality of different field of view, and generates a panoramic image, and outputs the panoramic image as the first image. Two images. The multi-camera integrated processing system of claim 1, wherein the control module determines the viewing target according to a result of motion detection, an outcome of an object detection process, or a user. The multi-camera integrated processing system of claim 1, wherein the plurality of reference points are three of the reference points having the shortest distance from the viewing target. An integrated processing method for a multi-camera includes the steps of: capturing a first image by a first camera and a second image by a second camera, and the field of view of the first image at least partially covers the second image a field of view of the image; selecting a reference point of at least three corresponding positions from the first image and the second image; selecting a viewing target from the first image; and selecting a plurality of the reference points to calculate a set of coordinate conversion a parameter; performing a label conversion process according to the set of coordinate conversion parameters to calculate a position of the viewing target in the second image; and adjusting a field of view taken by the second camera, so that the second camera is used for shooting The viewing target. The method of processing the multi-camera according to claim 8, wherein the step of capturing the second image comprises: the second camera panoramicly connecting the second images of the plurality of different shooting fields to generate a panoramic view The image is output as the second image by the panoramic image. The multi-camera integrated processing method according to claim 8, wherein the coordinate conversion is an affine transformation, and the operation is performed according to the following formula: Wherein, X _i and Y _{i are} coordinates of pixels of the first image, and X _o and Y _{o are} coordinates of pixels of the second image, and a ₁ , a ₂ , a ₃ , b ₁ , b ₂ , and b ₃ are The coordinate conversion parameter. The multi-camera integrated processing method of claim 8, wherein the selected plurality of reference points are three reference points having the shortest distance from the viewing target.