TWI542194B - Three-dimensional image processing system, apparatus and method for the same - Google Patents

Description

Stereo image processing system, device and method

The present invention relates to a stereoscopic image processing system, apparatus and method, and more particularly to a stereoscopic image processing system, apparatus and method constructed by a plurality of cameras disposed in a space.

Stereoscopic image processing technology has become more and more mature with the development of hardware and technology. Stereoscopic images can not only provide a sense of space that cannot be replaced by flat images, but also provide better observation in many fields. Monitoring function.

For example, when applied to monitoring, stereo images will provide more accurate and more angled monitoring images. One of the ways in which the conventional technology produces a stereoscopic image includes a camera that uses two different shooting angles, such as a multifunctional portable stereoscopic monitoring system disclosed in the Republic of China Patent No. 364715 (Announcement Date: 1999/07/11). In addition to discussing the mobile device in the case, the method for generating a stereoscopic image includes providing two sets of cameras, two sets of signal converters, and a main body image display. The images captured by the two sets of cameras are processed by the signal converter to generate a stereoscopic image. A stereoscopic image of the image display. Such stereoscopic image presentation is mainly a technique for generating stereoscopic effects by using the left and right eye parallax of the human eye, and devices such as polarized glasses and liquid crystal shutter glasses are used.

In addition, the technical department presents stereoscopic images through image reconstruction. These stereoscopic images mainly capture the same scene through different cameras, and after image processing, The splicing technology reconstructs the images captured by different cameras. Different cameras can complement the images and depth of field of different objects in the image, so that stereoscopic images can be presented. Related technologies such as the Republic of China Patent No. M373507 (Announcement Day 2010/02/ 01) The stereoscopic panoramic splicing device.

The stereo camera used in the above prior art includes a left camera and a right camera, and a signal processor processes the signals taken from the left camera and the right camera, and the obtained image signals overlap, and the signal processor displays the left image with The plurality of stereo images captured by the right image are combined into a panoramic mosaic image. The case also refers to the mathematical model to make the stitched image depth.

A prior art of a monitoring system constructed using a three-dimensional stereoscopic image, such as the Republic of China Patent Publication No. 201136313 (Publication Date: 2011/10/16), in which a dual-lens camera is used to monitor a monitoring scene, and a left-lens camera shooting scene of a dual-lens camera is monitored. The left and right screens can obtain the left and right scene images of the visual depth information, and then use the spatial image processing method to filter out the same image information of the same part or the content close to the left and right scene images, and analyze the difference part. The information is stored according to the time, and the video compression image at the time point is read as needed to reconstruct the three-dimensional digital image frame. The dual-lens camera used in the present invention is a two-lens camera with a two-lens camera, which is used for image capturing of the same scene angle of the monitoring scene through the left and right lenses thereof, and is also a method for establishing a stereoscopic image by applying parallax caused by the distance between the left and right eyes of the human eye.

Different from the prior art technology for generating stereoscopic images, the stereoscopic image processing system, device and method proposed by the present invention specifically proposes to photograph multiple cameras in the same place and the same space, preferably more than two cameras, so that a place can be obtained. The image of multiple angles of the space can be obtained by image analysis to obtain the spatial relationship between the size, depth of field and mutual relationship of various objects in the space, thereby completely reconstructing the stereoscopic image in the space.

According to an embodiment, a stereoscopic image processing system includes a plurality of cameras, each of which is disposed at a plurality of positions, and captures a scene for generating a continuous image for capturing the scene, and a stereoscopic image processing device. The stereoscopic image processing device is connected to a plurality of cameras for acquiring image signals captured by the cameras, and further comprising an input unit for receiving image signals of the cameras, and a stereoscopic image forming unit, the stereoscopic image forming unit is connected to the input unit, and the software is connected through the software. The method analyzes the received image signal, obtains one or more object images in the scene, and obtains the position and contour of each object at each camera shooting angle, and the distance between each object and each camera to reconstruct a stereoscopic image of the object in the scene. .

The embodiment further analyzes the image signal by using the software method, thereby identifying one or more objects in the scene, constructing the outline of each object, judging the distance between each object and each camera, and combining the images taken by the cameras at the same time. It is even possible to judge the moving objects in the scene by comparing the images one time before and after.

In an embodiment of the stereoscopic image processing method, first, a plurality of cameras are captured to capture images of the same scene at different positions, and image processing is performed to obtain positions, contours, and contours of at least one object in the scenes captured by the cameras at different positions. The distance between the object and each camera, and the splicing according to the position, contour and distance information of at least one object in the scene of each camera to form a stereoscopic image with depth of field.

In an embodiment, in the spliced stereoscopic image, the type of the object is recognized by using the features of each object, and the step of determining the moving object in the scene after the image comparison before and after the time is further included.

In another embodiment, a plurality of cameras are provided on a vehicle to respectively capture scenes in different directions, that is, images of scenes outside the vehicle, and obtain the distance of the objects in the scene through image processing, even after image processing. The depth of the object is obtained, and the image with the depth of field in the external scene of the vehicle can also be obtained through splicing.

In order to further understand the technology, method and effect of the present invention in order to achieve the intended purpose, reference should be made to the detailed description and drawings of the present invention. Understand, of course The drawings and the annexed drawings are only for the purpose of illustration and description, and are not intended to limit the invention.

10‧‧‧ Shooting scene

101‧‧‧First camera

102‧‧‧Second camera

103‧‧‧ Third camera

104‧‧‧Fourth camera

11,12,13‧‧‧ objects

20‧‧‧Scenario

201‧‧‧First camera

202‧‧‧Second camera

203‧‧‧ Third camera

204‧‧‧Fourth camera

22‧‧‧ objects

25‧‧‧3D image processing device

30‧‧‧3D image processing device

301‧‧‧Signal Processing Unit

302‧‧‧Signal input unit

303‧‧‧ memory unit

304‧‧‧Communication unit

311, 312, 313, 314‧ ‧ camera

38‧‧‧Handheld devices

32‧‧‧Network

34‧‧‧Cloud Server

41‧‧‧Image capture device

42‧‧‧Input unit

43‧‧‧3D image forming unit

431‧‧‧Contour creation module

432‧‧‧ distance judgment module

433‧‧‧Image splicing module

44‧‧‧Object Identification Unit

45‧‧‧Character extraction unit

46‧‧‧ comparison unit

47‧‧‧Moving detection unit

48‧‧‧Database

49‧‧‧Output unit

50‧‧‧ Vehicles

501‧‧‧First camera

502‧‧‧Second camera

503‧‧‧ Third camera

504‧‧‧Fourth camera

Step S601~S609‧‧‧One of the embodiments of the stereo image processing method

Step S701~S707‧‧‧Three-dimensional image processing method

1 is a schematic view showing an embodiment of a stereoscopic image processing system of the present invention; FIG. 2 is a schematic view showing an implementation of a camera shooting scene of the stereoscopic image processing system of the present invention; FIG. 3 is a schematic view showing an embodiment of a stereoscopic image processing device of the present invention; FIG. 5 is a schematic diagram showing an embodiment of a stereoscopic image processing apparatus of the present invention; FIG. 6 is a flow chart showing an embodiment of a stereoscopic image processing apparatus of the present invention; FIG. The flow of the display describes the second flow of the embodiment of the stereoscopic image processing method of the present invention.

For an embodiment of the stereoscopic image processing system, reference may be made to the schematic diagram of an embodiment of the stereoscopic image processing system of the present invention shown in FIG.

This example shows a shooting scene 10 with multiple cameras in the vicinity, especially a network camera (IP CAM) that can transmit video signals over the network or other camera units with communication capabilities, preferably located in one a plurality of high-resolution cameras in the space, wherein the first camera 101, the second camera 102, the third camera 103, and the fourth camera 104 are schematically displayed, and one or more objects 11, 12 in the shooting scene are respectively acquired. 13 images.

According to an embodiment of the invention, the first camera 101, the second camera 102, the third camera 103, and the fourth camera 104 respectively capture continuous images at different angles, and when setting up the cameras, all objects in the captured scene 10 can be completely considered. 11,12,13 are set at various angles, so that stereoscopic images of objects 11, 12, and 13 in all scenes 10 can be completely obtained through analysis, recognition, and image stitching.

FIG. 2 is a schematic diagram showing the implementation of each camera shooting scene of the stereoscopic image processing system of the present invention.

The schematic display shows a first camera 201, a second camera 202, a third camera 203 and a fourth camera 204. The scene 20 has an object 22, and the system includes a computer-implemented stereo image processing device 25, which can be fully A continuous image captured by the first camera 201, the second camera 202, the third camera 203, and the fourth camera 204, such as a digital video recorder (DVR), is obtained.

The stereoscopic image processing device 25 is connected to a plurality of cameras 201, 202, 203, 204 for acquiring image signals captured by the cameras, and is provided with input terminals for connecting and receiving the plurality of cameras 201, 202, 203, 204. The stereoscopic image forming unit is mainly provided with a plurality of software or firmware modules. For analyzing the received image signal, obtaining one or more objects 22 in the scene 20, obtaining the position and contour of each object 22 at each camera shooting angle, and the distance between each object and each camera to reconstruct the scene 20 A stereoscopic image of one or more objects 22 within.

When the stereo image processing device 25 obtains the image signals captured by the cameras 201, 202, 203, and 204 connected to the line, it can be compressed and stored, and can be used as the data for subsequent interpretation, and can also perform real-time image processing to instantly reflect changes in the scene 20, thereby being able to monitor the use of. In particular, the stereoscopic image processing device 25 of the embodiment of the present invention is a device that can process a three-dimensional image signal. When the cameras 201, 202, 203, and 204 acquire images of different angles in the same scene 20, the image is analyzed by software or firmware. The signal obtains the information of the object 22 in the scene 20, including obtaining the position of the object 22 through the image processing, the distance between the same object 22 and the cameras 201, 202, 203, 204, and recognizing the image contour of the object 22. Further, a stereoscopic image of each object can be calculated based on the information.

Other embodiments may not be limited to a specific indoor place, but an embodiment for stereoscopic outdoor space, such as a plurality of photographs taken at different angles on a vehicle carrier. The video camera and the stereoscopic image processing device (25) loaded thereon can be used with an on-board computer, an in-vehicle display system, etc., and after capturing the scene around the carrier, the image processing technology is used to obtain the contour of each object in a specific spatial range, The relationship between the depth and the object, according to which can construct a three-dimensional image of the surrounding scenery. If used in a vehicle, it can provide a reference for driving to easily judge the surrounding environment; if used for monitoring, clearer image analysis conditions can be obtained.

Fig. 3 is a view showing an embodiment of a circuit unit of the stereoscopic image processing apparatus of the present invention.

The stereoscopic image processing device 30, such as a computer-implemented device, is provided with a signal processing unit 301, a signal input unit 302, a memory unit 303, and a communication unit 304 in accordance with the circuit function. The signal processing unit 301 of the stereoscopic image processing device 30 serves as the core of the operation data, and the image signal generated by the camera 311, 312, 313, 314 is connected and received by the signal input unit 302, wherein the memory unit 303 functions as a stored image signal.

The stereoscopic image processing device 30 is then connected to the external terminal device through the communication unit 304, for example, connected to the handheld device 38 via the illustrated network 32, or connected to the cloud server 34, so that the processed image signal can be immediately processed, especially The formed stereo image, or the compressed or otherwise processed file of the unprocessed image is transmitted to the handheld device 38 for immediate display as a purpose of motion monitoring. When the data is delivered to the cloud server 34, it can be used as a space for storing backups, and can be transmitted to other devices through the cloud server 34.

However, for a terminal device such as the handheld device 38, an associated monitoring or browsing application, such as a mobile application (APP), can be installed in the handheld device 38 when receiving the stereoscopic image signal transmitted from the stereoscopic image processing device 30. After that, the stereo image in the shooting scene can be displayed smoothly.

For the entire stereoscopic image processing system, in which the processing technology is implemented by means of software or hardware, the functional block diagram of the related embodiment can be seen in FIG. 4.

At one end of the system, there is an image capturing device 41, which is arranged in a space or a different position in the scene, and is generally arranged to be fully rendered so that each can be stereoscopically Placed in the position of the object in the scene.

The system is connected to the image capturing device 41 (which may be a plurality of) through the input unit 42. The related image signal is processed by the stereoscopic image forming unit 43 in the system to form a reconstructed stereoscopic image of the object, such as the output through the graphic display. Unit 49 outputs a signal.

The stereoscopic image forming unit 43 mainly obtains the image signals captured by the image capturing devices 41, and then obtains image parameters reflecting various objects in the scene in the signal. For example, the contour building module 431 implemented by the software or the firmware means that the contour of the object is constructed by acquiring image signals of the same object from different angles, and the manner of presentation can represent a contour by data parameters; and the distance determining module 432 obtains each object and The distance parameter of each image capturing device 41, the distance parameter of the same object obtained from the different image capturing device 41, can determine the depth of field of each part of each object.

The images acquired by the adjacent different image capturing devices 41 in a scene may overlap, so that the overlapping of the images captured by each image capturing device 41 can be obtained by the image processing technology. The image splicing module 433 is implemented by using a software splicing module 433 in the system to combine different images taken by the adjacent image capturing devices 41 at the same time. The complete image of the scene, and then the stereoscopic image reconstructed by the image signals of the objects obtained by the different image capturing devices 41, achieves the purpose of forming a stereoscopic image.

After the stereoscopic image reconstruction, the system further includes an object recognition unit 44, thereby identifying various objects in the scene, and the number of one or more objects and the position of the display; and the feature extraction unit in the system 45 After the data analysis, the characteristics of the object are obtained to obtain the object, for example, according to the appearance form of the object, it is determined which object is the object.

The system can further accurately obtain the detailed features of the object according to the characteristic parameters of various objects recorded in the database 48 through the comparison unit 46, for example, according to the stereoscopic The object features in the image determine the face, and even the relevant person's data is obtained according to the face recognition, which can be used for monitoring or access control.

It is worth mentioning that if it is to be used for monitoring or access control, the identification of the entry and exit personnel is the main purpose. Therefore, in this database 48, the facial image of the entering and leaving personnel can be recorded; or some preservation database can be combined as an illegal invasion. Judge.

The system can determine the moving objects in the scene through the motion detection unit 47 and compare the images before and after the time. This part can obtain the changed image through the comparison of frames in the continuous image in the front and back time, and obtain the moving object and the related moving track from the change of the time. Finally, the output is output via the output unit 49, including the above-mentioned output to the monitoring device of the terminal, or uploaded to the cloud system for other use.

As an example, the database 48 may be a database that links a community network or a specific user database to obtain characteristics of the users therein. After the system obtains the image of the person in the shooting scene, the identity of the person can be identified through the comparison of the database 48, thereby achieving the functions of monitoring, preservation and deterrence. For example, the database 48 using Google (Google ^TM), Facebook (Facebook ^TM), when the introduction of such a huge amount of information can, through facial features than to identify their status.

5 is a schematic diagram showing an embodiment of a stereoscopic image processing device of the present invention applied to a vehicle, that is, the system formed by the plurality of cameras can be applied to obtain an image of an external scene of the vehicle, and obtain various objects in the scene (such as buildings, other vehicles, Pedestrian) video information, the overlapping parts of different camera images can also obtain the depth information of the part. The image processing technology can still be used to obtain the distance between each object and the vehicle, the depth of the object, etc. . For example, the depth of field information of the object can be analyzed according to the light and shadow of the object in the scene, the positional relationship of the objects, and the like.

In this example, a plurality of image capturing devices are disposed on a vehicle 50 to respectively capture scenes in different directions, such as a first camera 501 and a second camera that respectively capture a foreground image of the vehicle, a scene of the left side of the vehicle, a scene of the rear of the vehicle, and a scene of the right side of the vehicle. The camera 502, the third camera 503, and the fourth camera 504.

Through the software and hardware devices as described in FIG. 4 above, image acquisition, processing, and splicing can be performed, for example, image processing is used to obtain the distance of objects in the scene, and even image processing is used to obtain object depth, which can also be obtained through splicing. An image with depth of field in the exterior scene of the vehicle. Similarly, one or more objects in the scene can be judged by image processing, and the position and contour of each object in the shooting angle of each image capturing device, and the distance between each object and each image capturing device are obtained, thereby forming a depth of field. The stereoscopic image will provide the driver with a view of the surrounding environment, obtain images of one or more objects in the scene, and provide software to determine abnormal conditions.

Figure 6 is a flow chart showing the flow of an embodiment of the stereoscopic image processing method of the present invention.

The system shown in FIG. 1 and FIG. 2 includes a first camera, a second camera, a third camera, and a fourth camera disposed at different positions, which may be respectively disposed in four corners of a certain scene, or may be located in a vehicle or A specific scene is shot on a specific object, and the number of cameras can be changed according to the actual scene or the design of the object. In principle, the design of each corner of the scene can be smoothly obtained.

The flow display shows that, in step S601, the system obtains, by the stereoscopic image processing device, a plurality of cameras to capture images of the same scene at different positions; and then, the stereoscopic image processing device performs image processing through the software or firmware module, wherein The image signals captured by different cameras at different angles can at least determine the contour of each object and the distance between the cameras.

In step S603, the stereoscopic image forming unit implemented by the software or the firmware in the stereoscopic image processing device can be obtained from the cameras (such as the first camera, the second camera, the third camera, and the fourth camera) through the software method. The image signal identifies one or more objects in the scene through the object recognition function (object recognition unit), constructs the contour of each object (such as through the contour creation module), and determines the distance between each object and each camera (such as the transmission distance) Judgment module).

The embodiment shows that an object can be photographed by at least two cameras, so that a different angle of the object can be obtained, the depth (or width) of the object can be obtained according to the contours drawn, and various distance data, and through various Depth of field The data determines the positional relationship between the objects, so that the spatial relationship of various objects of the entire scene can be obtained to facilitate image mosaic. For example, the image splicing function in the stereoscopic image processing device (image splicing module) combines the images captured by the at least two cameras at the same time, wherein the method mainly obtains an environment image in a space to form an instant stereo. The image includes details of the edge, size, and distance of the objects in the space. The images are spliced for each item in the effective illumination range of each camera in multiple images. According to the distance and size, a stereoscopic image with depth of field is constructed according to the spatial relationship of each object. , as in step S605.

In addition to storing the stereoscopic image formed at each time point for future use, the image processing system can be used to identify the object in real time or afterwards, such as step S607, such as first identifying the feature of the object, comparing the database. Get the object type. Finally, the result is output as in step S609.

In the process of forming a stereoscopic image, as shown in FIG. 7, the contour, the distance and the size of at least each object in the scene are analyzed and determined through the image processing, as in step S701; the embodiment may be added to expose the images captured by the cameras. The analysis is performed by the function of the feature analysis, and the feature of each object is obtained, as in step S703. The comparison database is obtained, and the identification result is obtained and output, as in step S707. In addition, the information of the moving object can be obtained by matching the image before and after.

The output result can be transmitted to a user device along with the identified object type, size, position, movement, etc., for example, the user device is installed with an application capable of accessing the stereoscopic image processing device, and is instantaneously obtained by streaming technology. Image signal.

In summary, in order to obtain a spatial relationship between various objects in a space or a scene, the stereoscopic image processing system of the present invention includes a plurality of image capturing devices disposed in the space, and a device for processing image signals, wherein Through the software module to analyze image signals, identify objects, obtain the position, contour, distance and other information of each object, you can combine the images captured by each camera at the same time through image stitching technology. Thereby, it is possible to continue to identify the object features and to determine the moving objects in the scene. Pieces. In particular, in addition to constructing a stereoscopic image for an indoor space, the invention does not exclude that a stereoscopic image can be constructed for an outdoor scene, including the position, size, and spatial relationship of the object, and further, the moving route of the object can be obtained.

However, the above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Therefore, equivalent structural changes that are made by using the specification and the contents of the present invention are equally included in the present invention. Within the scope, it is combined with Chen Ming.

20‧‧‧Scenario

201‧‧‧First camera

202‧‧‧Second camera

203‧‧‧ Third camera

204‧‧‧Fourth camera

22‧‧‧ objects

25‧‧‧3D image processing device

Claims (13)

A stereoscopic image processing system includes: a plurality of image capturing devices respectively disposed at a plurality of positions in different directions, shooting a scene for generating a continuous image for capturing the scene; and a stereoscopic image processing device connecting the plurality of The image capturing device is configured to obtain an image signal captured by each image capturing device, comprising: an input unit for receiving image signals captured by each image capturing device; a stereoscopic image forming unit connected to the input unit for analyzing the received Image signal, obtain one or more object images in the scene, obtain the position and contour of each object at the shooting angle of each image capturing device, and the distance between each object and each image capturing device to reconstruct the scene. A stereoscopic image of one or more objects; wherein the stereoscopic image forming unit analyzes the image signal through a software method, identifies the one or more objects in the scene through an object recognition unit, and constructs a contour of each object through a contour building module Through a distance determination module to determine the distance between each object and each image capturing device, and through a Splicing module as combinations of the image capturing apparatus image taken at the same time. The stereoscopic image processing system of claim 1, further comprising a feature extraction unit for analyzing characteristics of each object, and comparing with a database to determine the type of each object. The stereoscopic image processing system of claim 2 further includes a motion detecting unit that compares the images before and after the time to determine the moving object in the scene. The stereoscopic image processing system of claim 1, wherein the stereoscopic image processing device further comprises a communication unit for outputting a stereoscopic image signal. The stereoscopic image processing system of claim 1, wherein the system comprises a first camera, a second camera, a third camera and a fourth camera disposed at different positions, and the stereo image forming unit analyzes through a software method. Since the first photo The image signal obtained by the camera, the second camera, the third camera and the fourth camera image identifies the one or more objects in the scene through an object recognition unit, and constructs each object through a contour building module. The contour is determined by the distance determining module to determine the distance between each object and each camera, and the first camera, the second camera, the third camera, and the fourth camera image are combined at the same time through an image mosaic module. Image. A stereoscopic image processing device is connected to a plurality of image capturing devices for acquiring image signals captured by each image capturing device, comprising: an input unit for receiving image signals captured by each image capturing device; and a stereoscopic image forming unit; Connecting the input unit, analyzing the received image signal, obtaining one or more object images in the scene, obtaining the position and contour of each object in the shooting angle of each image capturing device, and the objects and the image capturing devices a stereoscopic image of the one or more objects in the scene; wherein the stereoscopic image forming unit analyzes the image signal through a software method, and identifies the one or more objects in the scene through an object recognition unit, through a contour The module is configured to construct the contour of each object, determine the distance between each object and each image capturing device through a distance determining module, and combine the image captured by the image capturing device at the same time through an image mosaic module. The stereoscopic image processing device of claim 6, further comprising a feature extracting unit configured to analyze features of the objects and compare the types of the objects with a database. The stereoscopic image processing device of claim 7, further comprising a motion detecting unit that compares the images before and after the time to determine the moving object in the scene. The stereoscopic image processing device of claim 6, further comprising a communication unit for outputting a stereoscopic image signal. A stereo image processing method includes: Acquiring a plurality of image capturing devices to capture images of the same scene at different positions; and acquiring, by the software method, an object recognition unit and a contour creation module to obtain at least one object in the scene captured by the image capturing device at different positions The position, the contour, and the distance between each object and each image capturing device are obtained through a distance determining module; and the position, contour and distance information of at least one object in the scene is transmitted through an image mosaic module according to each image capturing device The images captured by the image capturing devices at the same time are combined to form a stereoscopic image with depth of field. The method of processing a stereoscopic image according to claim 10, further comprising the steps of: obtaining the at least one object feature, and identifying the at least one object based on the feature of the at least one object and comparing the one library. The stereoscopic image processing method of claim 11, wherein after the type of the at least one object is recognized, the result is output to a user device. The method of processing a stereoscopic image according to claim 10, further comprising the step of determining a moving object in the scene after comparing the images before and after a time.