JP2015084512A - Image processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing system having an image processing apparatus which synthesizes images of a plurality of cameras, and provides an image required by a user for each scene, by properly switching image processing control using one or different parameters, thereby capturing a high-quality image desired by the user.SOLUTION: An image processing system includes: multiple pieces of image input means 2, 3 to be used for inputting images from a plurality of camera devices; multiple pieces of image processing means 4, 5 performing image processing on the images input from the input means 2, 3; synthesizing means 6 which synthesizes the images input from the image processing means 4, 5; detection means 7 which recognizes whether a seam part of the images synthesized by the synthesizing means 6 includes an important portion or not; and parameter determination means 8 which generates an operation parameter from the detection means 7. The parameter determination means 8 determines a parameter, on the basis of a result of the detection means 7.

Description

  The present invention relates to an image processing system that combines a plurality of images into one image.

  In recent years, there are many opportunities to use video conference systems capable of sharing information not only by voice but also by video in remote conferences. In addition, when providing video, not only a part of the four walls is displayed by one camera device, but also by using a plurality of camera devices, it is possible to provide an omnidirectional video such as a panoramic photo. It is being considered.

  An important point in the conference system is that the portion that the user is paying attention to (for example, the whiteboard or the facial expressions of the conference attendees) is clearly displayed, and the entire screen does not necessarily have to be clearly displayed.

  In general, as a technology related to image quality improvement, a technology that automatically determines the exposure (luminance) for a specific subject using the auto exposure function and a technology that automatically determines the color balance using the auto white balance function are known. Yes.

  In Patent Document 1, in a plurality of in-vehicle camera devices mounted on a vehicle, (1) the balance of brightness and color varies depending on the position where the vehicle is disposed and the subject to be imaged, and (2) the above In contrast to (1), if the exposure time and gain are controlled at the same level for each image, the difference in brightness and color balance between the images can be absorbed, but it should have been visible if it was originally controlled separately. (3) In contrast to the above (1), when each image is synthesized, there is a problem that a difference in brightness and color balance is noticeable at the joint.

  In order to be able to adjust the brightness and color of the adjacent camera at the time of image composition regardless of the mounting position of each camera device, multiple camera devices are connected by a camera control unit and arbitrarily selected A configuration is disclosed in which an image signal obtained by a first camera device is transferred as reference data to a second camera device, signal processing is performed in the second camera device to generate a corrected image, and a composite image is output. Has been.

  However, in the camera system that combines images via the control unit disclosed in Patent Document 1 to improve the image quality of the combined image, only the mobile camera system (in-vehicle camera system) is mentioned, and the user in the conference room is referred to. There is no mention of optimizing the required image information quality.

  When combining images taken by multiple cameras into a single image, if the parameters set for multiple cameras are shared, the joints between the images will be connected beautifully, but the optimal parameters for each camera. It may disappear. On the other hand, if individual parameters are set for each camera, an optimal image will be obtained for each camera, but the seam between the images may become unnatural, and the problem of how to switch these controls was there.

  The present invention is for solving the above-mentioned problems, and an object of the present invention is to perform image processing with one parameter and image processing with individual parameters in an image processing system that combines images from a plurality of cameras. It is to provide an image processing system that captures the image that the user wants to see more beautifully by providing a video that the user needs for each scene by appropriately switching the control to perform the above.

  In order to achieve this object, the present invention has the following features.

  An image processing system according to the present invention includes a plurality of image input means for inputting images from a plurality of camera devices, a plurality of image processing means for performing image processing on images input from the plurality of image input means, A synthesizing unit that synthesizes a plurality of images input from the plurality of image processing units; a detecting unit that recognizes whether there is an important seam portion of the image synthesized by the synthesizing unit; and an operation parameter from the detecting unit Parameter determining means for generating the parameter, wherein the parameter determining means determines the parameter based on the result of the detecting means.

  According to the present invention, in an image processing apparatus that synthesizes images from a plurality of cameras, a user needs to change scene control by appropriately switching between performing image processing with one parameter and controlling image processing with individual parameters. By providing a video image, the user can see what he / she wants to see more beautifully.

1 is a block diagram illustrating an overall configuration of an image processing system according to a first embodiment. It is the schematic of the image image image | photographed and synthesize | combined by the image processing system. It is a flowchart which shows the parameter determination set to each image processing means and an input means. It is a figure explaining determination of a parameter. It is a figure explaining use of an intermediate setting about determination of a parameter. It is a block diagram which shows the whole structure of the image processing system which concerns on 2nd Embodiment. It is the schematic which shows arrangement | positioning of a human sensitive sensor.

  Hereinafter, the present embodiment will be described in detail with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram showing the overall configuration of the image processing system according to the first embodiment. The image processing system 1 includes two input units 2 and 3, two image processing units 4 and 5, a synthesis unit 6, a detection unit 7, a parameter determination unit 8, and a control unit 9.

  The input means 2 and 3 are configured to include an optical system such as a lens and an optical filter, a sensor, and the like. The image formed by the lens is input to the sensor, converted into digital data, and the subsequent image processing means 4 Enter 5. By using a wide-angle lens as the lens, a wide range of images can be taken with one input means. Also, sensor setting (gain, exposure time, etc.) and lens control settings (aperture, focus, etc.) are set by the parameter determination means 8.

  The input image data is input to the image processing means 4 and 5. Image processing is performed on the images input from the input means 2 and 3, such as lens characteristic correction processing (shading correction, chromatic aberration correction, etc.) and sensor characteristic correction processing (gamma correction, defective pixel correction, etc.). ), Parameters for performing processing for improving image quality (noise removal, edge enhancement, color correction, dynamic range expansion, etc.) and image processing are set by the parameter determination means 8.

  The image processing means 4 and 5 also have a function of acquiring image brightness, white balance, and MTF in order to perform AWB, AE, AF, and the like. A plurality of image data input from the image processing means 4 and 5 are combined into a single image by the combining means 6.

The detection unit 7 performs recognition processing on one image synthesized by the synthesis unit 6 and determines what information is present at the joint portion of the image. The parameter determination unit 8 determines parameters to be set in the image processing units 4 and 5 and the input units 2 and 3 based on the result of the detection unit 7.
In the present embodiment, the input unit and the image processing unit have two configurations, but the input unit and the image processing unit may include three or more units.

When it is reported that the important thing is reflected, the control means 9 determines the scene by looking at the images given from the image processing means 4 and 5, and sets each scene parameter based on the scene characteristics of each camera. It is determined whether they are the same or different and is given to the parameter determining means 8.
Image data is sent from the image processing means 4, 5 to the control means 9. The detection unit 7 determines whether or not an important image is captured at the joint of the images, and sends the determination result to the control unit 9.

  FIG. 2 is a schematic view of an image image synthesized by the image processing system. Depending on the use case in the image processing system in which a plurality of images are combined, the importance may be different for each position in the combined 360 degree panoramic image. For example, in a TV conference system, the faces of attendees, whiteboards, screens, etc. are very important, and the other parts are less important.

As shown in FIG. 2A, the attendee A and the attendee B are shown in the shooting range of the first input means 2, and the whiteboard is shown in the shooting range of the second input means 3.
When it is determined that important things are not shown at the joint between the images synthesized and the scene characteristics captured by the image input means are different between the two images to be synthesized, the first input means 2. The parameters set for the first image processing means 4 and the parameters set for the second input means 3 and the second image processing means 5 are respectively set to optimum values for each image.

  As a result, there is a difference between the images (for example, brightness, white balance, etc.) in the joint portion, so the image of the joint portion may become unnatural, but it has been optimized for each scene. Since the parameter is set in the input means image processing means, information important to the user can be provided with high image quality.

  As shown in FIG. 2B, the attendee A and the attendee B are shown in the shooting range of the first input means 2, and the shooting range of the first input means 2 and the shooting range of the second input means 3 are A whiteboard is visible at the joint.

  Further, as shown in FIG. 2C, the attendee A is shown in the shooting range of the first input means 2, and the whiteboard is shown in the shooting range of the second input means 3. The attendee B is shown at the joint between the shooting range of the first input means 2 and the shooting range of the second input means 3.

  If it is determined that an important image is present at the joint of the plurality of synthesized images, the parameters to be set in the first input means 2, the first image processing means 4, the second input means 3, the second The parameters to be set in the image processing means 5 are parameters common to the input means and the image processing means. The common parameter is set by averaging parameters optimized in a scene of a plurality of input means.

  As a result, each image is not optimized, but the joint portion can be smoothly connected, and the subject reflected in the joint portion can be naturally projected.

  FIG. 3 is a flowchart showing parameter determination to be set for each image processing unit and input unit. It is determined whether the features of the scenes photographed by the input means 2 and 3 are the same or different (step S1). If the characteristics of the scenes photographed by the input means 2 and 3 are the same (step S1, Yes), common parameters are set by the input means 2 and 3 and the image processing means 4 and 5 (step S4).

  When the features of the scenes photographed by the input means 2 and 3 are different (No in step S1), it is determined whether important items are shown in the joint portion of the input image (step S2). When it is determined that important items are not shown in the seam portion of the input image (No in step S2), optimized parameters are individually set by the input units 2 and 3 and the image processing units 4 and 5 (step S2). S3). When it is determined that an important image is reflected in the joint portion of the input image (step S2, Yes), common parameters are set in the input means 2 and 3 and the image processing means 4 and 5 (step S4).

  Examples of the scene characteristics determined in step S1 include brightness and color temperature. For example, with respect to the brightness, when the light source is on only one side of the area photographed by the input means 2 and 3, one is a bright image and the other is a dark image. Alternatively, when there are a plurality of light sources such as fluorescent lamps and sunlight, the brightness level of the captured image may be different for each input means.

  FIG. 4 is a diagram for explaining determination of parameters. If brightness is an example of a scene feature to be determined, a gain value may be cited as a parameter that needs to be determined. The gain value exists as a parameter of the sensors of the input means 2 and 3 and the image processing means 4 and 5. In general, when the luminance is low (when dark), the gain is set to a large value in order to capture a bright image. Conversely, when the luminance is high (bright), the gain is set low in order to prevent the image from being overexposed.

  When the input means 2 side is dark and the input means 3 side is bright, and when a common parameter is set in each input means 2 and 3, an intermediate value of each individual optimum parameter is obtained and used as a common parameter.

  FIG. 5 is a diagram illustrating an embodiment in which an intermediate setting is used instead of individual optimization or common two options for parameter determination. In the present embodiment, the parameters set in the input units 2 and 3 and the image processing units 4 and 5 are changed instead of determining the parameters by two options as shown in FIG.

  A determination result as to whether or not an important thing is reflected in the joint portion of the image is calculated not by binary determination but by a numerical value, and a parameter correction value is determined based on the value. The gain amount correction value X is expressed by Equation (1).

  X is a correction value of gain amount, α is a proportional coefficient, I is a determination result of whether or not an important thing is reflected in the joint portion of the image (I is large → important is reflected, I is small → Important things are not shown). The importance level I may be a probability that an important thing is reflected.

In addition, when the importance of what is shown at the joint is the maximum, the relationship between the optimum values of the input means 2 and 3 is set to the relationship shown in the following equation (2). ΔP is the optimum parameter difference value for each input means, and I _MAX is the maximum value of I.

(Second Embodiment)
FIG. 6 is a block diagram showing the overall configuration of an image processing system in which important information is determined by the information acquisition unit 11. In the second embodiment, information acquisition means 11 is added. The same configurations as those of the first embodiment are denoted by the same reference numerals and description thereof is omitted. In the first embodiment, the image synthesized by the synthesizing unit 6 is used to determine whether the detection unit 7 shows what is important for the user at the joint portion.

  However, in the second embodiment, based on the peripheral information of the image processing system acquired by the information acquisition unit 11 in addition to the determination by the image, the detection unit 7 is important for the joint portion synthesized by the synthesis unit 6. It is determined whether an object is captured, and the parameter determination unit 8 determines parameters to be set in the input units 2 and 3 and the image processing units 4 and 5. In addition to the information from the image, it is possible to improve the accuracy of the determination by determining whether or not an important thing is reflected in the joint portion of the image using the information from the information acquisition unit 11.

  FIG. 7 is a schematic diagram showing the arrangement of the human sensors 21 and 22. Specifically, the information acquisition means 11 includes human sensors 21 and 22. This is a system in which two input means 23 and 24 each having a lens having an angle of view of 180 degrees or more are combined. A region where the two input units 23 and 24 are overlapped to form an image is a joint portion 25 of the image.

  By arranging the human sensors 21 and 22 in the direction to become the joint portion 25 of the image, the information of the human sensors 21 and 22 is used to determine whether or not there is a person at the joint portion 25, and the determination accuracy is improved. The determination accuracy can be improved as compared with the case where only image recognition is performed. The arrows in the figure indicate the sensing directions of the human sensors 21 and 22.

  In addition, an embodiment in which determination is made only by the human sensors 21 and 22 without using recognition of the composite image is possible. In addition, a person who is talking may be detected using a microphone array instead of a human sensor.

  According to the present embodiment, when a thing that is highly necessary for the user is reflected in a joint portion of images taken by a plurality of cameras, when a highly necessary portion is reflected in the joint portion, 1 When operating in a mode that performs image processing with one parameter to improve the image quality of the seam, and if there is no necessity in the seam part, image processing is performed with individual parameters to improve the quality of each image. Optimize.

  Therefore, in an image processing device that synthesizes images from multiple cameras, it provides video that the user needs for each scene by appropriately switching between image processing with one parameter and control that performs image processing with individual parameters. By doing so, the user can see what he / she wants to see more clearly.

  The above-described embodiment shows an example of a preferred embodiment of the present invention, and the present invention is not limited thereto, and various modifications can be made without departing from the scope of the invention. is there.

DESCRIPTION OF SYMBOLS 1 Image processing system 2 Input means 4 Image processing means 6 Synthesis | combination means 7 Detection means 8 Parameter determination means 9 Control means 11 Information acquisition means 21 Human sensor 25 Joint part

Japanese Patent No. 4325642

Claims (6)

A plurality of image input means for inputting images from a plurality of camera devices;
A plurality of image processing means for performing image processing on images input from the plurality of image input means;
Combining means for combining a plurality of images inputted from the plurality of image processing means;
Detection means for recognizing whether there is an important seam portion of the image synthesized by the synthesis means;
Parameter determining means for generating operating parameters from the detecting means,
The parameter determining means determines a parameter based on the result of the detecting means. Having one or more information acquisition means;
The image processing according to claim 1, wherein the detection unit uses the information acquired by the information acquisition unit for detection of whether there is an important seam portion of the image combined by the combining unit. system.   The detection means uses individually optimized parameters when there is nothing important in the seam portion of the image, and uses an intermediate value of the individually optimized parameters when there is something important in the seam portion of the image. The image processing system according to claim 1, wherein:   The detection means is a parameter obtained by individually optimizing the parameters of the plurality of image processing means and the plurality of image input means, and an intermediate value between the individually optimized parameters according to the importance of the joint portion of the image The image processing system according to claim 1, wherein the image processing system is switched between.   The image processing system according to claim 2, wherein the information acquisition unit is a human sensor.   The image processing system according to claim 2, wherein the information acquisition unit is a microphone array.

Images