JP2009069114A - Stereo image adjusting method, stereo image adjusting system, and stereo image adjusting program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To improve the accuracy and stability of a stereo image without multiplying the error between images constituting the stereo image. <P>SOLUTION: A main camera of a camera 10 photographs a main image constituting the stereo image. A reference camera of the camera 10 photographs a reference image constituting the stereo image. A differential image creating section 216 detects the difference in coordinate value between the main image and the reference image and creates the differential image. An error calculation section 217 calculates the correction values of coordinate values of the main image and the reference image based on the differential image. A partially cutting section 205 extracts a partial image from the main image based on the shift value calculated by a camera center detection section 202. A partially cutting section 215 extracts a partial image from the reference image based on the correction value calculated by the error calculation section 217. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a stereo image adjustment method, a stereo image adjustment system, and a stereo image adjustment program for adjusting a stereo camera so that a floor surface can be photographed without tilting from a ceiling, and more particularly, after adjusting the camera, The present invention relates to a stereo image adjustment method, a stereo image adjustment system, and a stereo image adjustment program capable of performing fine adjustment on an image in order to increase accuracy.

  By taking a stereo image using a plurality of cameras, various measurements may be performed using the taken stereo image. For example, Patent Document 1 describes a stereo image processing apparatus capable of performing accurate distance measurement by image processing using a stereo method even in a blooming region, as an applied technique of a stereo camera including a plurality of cameras. .

  In addition, the parallax of multiple cameras (for example, main camera and reference camera) that make up the stereo camera is obtained from the stereo camera image that captures the floor from the ceiling, and the height of the object is obtained through the process of obtaining the distance from the camera. There is a technology to seek. For example, by photographing a certain area directly below using a stereo camera installed on the ceiling, a person passing or staying in the certain area is photographed, and the number of people passing, the number of stays, the staying time, etc. It is being measured.

  When adjusting a stereo camera installed on the ceiling, in general, a mark is placed on the floor surface that is vertical when viewed from both cameras that make up the stereo camera, and the center of each camera is adjusted to the mark. It is.

JP 2005-91173 A

  In stereo image processing in which a floor camera is photographed from a stereo camera installed on the ceiling and the captured image is processed, it is generally necessary to satisfy the following conditions. (1) The floor surface can be photographed without inclination. (2) There is no deviation in the shooting range between the reference image and the main image. (3) The difference in luminance between the reference image captured by the reference camera and the main image captured by the main camera is sufficiently small. Therefore, when stereo image processing is performed using a stereo camera installed on the ceiling, it is important to appropriately adjust the stereo camera that is performed in advance.

  However, when manually adjusting individual cameras using marks on the floor, an image in which errors of individual camera adjustments are added is captured as a captured stereo image. If the error of each camera is small, there is no problem, but since the adjustment is performed manually, the error is not so small that it can generally be ignored. In general, since the error is detected visually, mechanical correction is performed when correction is performed, and accuracy and stability are lacking.

  Therefore, the present invention provides a stereo image adjustment method, a stereo image adjustment system, and a stereo image adjustment program capable of improving the accuracy and stability of a stereo image without doubling an error between images constituting the stereo image. The purpose is to provide.

  The stereo image adjustment method according to the present invention includes a main image capturing step for capturing a main image constituting a stereo image, a reference image capturing step for capturing a reference image constituting the stereo image, and a coordinate value between the main image and the reference image. Based on the generated difference image, a difference image generation step for detecting a difference between the difference image generation step, a main partial image extraction step for extracting a predetermined partial image from the main image based on the generated difference image, And a reference partial image extracting step of extracting a predetermined partial image from the reference image.

  Further, the stereo image adjustment method includes a luminance difference detection step for detecting a difference in luminance value between the main image and the reference image based on the generated difference image, and a luminance of the reference image based on the detected difference in luminance value. And a luminance value correcting step for correcting the value. According to such a configuration, it is possible to improve the accuracy and stability of the adjustment of the luminance value of the reference image and to automate the luminance adjustment of the reference image.

  Further, the stereo image adjustment method may include a processing area limiting step for limiting an image processing area in the stereo image to a central portion in the image. According to such a configuration, it is possible to eliminate those having unnecessary heights such as surrounding walls included in the image.

  The stereo image adjustment system according to the present invention includes a main image input unit (for example, a pre-processing unit) that inputs a main image captured by a main image capturing unit (for example, a main camera of the camera 10) that captures a main image constituting a stereo image. 201), a reference image input unit (for example, realized by the preprocessing unit 211) that inputs a reference image captured by a reference image capturing unit that captures a reference image constituting a stereo image, and a main image Based on the difference image generation means (for example, realized by the difference image generation unit 216) that detects the difference in the coordinate value between the reference image and the reference image, and the difference image generated by the difference image generation means, Main partial image extraction means for extracting a predetermined partial image from the main image (for example, realized by the partial cutout unit 205) and the difference generated by the difference image generation means Based on the image, the reference partial image extracting means from the reference image to extract a predetermined partial image (e.g., as implemented by the partial cutting unit 215), characterized in that a.

  The stereo image adjustment system is realized by a coordinate correction value calculation unit (for example, an error calculation unit 217) that calculates a correction value of the coordinate values of the main image and the reference image based on the difference image generated by the difference image generation unit. And the main partial image extracting unit extracts a predetermined partial image from the main image based on the correction value calculated by the coordinate correction value calculating unit, and the reference partial image extracting unit includes the coordinate correction value calculating unit. A predetermined partial image may be extracted from the reference image based on the calculated correction value.

  The stereo image adjustment system is realized by a luminance difference detection unit (for example, an error calculation unit 217) that detects a difference in luminance value between the main image and the reference image based on the difference image generated by the difference image generation unit. ) And luminance value correcting means (for example, realized by the luminance value offset adding unit 212) for correcting the luminance value of the reference image based on the difference between the luminance values detected by the previous luminance difference detecting means. It may be. According to such a configuration, it is possible to improve the accuracy and stability of the adjustment of the luminance value of the reference image and to automate the luminance adjustment of the reference image.

  In addition, the stereo image adjustment system may include a processing area limiting unit (for example, realized by the error calculation unit 217) that limits an image processing area in the stereo image to a central portion in the image. According to such a configuration, it is possible to eliminate those having unnecessary heights such as surrounding walls included in the image.

  Further, in the stereo image adjustment system, the processing area limiting means limits the image processing area in the stereo image to a central portion in the image by applying a predetermined area mask to the difference image generated by the difference image generating means. It may be. According to such a configuration, the image processing area can be easily limited to the central portion in the image by performing the mask process.

  The stereo image adjustment program according to the present invention captures a main image input process for inputting a main image captured by a main image capturing unit that captures a main image constituting a stereo image and a reference image constituting the stereo image on a computer. Based on a reference image input process for inputting a reference image captured by a reference image capturing unit, a difference image generation process for generating a difference image by detecting a difference in coordinate values between the main image and the reference image, and the generated difference image A main partial image extraction process for extracting a predetermined partial image from the main image, and a reference partial image extraction process for extracting the predetermined partial image from the reference image based on the generated difference image. is there.

  In the present invention, the following items are realized by image processing to minimize errors. (A) The shooting direction of the main image is matched with the normal direction of the floor surface. (B) Adjust the shooting range of the reference image to the shooting range of the main image on the floor. (C) Match the luminance of the reference image to the luminance of the main image. (D) In order to avoid the influence of surrounding walls and the like, image processing is limited to the central portion. The items (B) and (C) are not individual adjustments, but the difference between the reference image and the main image is detected and corrected with the reference image. Therefore, unlike the case where the main image and the reference image are adjusted independently, It is an important feature that the error does not double. Further, the above (A) to (D) require accurate detection and accurate correction. Further, various types of detection are obtained as numerical values not by visual observation but by image processing. For various corrections, numerical correction on the image is realized instead of mechanical adjustment.

  In the present invention, (1) the main image shooting range (shooting direction) is adjusted. Specifically, the fact that the normal direction of the plane can be detected when a mirror is placed on the plane and the camera itself is projected is used. When the shooting direction is adjusted so that the center of the camera reflected on the monitor is at the center of the display screen of the monitor, the camera faces the normal direction of the plane, and the plane can be shot without tilting. In this case, the detected value is a difference in coordinate values between the center of the camera reflected on the monitor and the center of the display screen of the monitor. Further, fine adjustment of the shooting range, which is difficult by mechanical adjustment, is numerically realized by providing a partial image cutout function on the main image side and changing the cutout start point.

  In the present invention, (2) the reference image shooting range is adjusted. Specifically, the shooting range is matched with the reference image and the main image using the floor pattern as a clue. In this case, when there is no pattern on the floor surface, paper having a thickness different from the floor surface, such as paper having a thickness, is placed. A difference image between the reference image and the main image is created, and the square value of the luminance value difference is added to the entire floor surface in units of pixels. Further, the square sum of the error is obtained while moving the reference image in the left-right direction and the front-rear direction, and the amount of movement that minimizes the square sum of the error is obtained. If the shooting range of the main image is different from the shooting range of the reference image, an error is detected from the difference image. If there is no deviation in the left-right direction and the front-back direction, the error is minimized.

  Further, fine adjustment of the shooting range, which is difficult by mechanical adjustment, is numerically realized by providing a partial image cutout function on the reference image side and changing the cutout start point.

  In the present invention, (3) the reference image luminance value is adjusted. Specifically, a difference image between the reference image and the main image is created, and the luminance value difference is added to the entire floor surface in units of pixels. Then, an average value of the luminance value differences is obtained by calculating the total sum of the luminance value differences and dividing by the number of pixels. Further, the fine adjustment of the luminance value, which is difficult by mechanical adjustment, is numerically realized by adding a luminance value offset function to the reference image side and adding the average value of the luminance value differences.

  In the present invention, (4) the central portion of the image processing area is limited. Specifically, this is realized by referring to the area mask when calculating the sum of errors from the difference image. In this case, an area mask is set in advance, and an area mask is set in advance so as to exclude surrounding walls having the same height. In addition, you may specify by a coordinate value not only when using a region mask. For example, the same effect can be obtained by preparing in advance information indicating a rectangle including the central portion by coordinates of diagonal vertices.

  According to the present invention, a main image constituting a stereo image is photographed, and a reference image constituting a stereo image is photographed. Further, a difference image is generated by detecting a difference in coordinate values between the main image and the reference image. Then, based on the generated difference image, a predetermined partial image is extracted from the main image. A predetermined partial image is extracted from the reference image based on the generated difference image.

  By configuring as described above, in the stereo image processing, the difference between the main image and the reference image is detected as a numerical value, and the main image and the reference image are corrected as numerical values, thereby minimizing the error and stereo. You can adjust the image. Further, since the error between images is detected and corrected, the error between images does not double and the accuracy and stability can be remarkably improved.

  In addition, according to the present embodiment, since an error between images is detected by image processing and corrected on the image, stereo image adjustment processing can be automated, and manual operation can be eliminated. . In addition, since manual adjustment of the camera can be performed at the coarse adjustment level and fine adjustment can be automated, it is possible to realize a significant reduction in the adjustment time of the stereo image.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing an example of a configuration of a stereo image adjustment system using a stereo image adjustment method according to the present invention. In this embodiment, the stereo image adjustment system is applied to automatic adjustment of the shooting direction when the camera 10 is installed on a ceiling with a height of 2.5 m and wide-angle shooting is performed at an angle of 90 degrees directly below the camera 10. An example will be described. Moreover, in this Embodiment, it can apply to uses, such as a system etc. which image | photograph the person who passes or stays in a fixed area | region, and measures the number of passing persons, the number of stays, stay time, etc.

  As shown in FIG. 1, the stereo image adjustment system includes a preprocessing unit 201, a camera center detection unit 202, a cutout start point calculation unit 204, a partial cutout unit 205, a surrounding complement unit 206, a composite display unit 207, and overall control. A unit 208, a display unit 209, an input unit 210, a preprocessing unit 211, a luminance value offset addition unit 212, a partial cutout unit 215, a difference image generation unit 216, and an error calculation unit 217. Note that the stereo image adjustment system is specifically realized by an information processing apparatus such as a personal computer.

  Specifically, the camera 10 is a stereo camera configured by a plurality of (two in this example) cameras. In the present embodiment, the camera 10 includes a main camera that captures a main image and a reference camera that captures a reference image. The camera 10 is installed on the ceiling and is installed in advance so as to photograph a direction almost directly below the ceiling. In the present embodiment, fine adjustment when taking an image is performed by adjusting the orientation of the camera 10 or the like. Moreover, in this Embodiment, a mirror is installed in the location of the floor surface facing the camera 10, and the camera 10 image | photographs the image containing the camera 10 itself reflected in a mirror.

  Specifically, the preprocessing unit 201 is realized by a CPU of an information processing apparatus that operates according to a program. The preprocessing unit 201 has a function of inputting a main image captured by the main camera of the camera 10. In addition, the preprocessing unit 201 has a function of executing preprocessing such as removing isolated points (isolated points included in the image) from the image input from the main camera constituting the camera 10.

  Specifically, the camera center detection unit 202 is realized by a CPU of an information processing apparatus that operates according to a program. The camera center detection unit 202 has a function of detecting the center position of the camera 10 reflected in the center portion of the image captured by the main camera.

  Specifically, the cutting start point calculation unit 204 is realized by a CPU of an information processing apparatus that operates according to a program. The cutout start point calculation unit 204 has a function of calculating a cutout start point when cutting out an image from a captured image of the main camera.

  Specifically, the partial cutout unit 205 is realized by a CPU of an information processing apparatus that operates according to a program. The partial cutout unit 205 has a function of extracting a partial image from the main image captured by the main camera based on the cutout start point calculated by the cutout start point calculation unit 204.

  Specifically, the surrounding complement unit 206 is realized by a CPU of an information processing apparatus that operates according to a program. The surrounding complement unit 206 has a function of executing a complementing process that complements the periphery of the partial image extracted by the partial cutout unit 205. For example, the surrounding complement unit 206 complements a predetermined fixed color image around the partial image, and generates an image having the same size as the captured image captured by the main camera.

  Specifically, the composite display unit 207 is realized by a CPU of an information processing apparatus that operates according to a program. The composite display unit 207 has a function of causing the display unit 209 to display, as an output image, an image that has been subjected to the complement processing by the surrounding complement unit 206. The composite display unit 207 applies predetermined processing to the output image based on the display information from the overall control unit 208 and causes the display unit 209 to display the output image as an output image.

  Specifically, the overall control unit 208 is realized by a CPU of an information processing apparatus that operates according to a program. The overall control unit 208 has a function of causing the display unit 209 to display various setting screens when performing image adjustment. The overall control unit 208 also has a function of inputting various operation information from the input unit 210 in accordance with a user operation.

  Specifically, the display unit 209 is realized by a display device such as a display device. The display unit 209 has a function of displaying an image according to an instruction from the composite display unit 207 and displaying various setting screens according to an instruction from the overall control unit 208.

  Specifically, the input unit 210 is realized by an input device such as a keyboard or a mouse. The input unit 210 has a function of inputting various operation information and outputting it to the overall control unit 208 in accordance with a user operation.

  Specifically, the preprocessing unit 211 is realized by a CPU of an information processing apparatus that operates according to a program. The preprocessing unit 211 has a function of inputting a reference image taken by the reference camera of the camera 10. In addition, the preprocessing unit 211 has a function of executing preprocessing such as removing isolated points (isolated points included in the image) from an image input from a reference camera constituting the camera 10.

  Specifically, the brightness value offset adding unit 212 is realized by a CPU of an information processing apparatus that operates according to a program. The luminance value offset adding unit 212 has a function of performing a process of adding a predetermined offset value to the luminance value of the reference image that has been preprocessed by the preprocessing unit 211.

  Specifically, the partial cutout unit 215 is realized by a CPU of an information processing apparatus that operates according to a program. The partial cutout unit 215 has a function of extracting a partial image from a reference image taken by the reference camera. In addition, when the correction value of the coordinate value of the main image and the reference image is calculated by the error calculation unit 217, the partial cutout unit 205 refers to the reference based on the correction value of the coordinate value of the main image and the reference image. Extract a partial image from an image.

  Specifically, the difference image generation unit 216 is realized by a CPU of an information processing apparatus that operates according to a program. The difference image generation unit 216 detects a difference in coordinate values between the partial image of the reference image extracted by the partial cutout unit 215 and the partial image of the main image extracted by the partial cutout unit 205 to generate a difference image. It has a function.

  Specifically, the error calculation unit 217 is realized by a CPU of an information processing apparatus that operates according to a program. The error calculation unit 217 has a function of obtaining an error between the main image and the reference image based on the difference image generated by the difference image generation unit 216. Further, the error calculation unit 217 has a function of detecting a difference in luminance value between the main image and the reference image based on the difference image generated by the difference image generation unit 216.

  In the present embodiment, the preprocessing unit 201, the camera center detection unit 202, the extraction start point calculation unit 204, and the partial extraction unit 205 adjust the imaging range of the main image captured by the main camera of the camera 10. It functions as a main image shooting range (shooting direction) adjustment unit. The partial cutout unit 215, the difference image generation unit 216, and the error calculation unit 217 function as a reference image shooting range adjustment unit that adjusts the shooting range of the reference image shot by the reference camera of the camera 10. Furthermore, the luminance value offset addition unit 212, the difference image generation unit 216, and the error calculation unit 217 function as a reference image luminance value adjustment unit that adjusts the luminance value of the reference image.

  In the present embodiment, the storage device (not shown) of the information processing apparatus that implements the stereo image adjustment system stores various programs for adjusting the stereo camera so that the floor surface can be photographed without tilting from the ceiling. is doing. For example, the storage device of the information processing apparatus captures a main image input process for inputting a main image captured by a main image capturing unit that captures a main image forming a stereo image and a reference image forming the stereo image on a computer. A reference image input process for inputting a reference image captured by the reference image capturing means, a difference image generation process for generating a difference image by detecting a difference in coordinate values between the main image and the reference image, and a generated difference image A stereo for executing a main partial image extraction process for extracting a predetermined partial image from the main image and a reference partial image extraction process for extracting the predetermined partial image from the reference image based on the generated difference image An image adjustment program is stored.

  Next, the operation will be described. In this embodiment, the camera 10 (stereo camera) is installed on a ceiling with a height of 2.5 m, and wide-angle shooting is performed using a single mirror at an angle of 90 degrees directly below the camera 10. The process of is shown.

(1) Adjustment of main image shooting range (shooting direction) First, the adjustment operation of the shooting range of the main image shot by the main camera of the camera 10 will be described. FIG. 2 is a flowchart illustrating an example of processing when adjusting the shooting range of the main image.

  In the present embodiment, the stereo image adjustment system realizes automatic adjustment by detecting the center of the main camera constituting the camera 10 by image processing. In the present embodiment, it is assumed that the fine adjustment width is ± 5% of the angle of view, and the effective angle of view is 90% of the angle of view of the entire captured image. Accordingly, if a lens having a horizontal angle of view of 100 degrees is used as the lens of the main camera of the camera 10, an image having a horizontal angle of view of 90 degrees is output as a result. It is also assumed that the effective field angle in the vertical direction is 3/4 times the horizontal field angle of 90 degrees. Accordingly, an image with a vertical angle of view of 67.5 degrees is output.

  It is assumed that the shooting size (the size expressed using pixels) of an image shot by the main camera is 640 × 480 pixels. Therefore, 5% of the angle of view corresponds to 32 pixels in the horizontal direction and 24 pixels in the vertical direction when converted into pixels. Further, ± 5% of the angle of view that is the width of fine adjustment corresponds to ± 5 degrees and ± 3.375 degrees with respect to the horizontal direction, respectively.

  Further, as shown in FIG. 3, it is assumed that a mirror is installed at a location on the floor surface facing the camera 10. In this case, since the camera 10 itself is reciprocated by a mirror (that is, the camera 10 captures an image of the camera 10 itself reflected in the mirror), the shooting distance is 5 m. In this embodiment, since the wide-angle shooting at 100 degrees is performed, the camera itself is only small. Further, the focus is not suitable for the camera 10 itself having a shooting distance of 5 m.

  In the present embodiment, in consideration of the above-described points, what is detected in image processing is narrowed down to the center of the camera 10 at the time of adjustment, conditions are adjusted at the time of adjustment, and simple image processing is performed.

  First, in the present embodiment, processing is executed using the difference between the color of the ceiling reflected in the mirror and the color of the camera 10. Hereinafter, in order to simplify the description, a case will be described in which a method is used in which the ceiling is white and the camera 10 is detected as relatively black. Note that the colors of the ceiling and the camera 10 shown in this embodiment are merely examples, and if the color of the ceiling and the color of the camera 10 are easily distinguishable, processing is performed using, for example, light blue and brown other than white and black. May be performed.

  When the color of the ceiling and the color of the camera 10 are close to each other (for example, when the color is white and light gray), white paper is temporarily pasted around the camera 10 on the ceiling or only during the shooting. The camera 10 may be easily extracted by applying a black tape or the like.

  In addition, it is not necessary to use a large mirror to be installed on the floor surface, and it is sufficient to use a mirror having a size of about A4. Note that the size of the mirror used for photographing is not limited to a size of about A4, and for example, a larger mirror may be used. Further, the range for searching for the camera 10 is also limited to the central part of the mirror.

  First, the operation in the preparation stage before starting video shooting will be described. The user places the mirror almost directly below the camera 10, and while viewing the mirror image displayed on the monitor (for example, the display unit 209), the user positions the mirror so that the camera 10 appears in the center of the mirror. adjust. In this case, as shown in FIG. 4, the left-right position and the front-rear position of the mirror are adjusted so that the camera 10 itself reflected in the mirror in the video is substantially in the center of the mirror. Further, the mounting position of the camera 10 is adjusted so that the center position of the camera 10 in the image is the center of the display screen of the monitor.

  Next, the camera center detection unit 202 of the stereo image adjustment system detects the normal direction of the floor surface by detecting the center position of the camera 10 in the captured image by image processing according to the user's operation (step). S101). That is, when the camera 10 itself is projected with a mirror placed on a plane, the direction of the center of the camera 10 reflected in the mirror should always be a normal direction to the plane. Therefore, the camera center detection unit 202 detects the normal direction of the plane by detecting the center position of the camera 10. In this embodiment, the plane can be photographed without inclination by directing the photographing direction to the normal direction of the plane. Further, the cutout start point calculation unit 204 sets a cutout start point at the time of extracting a partial image from the captured image to a predetermined standard value described later (step S102).

  As described above, by performing the pre-adjustment, as shown in FIG. 5, even if the camera 10 is slightly tilted, the mirror is adjusted so as to move directly downward.

  Next, an image with a predetermined angle of view (100 degrees (as a result, 90 degrees in this example)) is taken using the camera 10 installed on the ceiling (for example, an image in which a human passes or stays within a certain area) The operation | movement which performs is demonstrated.

  First, the pre-processing unit 201 of the stereo image adjustment system inputs a camera image captured by the main camera of the camera 10 and performs pre-processing such as removing isolated points from the input camera video (step S103). Next, the camera center detection unit 202 detects the position (camera region) of the camera 10 that appears in the center portion in the image (step S104). Specifically, a predetermined threshold value (for example, a predetermined luminance value) for separating the white color of the ceiling and the black color of the camera 10 is set in advance. The camera center detection unit 202 converts each pixel in the image into a binary image based on a predetermined threshold value set in advance, and extracts a black portion regarded as the camera 10. The predetermined threshold is stored in advance in a storage unit such as a hard disk device or a memory provided in the information processing apparatus, for example.

  Next, the camera center detection unit 202 searches in the direction of the raster scan only in the center part of the image, and detects the minimum value of the coordinates in the image where the black part of the camera 10 is detected with respect to the white color of the ceiling. And the maximum value are obtained (step S105). In this case, the camera center detection unit 202 obtains a maximum value and a minimum value for each direction of the X axis and the Y axis.

  Next, the camera center detection unit 202 calculates an average value of the obtained minimum value and maximum value for each of the X axis and the Y axis as the center position of the camera 10 (step S106). For example, as shown in FIG. 6, when the left end of the obtained camera area is Xmin, the right end is Xmax, the front end is Ymin, and the rear end is Ymax, the camera center detection unit 202 determines the center coordinates of the camera 10 position. It can be determined using equation (1).

X = (Xmin + Xmax) / 2
Y = (Ymin + Ymax) / 2 Formula (1)

  The deviation value between the coordinate value (center position of the camera 10) obtained here and the center position of the image represents the deviation value between the normal direction of the floor surface and the current shooting direction. The camera center detection unit 202 obtains a deviation value between the normal direction of the floor surface and the current shooting direction, and passes it to the extraction start point calculation unit 204 as a correction value.

  Next, the cutout start point calculation unit 204 calculates a cutout start point when cutting out an image from the captured image (step S107).

  When calculating the cutout start point, the adjustment of the shooting direction can be realized by providing the stereo image adjustment system with an image cutout function and changing the cutout start point. In the present embodiment, the standard value of the clipping start point is set to 5 from the origin position (for example, the upper left point of the captured image) of the original image (the image captured by the camera 10 (that is, the image before processing)). The percentage (32, 24) position. A point obtained by adding the deviation value obtained by the image processing to the cutout start point is obtained as an actual start point. The adjustment of ± 5% can be realized by varying ± 32 pixels and ± 24 pixels in the X-axis and Y-axis directions, respectively.

  Next, the partial cutout unit 205 performs a process of cutting out a 90% partial image (partial image of 576 × 432 pixels) of the original image from the cutout start point obtained by the cutout start point calculation unit 204 (step S108). .

  In this embodiment, instead of changing the direction of the camera 10, a part of the photographed image is cut out to be an output image, and processing is performed so as to change the center direction of the photographing by changing the cutout part of the image. Specifically, the center position of the shooting can be moved by moving the cutout start point. For example, the center position of the camera 10 and the center position of the display screen of the monitor as shown in FIG. 7 are obtained by executing the processing described above and correcting (adding with a sign) the shift value in the center direction of the display screen. The deviation value from (320, 240) can be corrected, and the center position of the camera 10 can be adjusted to the center of the output image.

  Next, when displaying the cut out partial image at the size of the original image, the surrounding complement unit 206 performs a complementing process for complementing the surrounding of the partial image extracted by the partial cutting unit 205 (step S109). For example, when the output image is processed into the size of the original image and displayed, the surrounding complement unit 206 performs the complement processing by enlarging the partial image extracted by the partial cutout unit 205 to a size of 111.1%. To do. Further, for example, the surrounding complement unit 206 processes the shortage around the partial image to the size of the original image by adding an image of a predetermined fixed color.

  Then, the composite display unit 207 outputs the image complemented by the surrounding complement unit 206 (step S110). For example, the composite display unit 207 causes the display unit 209 to display the complemented image.

  Further, it is desirable to display a cursor at the center of the display screen of the monitor (display unit 209) so that the user can confirm whether the automatic adjustment has been performed correctly. In this case, if the center position of the camera 10 is displayed at the center position of the cursor center (that is, if the center position of the camera 10 matches the center position of the display screen), the automatic adjustment by the stereo image adjustment system is performed. It can be seen that the adjustment was successful.

(2) Adjustment of Reference Image Shooting Range Next, an operation for adjusting the shooting range of the reference image shot by the reference camera of the camera 10 will be described. FIG. 8 is a flowchart illustrating an example of processing when adjusting the shooting range of the reference image. When adjusting the shooting range of the reference image, the adjustment work is performed using a mark such as a pattern drawn on the floor. At this time, since there is often no mark such as a pattern on the floor surface, for example, a paper of a predetermined color having a size of about A4 may be placed at the center of the photographing range on the floor surface. However, the color of the paper needs to be a color (for example, white) different from the color (for example, gray) of the floor surface so that the paper can be distinguished from the floor surface.

  First, the preprocessing unit 211 inputs a camera image captured by the reference camera of the camera 10 according to the same processing as the preprocessing unit 201, and performs preprocessing such as removing isolated points from the input camera image (step). S201). Next, the partial cutout unit 215 performs 90% partial image (576 × 432) of the reference image preprocessed by the preprocessing unit 211 from a predetermined cutout start point according to the same process as the partial cutout unit 205. A process of cutting out a partial image of the pixel is performed (step S202).

Next, the difference image generation unit 216 includes a partial image of the reference image extracted by the partial cutout unit 215,
A difference image with the partial image of the main image extracted by the partial cutout unit 205 is generated (step S203).

  Next, the error calculation unit 217 obtains an error between the main image and the reference image based on the difference image generated by the difference image generation unit 216. Specifically, the error calculation unit 217 changes the coordinate correction value of the partial cutout unit of the reference image one by one in the left-right direction or the front-rear direction (for example, one pixel at a time), and each of the reference image and the reference image. A sum of squares of errors between the partial images of the image is obtained (step S204). Further, the error calculation unit 217 obtains a coordinate correction value that minimizes the sum of squares of errors as an adjusted coordinate correction value (step S205).

  Then, the partial cutout unit 215 extracts a partial image from the reference image based on the coordinate correction value calculated by the error calculation unit 217.

  FIG. 9 is an explanatory diagram illustrating an example of a method for adjusting the shooting range of the reference image. By executing the processes in steps S201 to S205, the stereo image adjustment system causes a gap between the main image and the reference image when there is a shift in the shooting range between the main camera and the reference camera (see the left diagram in FIG. 9). A portion having a difference in luminance value is detected. Based on the detection result, the user performs adjustment work such as the orientation of the reference camera so that the shooting ranges overlap. Then, as the shooting ranges of the main camera and the reference camera overlap according to the user's adjustment work, the area of the portion with the difference in luminance value decreases (see the diagram in FIG. 9), and the luminance value is obtained by the stereo image adjustment system. The portion detected as having a difference is reduced. When the shooting ranges of the main camera and the reference camera completely match (see the diagram on the right in FIG. 9), there is no portion detected as having a difference in luminance value by the stereo image adjustment system, and the shooting range of the reference image is eliminated. Adjustment is completed.

  By executing the above processing, the shooting range of the reference image can be matched with the shooting range of the main image on the floor. Since the adjustment width of the reference image is set to ± 5%, which is the same as that of the main image, ± 32 pixels and ± 24 pixels can be varied in the X-axis direction and the Y-axis direction, respectively.

  Further, the difference image may be displayed on the display screen displayed on the display unit 209 when adjusting the reference image. By doing so, it can be seen that the position displayed on the display screen with the paper on the floor disappeared is the position where the main image and the reference image match. Therefore, it is possible to confirm whether or not the reference image has been adjusted by confirming that the paper finally disappears and cannot be seen on the display screen.

  In the present embodiment, the case where the error calculation unit 217 obtains the square value of the error has been described, but the absolute value of the error may be obtained.

(3) Adjustment of Reference Image Brightness Value Next, the adjustment operation of the brightness value of the reference image taken by the reference camera will be described. FIG. 10 is a flowchart illustrating an example of processing when adjusting the luminance value of the reference image. When adjusting the luminance value of the reference image, nothing may be placed on the floor. However, the paper used for adjusting the shooting range of the reference image may be left as it is. In order to adjust the luminance value of the reference image, it is assumed that the adjustment of the shooting range of the reference image has already been completed. Note that the adjustment of the reference image capturing range and the adjustment of the luminance value may be executed continuously.

  The luminance value offset adding unit 212 performs a process of adding a predetermined offset value to the luminance value of the reference image that has been preprocessed by the preprocessing unit 211. In this case, the luminance value offset adding unit 212 first sets the luminance correction value used in the processing system on the reference image processing side to 0 (step S301). Then, the partial cutout unit 215 extracts a partial image from the reference image from the luminance value offset addition unit 212 and outputs the partial image to the difference image generation unit 216 (step S302).

  Next, the difference image generation unit 216 generates a difference image between the partial image of the reference image extracted by the partial cutout unit 215 and the partial image of the main image extracted by the partial cutout unit 205 (step S303).

  Next, the error calculation unit 217 obtains an error between the main image and the reference image based on the difference image generated by the difference image generation unit 216. Specifically, the error calculation unit 217 obtains the sum of the differences in the brightness values of the difference image, and divides the obtained sum of the differences in the brightness values by the number of pixels, thereby obtaining the brightness values of the main image and the reference image. An average value of the differences is obtained (step S304).

  Next, the luminance value offset adding unit 212 adds the average value obtained by the error calculating unit 217 to the previous luminance correction value (set to 0 as described above), and sets it as a new luminance correction value. (Step S305). That is, the luminance value offset adding unit 212 corrects the luminance value of the reference image based on the average value of the luminance values difference between the main image and the reference image detected by the error calculating unit 217.

  By executing the above processing, the luminance value of the reference image can be adjusted. The adjustment of the luminance value of the reference image can be completed in one process without changing the setting value.

(4) Limiting the Image Processing Area to the Center Next, the operation for limiting the image processing area in the stereo image to the center will be described. This image processing area setting process is a process that is commonly used for adjustment of the shooting area of the reference area and adjustment of the luminance value.

  Specifically, the limitation on the center of the image processing area is realized by referring to the area mask when the error calculation unit 217 calculates the sum of errors from the difference image. For example, the error calculation unit 217 sets an area mask in advance (for example, stores the area mask in advance in a storage unit such as a hard disk device or a memory). Then, when calculating the sum of errors based on the difference image, the error calculation unit 217 excludes an image having a predetermined height such as a surrounding wall by applying a region mask to the difference image.

  It should be noted that there is nothing that can be an obstacle to detection processing such as a normal wall in the center of the image. Therefore, if a region mask having a rectangular central portion is set as an initial value, the error calculation unit 217 can perform processing using the region mask almost as it is. In this case, the size of the rectangle included in the region mask may be about 1/6 of the entire image in both the left-right direction and the front-rear direction.

  Instead of using the area mask to limit the center of the image processing area, the center of the image processing area may be specified using a predetermined coordinate value. For example, the same effect can be expected by setting in advance setting information in which a rectangle including the central portion of the image processing area is indicated by the coordinates of diagonal vertices and executing processing based on the setting information set in advance.

  As described above, according to the present embodiment, the main camera of the camera 10 captures the main image that forms the stereo image, and the reference camera of the camera 10 captures the reference image that forms the stereo image. In addition, the difference image generation unit 216 detects a difference in coordinate values between the main image and the reference image and generates a difference image. Then, the partial cutout unit 205 extracts a partial image from the main image based on the shift value calculated by the camera center detection unit 202. Further, the partial cutout unit 215 extracts a partial image from the reference image based on the difference image generated by the difference image generation unit 216.

  By configuring as described above, in the image processing of the stereo image, the difference between the main image and the reference image is detected as a numerical value, and the main image and the reference image (for example, the shooting range of the main image or the reference image) are used as numerical values. By correcting, the stereo image can be adjusted with minimum error. Further, since the error between images is detected and corrected, the error between images does not double and the accuracy and stability can be remarkably improved.

  In addition, according to the present embodiment, since an error between images is detected by image processing and corrected on the image, stereo image adjustment processing can be automated, and manual operation can be eliminated. . In addition, since manual adjustment of the camera can be performed at the coarse adjustment level and fine adjustment can be automated, it is possible to realize a significant reduction in the adjustment time of the stereo image.

  The present invention captures a person who passes or stays in a certain area by photographing a certain area directly below using a stereo camera installed on the ceiling, and measures the number of people passing, the number of stays, and the staying time. It can be applied to uses such as a system for measuring etc.

It is a block diagram which shows an example of a structure of the stereo image adjustment system using the stereo image adjustment method by this invention. It is a flowchart which shows an example of the process in the case of adjusting the imaging | photography range of a main image. It is explanatory drawing which shows the positional relationship of a camera and the mirror of a floor surface. It is explanatory drawing which shows the method of adjusting the position of the mirror with respect to a camera. It is explanatory drawing which shows the positional relationship of the camera and floor mirror when a camera inclines. It is explanatory drawing which shows the example of the camera area | region in a picked-up image. It is explanatory drawing which shows the example which correct | amends the deviation value of the center position of a camera, and the center position of the display screen of a monitor. It is a flowchart which shows an example of the process in the case of adjusting the imaging | photography range of a reference image. It is explanatory drawing which shows the example of the adjustment method of the imaging range of a reference image. It is a flowchart which shows an example of the process in the case of adjusting the luminance value of a reference image.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Camera 11 Mirror 100 Camera unit 201 Preprocessing part 202 Camera center detection part 204 Extraction start point calculation part 205 Partial extraction part 206 Surrounding complement part 207 Composite display part 208 General control part 209 Display part 210 Input part 211 Preprocessing part 212 Brightness value offset addition unit 215 Partial cutout unit 216 Difference image generation unit 217 Error calculation unit

Claims (9)

A main image capturing step for capturing a main image constituting a stereo image;
A reference image photographing step of photographing a reference image constituting the stereo image;
A difference image generation step of generating a difference image by detecting a difference in coordinate values between the main image and the reference image;
A main partial image extraction step of extracting a predetermined partial image from the main image based on the generated difference image;
And a reference partial image extracting step of extracting a predetermined partial image from the reference image based on the generated difference image. A luminance difference detection step for detecting a difference in luminance value between the main image and the reference image based on the generated difference image;
The stereo image adjustment method according to claim 1, further comprising: a luminance value correction step of correcting a luminance value of the reference image based on a difference between the detected luminance values.   The stereo image adjustment method according to claim 1, further comprising a processing region limiting step for limiting an image processing region in the stereo image to a central portion in the image. A main image input means for inputting a main image taken by a main image photographing means for photographing a main image constituting a stereo image;
A reference image input means for inputting a reference image taken by a reference image photographing means for photographing a reference image constituting the stereo image;
Difference image generation means for generating a difference image by detecting a difference in coordinate values between the main image and the reference image;
Main partial image extraction means for extracting a predetermined partial image from the main image based on the difference image generated by the difference image generation means;
A stereo image adjustment system comprising: reference partial image extraction means for extracting a predetermined partial image from the reference image based on the difference image generated by the difference image generation means. Coordinate correction value calculation means for calculating a correction value of the coordinate values of the main image and the reference image based on the difference image generated by the difference image generation means,
The main partial image extraction unit extracts a predetermined partial image from the main image based on the correction value calculated by the coordinate correction value calculation unit,
The stereo image adjustment system according to claim 4, wherein the reference partial image extraction unit extracts a predetermined partial image from the reference image based on the correction value calculated by the coordinate correction value calculation unit. A luminance difference detecting means for detecting a difference in luminance value between the main image and the reference image based on the difference image generated by the difference image generating means;
The stereo image adjustment system according to claim 4, further comprising: a luminance value correcting unit that corrects a luminance value of the reference image based on a luminance value difference detected by the luminance difference detecting unit in the previous period.   The stereo image adjustment system according to any one of claims 4 to 6, further comprising processing area limiting means for limiting an image processing area in the stereo image to a central portion in the image.   The stereo image adjustment according to claim 7, wherein the processing region limiting unit limits the image processing region in the stereo image to a central portion in the image by applying a predetermined region mask to the difference image generated by the difference image generating unit. system. On the computer,
A main image input process for inputting a main image taken by a main image photographing means for photographing a main image constituting a stereo image;
A reference image input process for inputting a reference image captured by a reference image capturing means for capturing a reference image constituting the stereo image;
A difference image generation process for generating a difference image by detecting a difference in coordinate values between the main image and the reference image;
A main partial image extraction process for extracting a predetermined partial image from the main image based on the generated difference image;
A stereo image adjustment program for executing a reference partial image extraction process for extracting a predetermined partial image from the reference image based on a generated difference image.