WO2012029658A1

WO2012029658A1 - Imaging device, image-processing device, image-processing method, and image-processing program

Info

Publication number: WO2012029658A1
Application number: PCT/JP2011/069316
Authority: WO
Inventors: 寿之猪子; 康毅斎藤
Original assignee: チームラボ株式会社
Priority date: 2010-08-30
Filing date: 2011-08-26
Publication date: 2012-03-08
Also published as: TW201225658A; JP2012050013A

Abstract

[Problem] To make it possible to perform more-accurate cropping without relying on the skill of the user when cropping out a subject from image information in which the image of the subject has been captured. [Solution] The present invention is characterized in comprising: an image camera (15) for generating the image of a subject by capturing the image; a distance camera (16) for measuring the distance to a target object displayed in various sections in a case in which the image is a visual range that includes the subject and a background, and generating distance information in which a correlation is established between the distance and the coordinates of the visual range on the image; a coordinates converter (101) for converting the coordinates of the distance information to the coordinates on the image of the subject and generating converted distance information; and an image-cropping unit (102) for extracting, from among the post-conversion coordinates included in the converted distance information, coordinates for which the correlated distance satisfies a predetermined condition, separating the region specified by the extracted coordinates and other regions in the image of the subject, and outputting the result.

Description

Imaging apparatus, image processing apparatus, image processing method, and image processing program

The present invention relates to an imaging device, an image processing device, an image processing method, and an image processing program, and more particularly, to a separation process between a subject included in an image generated by imaging and a background.

As a mode of utilizing an image obtained by photographing a subject, there is a mode in which a subject can be combined with other backgrounds by cutting out the subject along a contour. Such subject clipping processing may be realized by image processing on image information digitized by photoelectric conversion as well as by manual operation by an operator.

As a mode of realizing the clipping of the subject by image processing, an area of the image including the contour is designated by the operator, and the comparison of the density of the image in the designated area with a separately designated reference density is performed. A method for detecting a contour line has been proposed (see, for example, Patent Document 1).

In addition, in order to reduce the influence of the background image and realize more accurate clipping, pixel groups with similar features in the specified area are classified as clusters, and the clusters are inside and outside the contour of the subject. There has been proposed a method for detecting the contour line of a subject by classifying them as follows (for example, see Patent Document 2).

On the other hand, a method for detecting a background portion and a subject portion in an image based on image information of two identical images having different focus conditions has been proposed (for example, see Patent Document 3).

JP 63-5745 A Japanese Patent Laid-Open No. 9-83776 Japanese Patent Laid-Open No. 10-233919

In the techniques disclosed in Patent Documents 1 and 2, it is necessary for the user to designate one or two areas. A user who understands the characteristics of image clipping and the like can specify an appropriate area, but it is difficult for a general user to specify an area by understanding the characteristics of image clipping.

In any case of Patent Documents 1 to 3, since the determination is based on the digitized image information, subject clipping processing is appropriate depending on the state of the image information such as the relationship between the subject color and the background color. May not be executed.

The present invention has been made in consideration of the above circumstances, and enables more accurate separation processing regardless of the skill level of the user in the separation processing of the subject and the background in the image information obtained by capturing the subject. The purpose is to do.

In order to solve the above problems, according to one aspect of the present invention, an image capturing unit that generates a subject image on which a subject and a background are displayed by imaging, and a visual range including the subject and the background are images. A distance information generation unit that measures a distance to an object displayed in each unit of the image and generates distance information in which coordinates and distances on the image of the visual range are associated with each other, A coordinate conversion unit that converts the coordinates of the acquired distance information into coordinates on the subject image to generate converted distance information, and an association among the converted coordinates included in the generated converted distance information A coordinate extraction unit that extracts coordinates for which a predetermined distance satisfies a predetermined condition, and an image separation unit that separates and outputs a region specified by the extracted coordinates and another region of the subject image including And wherein the door.

According to another aspect of the present invention, there is provided an image processing apparatus that measures a distance to an object displayed on each unit when a visual range including a subject and a background is an image, and A distance information acquisition unit that acquires distance information in which coordinates on a target range image and a distance are associated; a subject image acquisition unit that acquires a subject image on which a subject and a background are displayed; and the acquired distance information A coordinate conversion unit that converts the coordinates of the subject image into coordinates on the subject image and generates converted distance information, and among the converted coordinates included in the generated converted distance information, an associated distance is predetermined A coordinate extraction unit that extracts coordinates satisfying the above condition, and an image separation unit that separates a region specified by the extracted coordinates from the other region of the acquired subject image. To do.

Further, another aspect of the present invention is an image processing method for measuring a distance to an object displayed on each part when a visual range including a subject and a background is an image, The distance information in which the coordinates on the image of the visual range and the distance are associated is acquired and stored in a storage medium, and the subject image on which the subject and the background are displayed is acquired and stored in the storage medium. The converted distance information is converted into coordinates on the subject image, converted distance information is generated and stored in a storage medium, and the converted coordinates included in the generated converted distance information are associated with each other. A coordinate satisfying a predetermined condition is extracted, and an area specified by the extracted coordinate is separated from another area of the acquired subject image and stored in a storage medium. And

Still another aspect of the present invention is an image processing program for measuring a distance to an object displayed on each part when a visual range including a subject and a background is an image, Obtaining distance information in which the coordinates on the image of the visual range and the distance are associated with each other and storing the distance information in a storage medium; obtaining a subject image displaying the subject and the background; and storing the subject image in the storage medium; Converting the stored distance information coordinates into coordinates on the subject image to generate converted distance information and storing the converted distance information in a storage medium; and the converted distance information included in the generated converted distance information Of the coordinates, the coordinates for which the associated distance satisfies a predetermined condition are extracted, and the area specified by the extracted coordinates and the other area are separated and stored in the acquired subject image. Characterized in that and a step of storing the body data processing device.

According to the present invention, in the separation processing of the subject and the background in the image information obtained by capturing the subject, it is possible to perform the separation processing with higher accuracy regardless of the skill level of the user.

It is a block diagram which shows the hardware constitutions of the imaging device which concerns on embodiment of this invention. It is a figure which shows the function structure of the imaging device which concerns on embodiment of this invention. It is a figure which shows the example of the distance information acquired by the distance camera which concerns on embodiment of this invention. It is a figure which shows the principle of the coordinate conversion function of the image plane / three-dimensional space which concerns on embodiment of this invention. It is a figure which shows the principle of the coordinate conversion function of rotation and translation which concerns on embodiment of this invention. It is a figure which shows the state which accumulated the distance information which concerns on embodiment of this invention on the to-be-photographed image. It is a figure which shows the extraction object area | region which concerns on embodiment of this invention. It is a figure which shows the correction | amendment aspect of the distortion of the lens which concerns on embodiment of this invention. It is a figure which shows the example of the converted distance information which concerns on other embodiment of this invention. It is a figure which shows the division | segmentation aspect of the point specified by the converted distance information which concerns on other embodiment of this invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the present embodiment, an image camera that picks up an image and a reduced image such as a gray scale are picked up, and a distance of a subject such as a subject or background displayed at a position on the image (hereinafter, distance information). An imaging apparatus that automatically executes a process of cutting out the contour of a subject displayed in an image captured by an image camera will be described.

FIG. 1 is a block diagram showing a hardware configuration of the imaging apparatus 1 according to the present embodiment. As shown in FIG. 1, the imaging apparatus 1 according to the present embodiment includes the above-described distance camera and image camera in addition to the same configuration as an information processing terminal such as a general server or a PC (Personal Computer). That is, the imaging apparatus 1 according to the present embodiment includes a CPU (Central Processing Unit) 10, a RAM (Random Access Memory) 11, a ROM (Read Only Memory) 12, a HDD (Hard Disk Drive) 13, and an I / F 14. Connected through. Further, an image camera 15, a distance camera 16, an LCD (Liquid Crystal Display) 17, and an operation unit 18 are connected to the I / F 14.

The CPU 10 is a calculation means and controls the operation of the entire imaging apparatus 1. The RAM 11 is a volatile storage medium capable of reading and writing information at high speed, and is used as a work area when the CPU 10 processes information. The ROM 12 is a read-only nonvolatile storage medium, and stores programs such as firmware. The HDD 13 is a nonvolatile storage medium that can read and write information, and stores an OS (Operating System), various control programs, application programs, and the like. The I / F 14 connects and controls the bus 19 and various hardware and networks.

The image camera 15 is an image capturing unit that includes a photoelectric conversion element and converts received light information into electronic information to generate image information. Similar to the image camera 15, the distance camera 16 generates a gray scale image by photoelectric conversion, and measures the distance from the object based on the time until the projected light is reflected by the object and returns. Thus, the distance information generating unit generates information on the distance to the object displayed at each position on the gray scale image. As the distance camera 16, for example, a three-dimensional image distance camera “ZC-1000” series manufactured by Optics Corporation can be used. The LCD 17 is a visual user interface for the user to check the state of the image forming apparatus 1. The operation unit 18 is a user interface such as a keyboard and a mouse for the user to input information to the image forming apparatus 1.

In such a hardware configuration, a program stored in a recording medium such as the ROM 12, the HDD 14, or an optical disk (not shown) is read into the RAM 11, and the CPU 10 performs an operation according to the program, thereby configuring a software control unit. . A functional block that realizes the function of the imaging apparatus 1 according to the present embodiment is configured by a combination of the software control unit configured as described above and hardware.

Next, the functional configuration of the imaging apparatus 1 according to the present embodiment will be described with reference to FIG. FIG. 2 is a block diagram illustrating a functional configuration of the imaging apparatus 1 according to the present embodiment. As illustrated in FIG. 2, the imaging apparatus 1 according to the present embodiment includes a function realized by the image processing unit 100 and a function realized by the display control unit 110. Note that, as described above, the image processing unit 100 and the display control unit 110 function when the software control unit and the hardware realized by the CPU 10 performing calculations according to the program read into the RAM 11 work together. .

The image processing unit 100 executes image processing for cutting out the outline of the subject Q displayed in the image information generated by the image camera 15 based on the distance information acquired by the distance camera 16. As shown in FIG. 2, the image processing unit 100 includes a coordinate conversion unit 101 and an image clipping unit 102.

Here, an example of distance information acquired by the distance camera 16 will be described with reference to FIG. As shown in FIG. 3, the distance information according to the present embodiment includes “u” (pixel) which is a horizontal coordinate on an image generated by the distance camera 16 and “v” (pixel) which is a vertical coordinate. ), And the distance “Z” (mm) from the light receiving surface of the distance camera 16 of the subject or background displayed at the position on the image specified by “u” and “v”. In other words, in the distance information, the coordinates on the image of the grayscale image captured by the distance camera 16 are associated with the distance between the object displayed at each coordinate on the image. With such information, it is possible to recognize the distance in the actual space of the subject Q and the background displayed in the image captured and generated by the distance camera 16.

The coordinate conversion unit 101 converts the coordinate system of the distance information acquired by the distance camera 16 from the coordinate system on the image captured by the distance camera 16 to the coordinate system of the image information captured by the image camera 15. As shown in FIG. 2, the coordinate conversion unit 101 includes coordinate conversion functions of “image plane / three-dimensional space”, “rotation / translation”, “three-dimensional space / image plane”, and “distortion correction”.

The image clipping unit 102 applies a predetermined threshold value to the distance information converted by the coordinate conversion unit 101, so that a pixel whose distance from the camera is within a predetermined range in the image captured by the image camera 15. Is extracted, the outline of the subject Q is cut out. The display control unit 110 causes the LCD 17 to display the subject image clipped by the image clipping unit 102.

Thus, in order to apply the distance information acquired by the distance camera 16 to the image captured by the image camera 15, it is one of the gist according to the present embodiment to convert the coordinate system. As a result, even when the resolution of the image captured by the distance camera 16 is low and an image of a desired image quality level cannot be obtained, or even when the distance camera 16 does not support full color, the image itself is Since the image is captured by the image camera 15, an image having a desired image quality level can be obtained.

Next, each function by the coordinate conversion unit 101 will be described. First, the coordinate conversion function of “image plane / three-dimensional space”, “rotation / translation”, and “three-dimensional space / image plane” will be described, and the coordinate conversion function of “distortion correction” will be described later. First, in order to explain the coordinate conversion function of “image plane / three-dimensional space”, the relationship between the coordinates on the image captured by the camera and the coordinates on the three-dimensional space will be described with reference to FIG.

4 shows the position and image of the subject in a three-dimensional space with the light receiving unit 16a of the distance camera 16 as the origin, the optical axis of the distance camera 16 as the Z axis, the horizontal direction as the Y axis, and the vertical direction as the X axis. It is a figure which shows the coordinate of an image by a perspective projection model. Note that “Z” in FIG. 3 corresponds to the value in the Z-axis direction in FIG. As shown in FIG. 4, an image captured by the camera has a virtual frame (the bold broken line shown in FIG. 4) arranged at the focal length f of the camera when the direction of the optical axis is viewed from the camera. This is the scenery that falls within the frame. The coordinates in the frame are the coordinates “u” and “v” on the image captured by the distance camera 16.

At this time, the light reflected from the imaging target within the frame including the subject Q is condensed toward the light receiving unit 16a of the distance camera 16, as shown in FIG. Therefore, when the optical axis and the Z-axis of the distance camera 16 are the same, and the image, that is, the aspect ratio of the frame in FIG. 4 is 1: 1, a certain point p (on the image captured by the distance camera 16 ( u _i , v _i ) can be expressed by the following equation (1) using a certain point P (X _i , Y _i , Z _i ) and the focal length f of the actual subject Q.

Based on the above equation (1), the coordinate conversion unit 101 calculates the coordinates p (u _i , v _i ) on the image captured by the distance camera 16 as coordinates in the three-dimensional space by calculating the following equation (2). Convert to P (X _i , Y _i , Z _i ).

Here, the matrix of 3 rows and 3 columns including “f _1x ”, “f _1y ”, “c _1x ”, and “c _1y ” in Expression (2) indicates the focal length and the optical axis shift of the distance camera 16. Internal parameter. “F _1x ” and “f _1y ” are the focal lengths of the distance camera 16 in the horizontal direction and the vertical direction, and are the same if the aspect ratio is 1: 1 as described above. “C _1x ” and “c _1y ” are deviations of the optical axis of the distance camera 16 in the horizontal and vertical directions.

The internal parameter of the distance camera 16 is that, for example, a check board is photographed from various angles with the focal length of the distance camera 16 fixed, and the position of the lattice point of the photographed check board is calculated, as in the Zhang method. It can ask for. The coordinate conversion unit 101 stores the internal parameters of the distance camera 16 obtained in this way, and calculates the expression (2) using the internal parameters, whereby an image on the image captured by the distance camera 16 is displayed. converting the coordinates _(u i, _{v i)} the coordinates of a three-dimensional space _{_{_{(X i, Y i, Z}}} i) to.

Next, the coordinate conversion function of “rotation / translation” and “three-dimensional space / image plane” will be described. As described above, the coordinate axes in the three-dimensional space are determined according to each camera. Therefore, the distance camera 16 and the image camera 15 have different coordinate axes as shown in FIG. The “rotation / translation” coordinate conversion function is a process of converting the coordinate system in the three-dimensional space of the distance camera 16 into the coordinate system in the three-dimensional space of the image camera 15.

When converting the coordinates in the three-dimensional space in the distance camera 16 to the coordinates in the three-dimensional space in the image camera 15, the coordinate conversion unit 101 converts the rotation vector “R” in 3 rows and 3 columns and the translation in 3 rows and 1 column. The external parameter “R | t” constituted by the vector “t” is used. Further, the coordinate conversion unit 101 performs a “three-dimensional space / image plane” coordinate conversion process simultaneously with the “rotation / translation” coordinate conversion process.

The “3D space / image plane” coordinate conversion process is performed by converting the coordinates in the 3D space to the coordinates on the image, contrary to the “image plane / 3D space” coordinate conversion realized by the above equation (2). It is processing to convert to. However, in this embodiment, since the purpose is to convert the coordinates on the image captured by the distance camera 16 to the coordinates on the image captured by the image camera 15, the coordinate conversion of “three-dimensional space / image plane” is performed. In the processing, the coordinates converted into the coordinates in the three-dimensional space of the image camera by the “rotation / translation” coordinate conversion processing are converted into the coordinates on the image captured by the image camera 15 using the internal parameters of the image camera 15. Convert. This conversion is realized by the following equation (3).

Here, the matrix of 3 rows and 3 columns including “f _2x ”, “f _2y ”, “c _2x ”, and “c _2y ” in Expression (3) indicates the focal length and the optical axis shift of the image camera 15. Internal parameter. “F _2x ” and “f _2y ” are the focal lengths of the image camera 15 in the horizontal direction and the vertical direction, and are the same if the aspect ratio is 1: 1 as described above. “C _2x ” and “c _2y ” are deviations in the horizontal and vertical optical axes of the image camera 15. The internal parameters of the image camera 15 can be obtained by the Zhang method, for example, in the same manner as the internal parameters of the distance camera 16.

In addition, the matrix including “r ₁₁ ” to “r ₃₃ ” and “t ₁ ” to “t ₃ ” in the expression (3) is the above-described external parameter “R | t”. The external parameter “R | t” can also be obtained by the Zhang method. As described above, the external parameter “R | t” is a parameter for converting the coordinate system of the distance camera 16 to the coordinate system of the image camera 15. Therefore, in order to obtain the external parameter “R | t”, a check board in a certain direction is attached to the image camera 15 and the distance in a state where the image camera 15 and the distance camera 16 are fixed to the imaging apparatus 1 in the same manner as in actual operation. By capturing images with both cameras 16, a set of checkboard images captured by the image camera 15 and the distance camera 16 are obtained.

Since the images of this set of check boards are images of the same check board, the positions of the lattice points can be converted by the external parameter “R | t”. Accordingly, by generating a plurality of sets of images by changing the position of the check board in various ways, the external parameter “R | t” can be obtained so as to solve the simultaneous equations. As described above, the distance camera 16 can generate a gray scale image by imaging. Therefore, when obtaining the internal parameter and the external parameter “R | t”, this gray scale image can be used.

The coordinate conversion unit 101 stores the internal parameter and the external parameter “R | t” of the image camera 101 thus obtained, and by performing the calculation of Expression (3) using the information, Simultaneously realizes "rotation / translation" and "three-dimensional space / image plane" coordinate conversion functions.

By such processing, as shown in FIG. 3, the distance information acquired as the coordinates on the image captured by the distance camera 16 is converted into the coordinates on the image captured by the image camera 15. The coordinate conversion unit 101 inputs distance information corresponding to the image captured by the image camera 15 generated in this way (hereinafter referred to as converted distance information) to the image clipping unit 102.

Next, the clipping process performed by the image clipping unit 102 will be described. FIG. 6A shows a state in which a coordinate point whose distance in the Z-axis direction is specified by the converted distance information is superimposed on an image including a subject imaged by the image camera 15 (hereinafter referred to as a subject image). FIG. Since the resolution when the distance in the Z-axis direction is acquired by the distance camera 16 is lower than the resolution of the image generated by the imaging by the image camera 15, the coordinates of the converted distance information are superimposed on the subject image. As shown in FIG. 6A, the coordinates of the converted distance information are displayed as discrete points.

The image clipping unit 102 extracts only points where the target is within a predetermined distance from the camera by applying a threshold to the distance in the Z-axis direction in the converted distance information. That is, the image cutout unit 102 functions as a coordinate extraction unit. FIG. 6B is a diagram showing a state where the points extracted in this way are superimposed on the subject image. As shown in FIG. 6B, points that overlap the subject are extracted. The points extracted as shown in FIG. 6B are hereinafter referred to as extraction points.

The image cutout unit 102 extracts a subject by erasing a portion that does not overlap with the extraction point from the subject image. However, as described above, since each point of the converted distance information is discrete in the subject image, the extracted point cannot be applied as it is. Therefore, the image cutout unit 102 performs image dilation processing so that each discrete point is connected to form a single region with each discrete point being a white pixel and the other region being a black pixel. Repeat. The image expansion process is a process of replacing a target pixel with a white pixel if even one pixel is present around a target pixel. The image cutout unit 102 repeats this expansion processing until adjacent points in the vertical, horizontal, and diagonal directions are connected.

FIG. 7A is a diagram showing a state in which adjacent points are connected at the extraction point as a result of the expansion process. In the transition from the state of FIG. 6B to the state of FIG. 7A, the image cutout unit 102 performs the smoothing process of the rough outline by the expansion process in addition to the repetition of the expansion process. Further, since the extraction point may appear at a position unrelated to the subject due to the noise of the distance camera 16, the image cutout unit 102 performs the labeling process so that the image clipping unit 102 has a widest area or an area having a predetermined threshold value or more. Leave noise alone and perform noise cut processing.

The image cutout unit 102 deletes a portion other than the region corresponding to the generated region (hereinafter, referred to as an extraction target region) in the subject image as illustrated in FIG. As shown, the subject and the background are separated, and the subject can be extracted. Here, as a result of the expansion process, the extraction target area is wider than the actual contour of the subject as shown in FIG. In FIG. 7B, the portion of the extraction target area that protrudes from the actual subject is shown in black.

It is preferable that the image cutout unit 102 erases an extra area outside the contour of the subject by performing, for example, conventional edge detection processing in the extracted image as shown in FIG. As shown in FIG. 7B, since the image is cut out substantially along the contour of the subject, it is considered that the density between the contour of the subject and the contour of the cut out image is substantially constant. Therefore, as in the prior art, edge detection can be performed with higher accuracy than in detecting the contour of a subject in an image captured and generated by the image camera 15.

Further, the image clipping unit 102 may perform the shrinking process on the extraction target area after generating the extraction target area as shown in FIG. 7A and before clipping the subject image. Contrary to the dilation process, the image contraction process is a process of replacing a target pixel with a black pixel if there is at least one pixel around the target pixel. Thereby, the contour expanded by the expansion process is contracted, and the protrusion from the subject as shown in FIG. 7B can be reduced.

Next, the coordinate conversion function of “distortion correction” of the coordinate conversion unit 101 will be described. FIG. 8A is a diagram illustrating a problem that the coordinate conversion function of “distortion correction” aims at. As shown in FIG. 8A, the extracted point extracted by applying a threshold value to the converted distance information may deviate from the subject in the subject image. This may be caused by radial and circumferential distortions in the camera lens. Therefore, when the coordinate conversion unit 101 converts the distance information acquired by the distance camera 16 to generate converted distance information, the coordinate conversion unit 101 performs conversion by correcting the distortion. In the present embodiment, correction is performed assuming that the lens of the distance camera 16 is distorted.

The coordinate conversion unit 101 according to the present embodiment also performs a “distortion correction” process in the calculation of Expression (3) described above. Here, the calculation of the above equation (3) is equivalent to the following equations (4) to (6). However, z ≠ 0.

On the other hand, considering the distortion of the lens, the above equation (6) is replaced by the following equations (7) and (8).

Here, “k ₁ ”, “k ₂ ” and “p ₁ ”, “p ₂ ” in the equation (7) are distortion coefficients in the radial direction and the circumferential direction, respectively. That is, Expression (7) is an expression for correcting distortion caused by the lens. In the present embodiment, the case where the coefficients expanded up to the second order are considered as an example. However, the coefficients up to the third order or higher may be considered. These distortion coefficients can also be obtained by calibration. That is, based on the images of a plurality of check boards generated when the internal parameters of the distance camera 16 described above are obtained, the positions of the respective grid points are applied to the above formulas to calculate “k ₁ ”, “k ₂ ”. ", And distortion coefficients of" p ₁ "and" p ₂ "can be obtained.

It should be noted that the dimension considering the above-described coefficients is preferably determined according to the distance between the camera and the subject. In general, the closer the distance between the camera and the subject, the greater the distortion. Therefore, as the distance between the camera and the subject is shorter, the distortion correction can be performed more suitably by performing calculation in consideration of higher-order coefficients.

The coordinate conversion unit 101 stores the distortion coefficient obtained in this way, and receives the three-dimensional coordinates (X _i , Y _i , Z _i ) in the distance camera 16 as an input, and the image camera according to the above equation (3). When the coordinates (u _j , v _j ) on the image obtained by the imaging of 15 are obtained, the distortion of the lens is corrected by using the above formulas (4), (5), (7), and (8). The coordinates on the image obtained by the image camera 15 can be obtained. Thereby, as shown in FIG.8 (b), the shift | offset | difference of an extraction point and a to-be-photographed object can be eliminated.

As described above, in the imaging apparatus 1 according to the present embodiment, the distance information acquired by the distance camera 1 is not used in principle when the portion where the subject is displayed is clipped from the subject image, without using the information on the image density. Process based on. Further, in the imaging apparatus 1 according to the present embodiment, the image processing unit 100 automatically executes a process based on given information without obtaining an operation from the user. Therefore, in the image clipping process, it is possible to perform a more accurate clipping process regardless of the skill level of the user.

In the above embodiment, as illustrated in FIG. 2, the imaging apparatus 1 including the image camera 15 and the distance camera 16 has been described as an example. However, the image processing unit 100 alone or a program for realizing the image processing unit 100 is described. It is also possible to provide as. In this case, it is necessary to separately acquire the internal parameters of the first camera that captured the subject image, the internal parameters of the second camera that acquired the distance information, and the external parameters of the first camera and the second camera.

As an external parameter acquisition method, in addition to the above-described stereo calibration, if the image camera 15 and the distance camera 16 are equipped with a positioning system such as GPS (Global Positioning System) and a high-accuracy one, It is also possible to use that information. Specifically, when the image camera 15 and the distance camera 16 acquire the subject image and the distance information, respectively, the position and orientation at the time of information acquisition are simultaneously acquired by the mounted positioning system and input to the coordinate conversion unit 101. .

Accordingly, the coordinate conversion unit 101 obtains external parameters for converting the coordinate system of the three-dimensional space in the distance camera 16 to the coordinate system of the three-dimensional space in the image camera 15 based on the input position and orientation information. be able to. Of the input information, the rotation vector R can be obtained from the difference in orientation, and the translation vector t can be obtained from the position.

Further, in the above-described embodiment, as illustrated in FIGS. 7A and 7B, the case where a portion other than the region corresponding to the extraction target region is deleted from the subject image is described as an example. In addition, the area corresponding to the extraction target area and other areas can be stored as separate layers. That is, the image clipping unit 102 functions as an image separation unit that separates at least the region of the subject image specified by the extraction target region and the other region and stores the separated region in the storage medium. It is possible to obtain As a result, the user can select whether or not the background image is specified in subsequent operations, and the convenience of the user can be improved.

In the above embodiment, when the coordinates of the converted distance information are superimposed on the subject image, the points specified by the coordinates of the converted distance information are converted as shown in FIGS. 6 (a) and 6 (b). The case where the distance information is arranged on the subject image at an interval corresponding to the resolution of the distance information, that is, the ratio between the resolution of the distance camera 16 and the resolution of the subject image has been described as an example. In other words, in the examples of FIGS. 6A and 6B, each point specified by the coordinates of the converted distance information is set at an interval according to the ratio between the resolution of the distance camera 16 and the resolution of the subject image. Correspond to each pixel on the subject image. That is, since the coordinates of the converted distance information are superimposed on the subject image having a higher resolution while maintaining the resolution as it is, a discrete state as shown in FIGS. 6A and 6B is obtained.

On the other hand, it is also possible to perform superimposition after making the resolution of the converted distance information correspond to the resolution of the subject image in advance. For example, the resolution of the converted distance information can be matched with the resolution of the subject image by dividing each pixel by using each point specified by the coordinates of the converted distance information as a pixel. Such an embodiment will be described below.

FIG. 9A is a diagram illustrating a state in which the distance information illustrated in FIG. 3 is converted into converted distance information by the coordinate conversion unit 101. As shown in FIG. 9A, the coordinates specified as (u ₁ , v ₁ ), (u ₂ , v ₂ ), etc. in FIG. 3 are converted to (u ′ ₁ , v ′ ₁ ), (u ′ ₂ , v ′ ₂ )... FIG. 9B is a diagram illustrating a state in which each converted coordinate shown in FIG. 9A is a pixel and each pixel is divided into four.

The points specified as (u ′ ₁ , v ′ ₁ ) in FIG. 9A are (u ′ ₁₁ , v ′ ₁₁ ), (u ′ ₁₂ , v ′ ₁₂ ) shown in FIG. This corresponds to four points (u ′ ₁₃ , v ′ ₁₃ ) and (u ′ ₁₄ , v ′ ₁₄ ). As shown in FIGS. 10A and 10B, each point has four pixels arranged by dividing the pixel specified by (u ′ ₁ , v ′ ₁ ) in the original resolution into two vertically and horizontally. It corresponds to.

By such processing, distance information of the same resolution is generated so that all pixels in the subject image are covered by 1: 1 instead of discrete points as shown in FIGS. 6 (a) and 6 (b). be able to. Further, as shown in FIG. 9B, the distance “Z ₁ ” before the division is associated with each of the four points after the division. For this reason, the image clipping unit 102 applies a threshold value to the distance Z and extracts only points where the target is within a predetermined distance, and the extraction result is substantially the contour of the subject as shown in FIG. In the matched state, the adjacent dots are filled, and the extraction target area can be suitably obtained.

9A, 9B, 10A, and 10B, a process for smoothing a rough outline due to pixel division, a labeling process for noise cutting, and the like are performed. It is preferable. 9A, 9B, 10A, and 10B, the extraction target area does not completely match the actual subject contour, and the extraction target region is the contour of the subject. Since there is a case of protruding from the outside, as in the above embodiment, conventional edge detection processing or the like may be performed to erase the extra area outside the contour of the subject. Even in this case, since the image is cut out substantially along the contour of the subject, the edge is detected with higher accuracy than in the case of detecting the contour of the subject in the image captured and generated by the image camera 15 as in the prior art. It is the same that detection is possible.

1 Imaging device 10 CPU
11 RAM
12 ROM
13 HDD
14 I / F
15 Image camera 16 Distance camera 17 LCD
18 Operation Unit 19 Bus 100 Image Processing Unit 101 Coordinate Conversion Unit 102 Image Clipping Unit 110 Display Control Unit

Claims

An image capturing unit that generates a subject image on which a subject and a background are displayed by capturing;
When the visual range including the subject and the background is used as an image, the distance to the object displayed in each part is measured, and distance information in which the coordinates on the image of the visual range and the distance are associated is obtained. A distance information generation unit to generate;
A coordinate conversion unit that converts the coordinates of the acquired distance information into coordinates on the subject image to generate converted distance information;
Among the converted coordinates included in the generated converted distance information, a coordinate extracting unit that extracts coordinates for which the associated distance satisfies a predetermined condition;
An image pickup apparatus comprising: an image separation unit that separates and outputs a region specified by the extracted coordinates and another region in the subject image.
The coordinate converter is
Based on the first parameter including the focal length and optical axis information of the distance information generation unit, the coordinate of the distance information, which is the coordinate on the image of the visual range, is a cubic with the distance information generation unit as a reference A first coordinate conversion function for converting to coordinates in the original space;
Based on the second parameter based on the deviation between the coordinate axis in the three-dimensional space with reference to the distance information generation unit and the coordinate axis in the three-dimensional space with reference to the image capturing unit, the distance information generation unit is used as a reference. A second coordinate conversion function for converting the coordinates of the distance information converted into coordinates on the three-dimensional space into coordinates on the three-dimensional space with the image capturing unit as a reference;
Based on the second parameter including the focal length and optical axis information of the image capturing unit, the coordinates of the distance information converted into the coordinates in the three-dimensional space with the image capturing unit as a reference are displayed on the subject image. The imaging apparatus according to claim 1, further comprising: a third coordinate conversion function that converts the coordinates into a plurality of coordinates.
The coordinate conversion unit generates the distance information based on a fourth parameter including information on at least one of a radial distortion and a circumferential distortion of a lens included in the distance information generation unit or the image capturing unit. The coordinates in the three-dimensional space with reference to the part or the coordinates of the distance information converted into the coordinates in the three-dimensional space with reference to the image capturing part are used as the radial distortion or the circumferential distortion of the lens. The imaging apparatus according to claim 2, further comprising a fourth coordinate conversion function for converting the coordinates into corrected coordinates.
The coordinate extraction unit extracts coordinates whose associated distance is equal to or less than a predetermined threshold among the converted coordinates included in the generated converted distance information. The imaging apparatus of Claim 1.
The image separation unit extracts an area where the subject is displayed in the subject image by erasing image information of a region other than the region specified by the extracted coordinates in the subject image. The imaging apparatus according to claim 4, wherein the imaging apparatus is characterized.
The resolution of coordinates in the distance information is lower than the resolution of the subject image,
The image separation unit divides the pixels in an image drawn with the extracted coordinates as pixels, thereby causing the resolution of the extracted coordinates to correspond to the resolution of the subject image, and draws with the divided pixels. The imaging apparatus according to claim 1, wherein a region is specified by an image to be processed.
When the visual range including the subject and the background is used as an image, the distance to the object displayed in each part is measured, and distance information in which the coordinates on the image of the visual range and the distance are associated is obtained. A distance information acquisition unit to acquire;
A subject image acquisition unit for acquiring a subject image in which a subject and a background are displayed;
A coordinate conversion unit that converts the coordinates of the acquired distance information into coordinates on the subject image to generate converted distance information;
Among the converted coordinates included in the generated converted distance information, a coordinate extracting unit that extracts coordinates for which the associated distance satisfies a predetermined condition;
An image processing apparatus comprising: an image separation unit that separates and outputs a region specified by the extracted coordinates and another region of the acquired subject image.
When the visual range including the subject and the background is used as an image, the distance to the object displayed in each part is measured, and distance information in which the coordinates on the image of the visual range and the distance are associated is obtained. Get it and store it in a storage medium,
Obtain a subject image displaying the subject and background, store it in a storage medium,
Converting the coordinates of the stored distance information into coordinates on the subject image to generate converted distance information and storing it in a storage medium;
Among the converted coordinates included in the generated converted distance information, a coordinate whose associated distance satisfies a predetermined condition is extracted, and specified by the extracted coordinates in the acquired subject image An image processing method characterized in that an area to be processed and another area are separated and stored in a storage medium.
When the visual range including the subject and the background is used as an image, the distance to the object displayed in each part is measured, and distance information in which the coordinates on the image of the visual range and the distance are associated is obtained. Obtaining and storing in a storage medium;
Acquiring a subject image on which a subject and a background are displayed and storing the subject image in a storage medium;
Converting the coordinates of the stored distance information into coordinates on the subject image to generate converted distance information and storing it in a storage medium;
Among the converted coordinates included in the generated converted distance information, a coordinate whose associated distance satisfies a predetermined condition is extracted, and specified by the extracted coordinates in the acquired subject image An image processing program for causing an information processing apparatus to execute a step of separating an area to be processed and another area and storing the separated area in a storage medium.