JP7300895B2 - Image processing device, image processing method, program, and storage medium - Google Patents

Description

The present invention relates to technology for estimating attribute information of an object in an image.

In the observation of wild animals and the growth management of livestock, it is sometimes difficult to directly measure the size (volume) and mass of objects, and non-contact acquisition of animal attribute information is required.

A pattern projection method, a multi-view photographing method, a TOF (Time of Flight) method, and the like are known as methods for non-contact shape measurement of an object surface. Patent Document 1 proposes a method for estimating a volume from a surface shape measured without contact. In Patent Document 1, in addition to surface shape measurement using the light section method as a method for estimating the volume of crops, the hidden surface of the crop is assumed to be the volume up to the ground surface, and then the volume is corrected by the correction coefficient to estimate the volume approximately. A method to do so is proposed.

It is difficult to control the movement of animals as compared to stationary objects such as crops. In order to obtain attribute information such as animal dimensions, it is necessary to know the animal's posture. Patent Document 2 proposes a method of estimating the orientation of a target object by extracting feature points from an image captured using an imaging device and comparing them with feature amounts acquired through learning in advance.

Japanese Patent No. 5686058 Japanese Patent No. 4449410

In Patent Document 1, the back surface shape of the object to be measured is not measured or estimated. and the accuracy of mass estimation decreases.

In the observation of wild animals and the growth management of livestock, it is difficult to control the posture of the target object when the target is an animal. . In Patent Document 2, a target object is recognized by estimating the orientation of the target object by comparing with pre-learned feature points for recognizing the target object. (mass, etc.) are not estimated.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and an object thereof is to realize a technique capable of estimating attribute information of an object in an image.

In order to solve the above problems and achieve the object, an image processing apparatus of the present invention includes: depth generation means for generating depth information indicating a distance distribution of a subject from a captured image of the subject; object detection means for detecting an area of a specific object from a captured image; posture estimation means for estimating a posture of the specific object; Attitude transforming means for transforming into a specific pose suitable for estimating attribute information; and attribute information estimating means for estimating attribute information of the specific object from the taken image whose pose has been transformed, depth information, and imaging conditions of the image. and have

According to the present invention, it is possible to estimate attribute information of an object in an image.

FIG. 1A is a block diagram showing the device configuration of Embodiment 1, FIG. 1B is a diagram showing the pixel array of an image sensor, and FIG. FIG. 2 is a schematic diagram for explaining the relationship between an image sensor, an optical system, and an image according to Embodiment 1; 4 is a flowchart showing attribute information estimation processing according to the first embodiment; 4 is a view exemplifying a target object selection screen according to the first embodiment; FIG. 4A and 4B are diagrams for explaining an example of posture estimation processing according to the first embodiment; FIG. 4A and 4B are views showing an example of attitude conversion processing and dimension measurement positions according to the first embodiment; FIG. 4A and 4B are diagrams for explaining an example of a method of inputting a dimension measurement position according to the first embodiment; FIG. 10 is a flowchart showing attribute information estimation processing according to the second embodiment; 10 is a flowchart showing posture estimation/conversion processing according to the second embodiment;

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In addition, the following embodiments do not limit the invention according to the scope of claims. Although multiple features are described in the embodiments, not all of these multiple features are essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant description is omitted.

[Embodiment 1] Embodiment 1 will be described below.
An example of an embodiment in which the present invention is applied to a digital camera capable of acquiring depth information indicating the distance distribution of a subject as an example of an image processing apparatus will be described below. However, the present invention can be applied to any device capable of estimating object attribute information (size, shape, volume, mass, etc.) based on a captured image, depth information corresponding to the captured image, and image capturing conditions. It is possible.

<Construction of Digital Camera> First, the construction and functions of a digital camera 100 according to the present embodiment will be described with reference to FIG.

The imaging optical system 10 is a photographing lens included in the digital camera 100 and forms an optical image of a subject on the imaging device 11 . The imaging optical system 10 is composed of a plurality of lenses (not shown) arranged on an optical axis 102 and has an exit pupil 101 at a predetermined distance from the imaging device 11 . In this specification, the direction parallel to the optical axis 102 is defined as the z direction or depth direction, the direction orthogonal to the optical axis 102 and parallel to the horizontal direction of the image sensor 11 is defined as the x direction, and the vertical direction of the image sensor 11 is defined as the x direction. Let us define the parallel direction as the y-direction, or provide an axis.

The imaging device 11 is, for example, a CCD (charge coupled device) or a CMOS sensor (complementary metal oxide semiconductor). The imaging device 11 photoelectrically converts a subject image formed on an imaging surface via the imaging optical system 10 and outputs an image signal related to the subject image. Further, in the present embodiment, the imaging device 11 has a ranging function of an imaging surface phase difference ranging method as described later, and in addition to the captured image, the distance from the imaging device to the subject (subject distance) is measured. It is possible to generate and output the indicated distance information.

The control unit 12 is a control device such as a CPU or a microprocessor, and controls the operation of each block included in the digital camera 100 . For example, the control unit 12 performs autofocus (AF: automatic focusing) at the time of imaging, change of focus position, change of F number (aperture), capture of image, storage unit 14, input unit 15, display unit 16, communication 17 is controlled.

The image processing device 13 is a block that implements various image processing of the digital camera 100 . As illustrated, the image processing device 13 includes image processing blocks of an image generation unit 130, a depth generation unit 131, an object detection unit 132, an orientation estimation unit 133, an orientation conversion unit 134, an attribute information estimation unit 135, and an image processing block. and a memory 136 used as a work area. The image processing device 13 can be configured using a logic circuit. As another form, it may be composed of a central processing unit (CPU) and a memory for storing an arithmetic processing program.

The image generation unit 130 performs various signal processing such as noise removal, demosaicing, luminance signal conversion, aberration correction, white balance adjustment, and color correction of the image signal output from the imaging device 11 . The image data (captured image) output from the image generating section 130 is accumulated in the memory 136 and used by the object detecting section 132 and the display section 16 .

The depth generation unit 131 generates a depth image representing the distribution of depth information based on signals obtained from ranging pixels of the imaging device 11, which will be described later. Here, the depth image is two-dimensional information in which the value stored in each pixel is the subject distance of the subject existing in the area of the captured image corresponding to the pixel.

The object detection unit 132 uses the captured image generated by the image generation unit 130 to detect in advance an object to be measured included in the captured image, and specifies the position and size in the captured image. If the type of object to be measured is not specified in advance, the object detection unit 132 identifies the type. Note that, in this embodiment, the target object is an animal other than a human being.

The posture estimation unit 133 acquires the posture of the target object in the object region detected by the object detection unit 132 by learning in advance and estimates it using information stored in the storage unit 14 .

The posture transforming unit 134 transforms the posture of the target object estimated by the posture estimating unit 133 in the viewing image and the depth image into a specific posture suitable for estimating attribute information. The specific orientation differs depending on the target object, and is determined from the orientation information stored in the storage unit 14 and/or the memory 136 in advance based on the type of object specified in advance or the object information specified by the object detection unit 132 .

The attribute information estimation unit 135 calculates at least one of size, shape, volume, and mass as attribute information of the target object from the viewing image and the depth image in which the posture of the target object has been converted to a specific posture by the posture conversion unit 134. presume. In dimension estimation, the position where the dimension is measured differs depending on the target object. Therefore, based on the type of object specified in advance or the type of object specified by the object detection unit 132, the information for dimension measurement stored in advance in the storage unit 14 and/or the memory 136 is used for estimation. In the shape estimation, the surface shape is obtained from the depth image whose posture has been transformed, and the three-dimensional shape of the target object stored in advance in the storage unit 14 and/or the memory 136 is obtained according to the type of object identified by the object detection unit 132. Make an estimate by referring to the data. In volume estimation, estimation is performed from the three-dimensional shape of the target object obtained by shape estimation and imaging parameters. In mass estimation, estimation is performed using density information according to the volume obtained by volume estimation and the target object. Density information is measured in advance for each object and stored in the storage unit 14 .

The storage unit 14 is a non-volatile recording medium in which captured image data, intermediate data generated during the operation of each block, parameters referred to in the operation of the image processing device 13 and the digital camera 100, and the like are recorded. be. The storage unit 14 may be any recording medium that can be read and written at high speed and has a large capacity as long as it guarantees an acceptable processing performance for realizing processing. A flash memory or the like is desirable.

The input unit 15 is, for example, a user interface such as a dial, button, switch, or touch panel that detects information input or setting change operation input to the digital camera 100 . The input unit 15 outputs a corresponding control signal to the control unit 12 when detecting the operation input made.

The display unit 16 is, for example, a display device such as a liquid crystal display or organic EL. The display unit 16 is used for confirming the composition at the time of photographing by displaying a through display of the captured image, and for notifying various setting screens and message information. In this embodiment, the display unit 16 also displays the detection result of the object and the estimation result such as shape, volume, mass, and the like.

The communication unit 17 is a communication interface provided in the digital camera 100 for realizing information transmission/reception with the outside. The communication unit 17 may be configured to be able to transmit the obtained captured image, depth information, estimation result of subject attribute information (size, shape, volume, mass), etc. to another device.

<Structure of Imaging Device> Next, the detailed structure of the imaging device 11 of the present embodiment will be described with reference to FIGS. 1(b) and 1(c).

As shown in FIG. 1B, the imaging device 11 is configured by connecting and arranging a plurality of pixel groups 110 of 2 rows×2 columns to which different color filters are applied. As shown in the enlarged illustration, the pixel group 110 has red (R), green (G), and blue (B) color filters arranged therein. An image signal indicating any one of color information is output. In this embodiment, as an example, the color filters are distributed as shown in the figure, but it will be easily understood that the implementation of the present invention is not limited to this.

In the imaging element 11 of the present embodiment, one pixel (photoelectric conversion element) is arranged in the horizontal direction of the imaging element 11, as shown in FIG. A plurality of photoelectric conversion units are arranged side by side in the II′ cross section. More specifically, as shown in FIG. 1C, each pixel includes a light guide layer 113 including a microlens 111 and a color filter 112, and a first photoelectric conversion unit 115 and a second photoelectric conversion unit 116. Contain and consist of.

In the light guide layer 113 , the microlenses 111 are configured to efficiently guide the light beams incident on the pixels to the first photoelectric conversion unit 115 and the second photoelectric conversion unit 116 . The color filter 112 passes light in a predetermined wavelength band, and passes only light in one of the R, G, and B wavelength bands described above. 2 photoelectric conversion unit 116 .

The light-receiving layer 114 is provided with two photoelectric conversion units (a first photoelectric conversion unit 115 and a second photoelectric conversion unit 116) that convert received light into analog image signals. Two types of signals output from are used for ranging. That is, each pixel of the image sensor 11 similarly has two photoelectric conversion units arranged in the horizontal direction, and an image composed of signals output from the first photoelectric conversion unit 115 of all the pixels. An image signal composed of a signal and a signal output from the second photoelectric conversion unit 116 is used. In other words, the first photoelectric conversion unit 115 and the second photoelectric conversion unit 116 each partially receive the light flux entering the pixel via the microlens 111 . Therefore, the two types of image signals finally obtained are a group of pupil-divided images related to light fluxes that have passed through different areas of the exit pupil of the imaging optical system 10 . Here, a combination of image signals photoelectrically converted by the first photoelectric conversion unit 115 and the second photoelectric conversion unit 116 in each pixel can be obtained in a mode in which only one photoelectric conversion unit is provided in the pixel. This is equivalent to an image signal (for viewing) output from one photoelectric conversion unit.

With such a structure, the imaging device 11 of the present embodiment can output an image signal for viewing and an image signal for distance measurement (two types of pupil-divided images). In this embodiment, all the pixels of the image sensor 11 are provided with two photoelectric conversion units, and are configured to output high-density depth information. is not limited to

<Ranging principle of imaging surface phase difference ranging method>
Here, the principle of deriving the subject distance based on the group of pupil-divided images output from the first photoelectric conversion unit 115 and the second photoelectric conversion unit 116 performed by the digital camera 100 of the present embodiment will be described with reference to FIG. 2 for explanation.

FIG. 2A is a schematic diagram showing the exit pupil 101 of the imaging optical system 10 and the luminous flux received by the first photoelectric conversion units 115 of the pixels in the imaging element 11 . FIG. 2B is a schematic diagram similarly showing a light beam received by the second photoelectric conversion unit 116. As shown in FIG.

The microlenses 111 shown in FIGS. 2A and 2B are arranged such that the exit pupil 101 and the light receiving layer 114 are in an optically conjugate relationship. A light beam that has passed through the exit pupil 101 of the imaging optical system 10 is condensed by the microlens 111 and guided to the first photoelectric conversion unit 115 or the second photoelectric conversion unit 116 . At this time, as shown in FIGS. 2A and 2B, the first photoelectric conversion unit 115 and the second photoelectric conversion unit 116 mainly receive light beams that have passed through different pupil regions. The first photoelectric conversion unit 115 receives the luminous flux that has passed through the first pupil region 210 , and the second photoelectric conversion unit 116 receives the luminous flux that has passed through the second pupil region 220 .

A plurality of first photoelectric conversion units 115 included in the imaging device 11 mainly receive light beams that have passed through the first pupil region 210 and output first image signals. At the same time, the plurality of second photoelectric conversion units 116 included in the image sensor 11 mainly receive the light beams that have passed through the second pupil region 220, and output second image signals. The intensity distribution of the image formed on the imaging device 11 by the light flux passing through the first pupil region 210 can be obtained from the first image signal. In addition, the intensity distribution of the image formed on the imaging element 11 by the light flux passing through the second pupil region 220 can be obtained from the second image signal.

A relative positional deviation amount (so-called parallax amount) between the first image signal and the second image signal has a value corresponding to the defocus amount. The relationship between the amount of parallax and the amount of defocus will be described with reference to FIGS. 2(c), (d), and (e) are schematic diagrams explaining the imaging element 11 and the imaging optical system 10 of this embodiment. Reference numeral 211 in the drawing denotes the first light flux passing through the first pupil region 210, and reference numeral 221 denotes the second light flux passing through the second pupil region 220. FIG.

FIG. 2C shows a state at the time of focusing, in which the first luminous flux 211 and the second luminous flux 221 converge on the imaging device 11. FIG. At this time, the amount of parallax between the first image signal formed by the first light flux 211 and the second image signal formed by the second light flux 221 is zero. FIG. 2D shows a state in which the image side is defocused in the negative direction of the z-axis. At this time, the amount of parallax between the first image signal formed by the first light flux and the second image signal formed by the second signal is not 0, but has a negative value. FIG. 2E shows a state in which the image side is defocused in the positive direction of the z-axis. At this time, the amount of parallax between the first image signal formed by the first light flux and the second image signal formed by the second light flux has a positive value. From the comparison between FIG. 2(d) and FIG. 2(e), it can be seen that the direction of positional deviation changes depending on whether the defocus amount is positive or negative. Also, it can be seen that positional deviation occurs according to the imaging relationship (geometric relationship) of the imaging optical system according to the defocus amount. The amount of parallax, which is the positional deviation between the first image signal and the second image signal, can be detected by a region-based matching method, which will be described later.

<Attribute Information Estimation Processing> Next, the processing for estimating the attribute information of the target object from the captured image executed by the digital camera 100 of the present embodiment will be described with reference to the flowchart of FIG. 3(a). Note that the processing corresponding to the flowchart of FIG. It can be realized by controlling each part of the camera 100 . The same applies to FIGS. 8 and 9, which will be described later.

In S301, the control unit 12 selects an object to be measured. A list of objects to be measured is displayed on the display unit 16 at the time of photographing, and the input unit 15 allows the user to select a desired object. As a result, it is possible to omit the estimation of the type of the object and prevent erroneous recognition. Here, the input unit 15 and the display unit 16 are configured separately, but the display unit 16 may be configured to have the function of the input unit 15 by using a touch panel or the like. FIG. 4 shows a display example of a list of objects to be measured. In FIG. 4, animal classifications and detailed animal types for each classification are displayed in a selectable manner, and the user may select one of the displayed options. If the object to be photographed does not correspond to the type of object prepared in advance, for example, as shown in FIG. 4, an option of "Others" may be displayed, and attribute information estimation may be performed using general parameters.

In S302, the control unit 12 performs processing to perform imaging with imaging settings such as the set focal position, aperture, and exposure time. More specifically, the control unit 12 controls the imaging device 11 to perform imaging, transmit the obtained captured image to the image processing device 13 , and store it in the memory 136 . Here, the captured image is composed of an image signal S1 composed of signals output only from the first photoelectric conversion unit 115 of the image sensor 11 and a signal output only from the second photoelectric conversion unit 116. It is assumed that there are two types of image signals S2.

In S303, the image processing device 13 generates a viewing image and a depth image from the obtained captured image. More specifically, the image generation unit 130 in the image processing device 13 first generates one Bayer array image by adding the pixel values of the pixels of the image signal S1 and the image signal S2. The image generation unit 130 performs demosaicing processing on the images of each of R, G, and B colors on the Bayer array image to generate an image for viewing. The demosaicing process is performed according to the color filters arranged on the image pickup device, and it goes without saying that any demosaicing method may be used. In addition, the image generation unit 130 performs processing such as noise removal, luminance signal conversion, aberration correction, white balance adjustment, and color correction, generates a final viewing image, and stores it in the memory 136 .

<Depth image generation processing>
On the other hand, the depth image is generated by the depth generation unit 131 . Here, processing related to depth image generation will be described with reference to the flowchart of FIG. 3(b).

In S311, the depth generation unit 131 performs light amount correction processing on the image signal S1 and the image signal S2. At the peripheral angle of view of the imaging optical system 10, due to vignetting, the shapes of the first pupil region 210 and the second pupil region 220 are different, and the light amount balance between the image signals S1 and S2 is lost. ing. Therefore, in this step, the depth generation unit 131 performs light amount correction on the image signal S1 and the image signal S2 using, for example, light amount correction values stored in the storage unit 14 and/or the memory 136 in advance.

In S<b>312 , the depth generation unit 131 performs processing to reduce noise generated during conversion in the image sensor 11 . Specifically, the depth generator 131 implements noise reduction by applying filtering to the image signal S1 and the image signal S2. In general, the higher the spatial frequency, the lower the SN ratio, and the more noise components there are. Therefore, the depth generation unit 131 performs processing to apply a low-pass filter whose pass rate decreases as the spatial frequency increases. Note that the light amount correction in S311 may not produce a desirable result due to manufacturing errors of the imaging optical system 10, etc. Therefore, the depth generation unit 131 uses a band-pass filter that cuts off DC components and has a low pass rate of high-frequency components. It is desirable to apply

In S313, the depth generator 131 derives the amount of parallax between these images based on the image signal S1 and the image signal S2. Specifically, the depth generator 131 sets a target point corresponding to the representative pixel information and a collation area centering on the target point in the image signal S1. The collation area may be, for example, a rectangular area such as a square area having one side of a predetermined length centered on the point of interest. Next, the depth generator 131 sets a reference point in the image signal S2 and sets a reference area centered on the reference point. The reference area has the same size and shape as the matching area described above. The depth generation unit 131 sequentially moves the reference points to derive the degree of correlation between the image included in the matching region of the image signal S1 and the image included in the reference region of the image signal S2. A high reference point is identified as a corresponding point corresponding to the point of interest in the image signal S2. The amount of relative positional deviation between the corresponding point specified in this way and the point of interest is the amount of parallax at the point of interest.

The depth generation unit 131 calculates the amount of parallax while sequentially changing the point of interest according to the representative pixel information in this way, thereby deriving the amount of parallax at a plurality of pixel positions determined by the representative pixel information. In this embodiment, for simplicity, the pixel positions (pixel groups included in the representative pixel information) for calculating the amount of parallax are set to be the same number as the viewing image in order to obtain the depth information with the same resolution as the viewing image. It shall be Methods such as NCC (Normalized Cross-Correlation), SSD (Sum of Squared Difference), and SAD (Sum of Absolute Difference) may be used as methods for deriving the degree of correlation.

Also, the derived amount of parallax can be converted into a defocus amount, which is the distance from the imaging element 11 to the focal point of the imaging optical system 10, by using a predetermined conversion coefficient. Here, assuming a predetermined conversion coefficient K and a defocus amount ΔL, the parallax amount d can be converted into a defocus amount by the following equation 1.

(Formula 1)
ΔL=K×d
Furthermore, the defocus amount ΔL can be converted into the object distance by using the following equation 2, which is a lens formula in geometrical optics.
(Formula 2)
1/A+1/B=1/F
Here, A is the distance (subject distance) from the object plane to the principal point of the imaging optical system 10, B is the distance from the principal point of the imaging optical system 10 to the image plane, and F is the focal length of the imaging optical system 10. shall be That is, in the lens formula, the value of B can be derived from the defocus amount ΔL, so the distance A from the object to the object surface can be derived based on the setting of the focal length at the time of imaging.

The depth generation unit 131 constructs two-dimensional information in which the derived object distance is used as a pixel value, and stores it in the memory 136 as a depth image.

On the other hand, in S304, the object detection unit 132 detects a target object region. Based on the type of target object selected in S301, the object detection unit 132 identifies the target object region using information acquired in advance and stored in the storage unit 14, and detects the contour of the identified region. Extract the target object along In this case, the depth image can be used to assist extraction of the target object region. A specific or constant value is set to values other than the extracted target object region, and an object extraction image in which only the target object region is left is generated. Similarly, in the depth image, areas other than the target object area are replaced with specific or constant values to generate an object extraction depth image in which only the target object area has valid values. The object extraction image and the object extraction depth image are stored in memory 136 and used for subsequent processing. Various machine learning methods such as deep learning can be used as the learning method for extracting the target object region, but the method is not limited to a specific method, and any method may be used.

In S305, the posture estimation unit 133 estimates the posture of the target object in the object extraction image. The posture estimating unit 133 extracts feature points in the region in the object extraction image from the detection result of the target object region in S304, and obtains the three-dimensional shape stored in the storage unit 14 by learning in advance. The posture is estimated using feature point data. Furthermore, by using the object extraction depth image, it becomes possible to estimate the posture change in more detail. Pose estimation mainly uses object extraction images to estimate information such as where the head, body, legs, and tail of an animal as a target object are located, and which direction the head is facing. Also, the orientation of the entire target object viewed from the photographing direction can be determined from the orientation of the body. An object extraction depth image is used to determine the orientation of the torso. The normal direction of each pixel is calculated from the object extraction depth image, and the normal direction of the body portion is obtained using the information of the body position. Since the body of an animal is a curved surface, the normal direction is not constant. Therefore, a principal component analysis or the like is performed to calculate the main normal direction. A plane perpendicular to this main normal direction is estimated as a plane representing the orientation of the target object. For example, as shown in FIG. 5, a plane Pv that cuts vertically from the head to the tail of the target animal and a plane Ph that cuts horizontally from the head to the tail are estimated, and used for posture conversion, which will be described later.

In the present embodiment, target object region detection and orientation estimation are performed separately, but target object region detection and object orientation estimation may be performed simultaneously using machine learning.

In S306, the posture transformation unit 134 uses the posture of the target object estimated in S305 to transform the posture of the target object in the object extraction image and the posture transformation in the object extraction depth image. For example, when estimating the size of an animal as the attribute information of the object from the extracted object image, the animal has a specific posture that is easy to measure depending on the type. For example, in the case of a mammal, as shown in FIG. 6, the photographed image (FIG. 6(a)) is converted into an image photographed from the side (FIG. 6(b)). - It becomes easy to measure a dimension such as body height Lc.

Birds and fishes should also be converted so that they are captured from the side. In the case of birds, dimensions such as total length La and wing length Ld are measured as shown in FIG. 6(c). However, in the case of birds with spread wings as shown in FIG. 6(d), it is desirable to convert the image into an image viewed from above, and the wing span Le is measured. In addition, it is desirable that arthropods such as reptiles, amphibians, and insects should also be a bird's-eye view image based on the information. However, for any animal, it is desirable to appropriately convert the image to an image from the side or an image from above, depending on the posture of the target object in the photographed image.

Ｐは変換前の画像上の位置（ｘ、ｙ、ｚ）を意味し、Ｐ’は変換後の画像上の位置である。変換により欠落した画素位置の情報は周辺の画素の情報から補間することで欠落のない変換画像を生成する。 Attitude transformation here basically transforms an image by geometric transformation. Taking the case of mammals as an example, the vertical cutting plane Pv representing the animal's posture estimated in the posture estimation is used, and the rotation angle is calculated so that the normal to this plane is perpendicular to the photographing device 1. . A rotation matrix R is generated from the obtained rotation angles, and the object extraction image and the object extraction depth image are rotationally transformed as shown in Equation 3 below, so that the posture of the target animal shown in FIG. b) to transform the pose.
(Formula 3)

P means the position (x, y, z) on the image before transformation, and P' is the position on the image after transformation. Information on pixel positions that are missing due to conversion is interpolated from information on surrounding pixels to generate a conversion image without missing points.

In this way, the object extraction image and the object extraction depth image whose orientation has been changed are stored in the memory 136 and used for subsequent processing. In the present embodiment, an example of attitude transformation using a rotation matrix has been described, but transformations that include translation, scaling, and the like in addition to rotation can also be used. There is also the advantage that the data to be measured and stored in advance for estimating attribute information, which will be described later, can be reduced by performing attitude transformation.

In S307, the attribute information estimation unit 135 estimates the attribute information of the target object. Attribute information represents the size, shape, volume, and mass of a target object, and at least one of these is estimated in attribute information estimation.

First, dimension estimation, which is one of the attribute information, will be described. In dimension estimation, an object extraction image whose posture has been changed is displayed on the display unit 16, and the user designates a desired measurement position on the animal in the image displayed by the input unit 15. FIG. You can specify two locations (Fig. 7(a)), specify three or more locations and connect them linearly (Fig. 7(b)), or connect them using a polynomial. and a method in which the user traces the portion to be measured (FIG. 7(c)). FIG. 7A shows a case where two points P1 and P2 are designated and the horizontal length between the two points is measured. In addition, it is desirable to be able to specify the case of measuring the length in the vertical direction or the case of measuring the Euclidean distance between two points. FIG. 7B shows an example in which four points P1 to P4 are specified and the points are connected by straight lines. Alternatively, a spline curve or the like may be used to interpolate between points and measure the length. FIG. 7(c) is an example of measuring the length of a curve traced by the user from P1 to P2 in the figure. In this way, there are various methods of specifying the measurement position and measuring the specified section. The length can also be measured, and the degree of freedom in measurement is improved. In particular, measurement using a curved line is effective for measurement when the posture cannot be converted to a desired posture. For example, as shown in FIG. 6(e), it is effective in measuring an animal that is difficult to stretch in a straight line, such as measuring the total length La of a snake.

After the measurement points are specified, the length is measured in units of pixels or sub-pixels in the image by counting the number of pixels between the specified measurement positions. The measurement in this case is the length in image space. In order to convert the measured length in pixel units into the actual length in the object space, first, the size of one pixel of the image sensor 11 is converted into the length in the international system of units in the image space. Next, the photographing parameter is used to determine the photographing magnification M, and the product of the measured length in the image space and the photographing magnification M is taken to calculate the actual length in the object space.

The photographing magnification M can be calculated by the following equation 4 using the focal length F of the imaging optical system 10 and the target object distance Z, which are parameters at the time of photographing.
(Formula 4)
M=Z/F
For the target object distance Z, the focus distance corresponding to the position of the focus lens included in the imaging optical system 10 is measured in advance, the focus lens position at the time of shooting is detected, and the corresponding focus distance is acquired as the target object distance Z. do.

Next, shape estimation, which is one piece of attribute information, will be described. Shape estimation estimates the three-dimensional shape of the target object. This is performed using information on the type of object selected in S301, and the object extraction image and the object extraction depth image generated in S306 and subjected to attitude transformation. Since the object extraction depth image after attitude conversion has a value that depends on the distance from the digital camera 100 to the target object, the depth image of the target object plane viewed from the digital camera 100 is obtained by subtracting the distance to the target object. (=shape) is calculated. In the following description, a specific surface of the measured target object is assumed to be the surface. Only one specific surface shape of the target object can be measured in one shot, and the opposite surface that cannot be seen from the shooting direction cannot be measured. In estimating the opposite surface of the target object, the 3D shapes of a plurality of objects to be measured are measured in advance, and the reference 3D shapes are stored in the storage unit 14 and/or the memory 136 . The reference three-dimensional shape to be stored may be one average three-dimensional shape for each object, but it is desirable to store a plurality of three-dimensional shapes in order to improve the accuracy of estimating the opposite surface. The reference three-dimensional shape may be data in units of voxels or data expressed in the International System of Units, but the conversion process below is changed depending on the unit of the reference three-dimensional shape data. A voxel here means a three-dimensional pixel size obtained by expanding one pixel in the xyz directions. Also, the data of the international system of units means the size of the target object on the object side measured in the international system of units. In the calculated shape information of the surface of the target object, the depth information (Z direction) is in the international system of units, but the size of the target object in the XY directions is in units of pixels. Depth information is changed to voxel units according to the data unit of the reference three-dimensional shape, or the size of the target object in the XY directions is changed to the international system of units. Conversion between the voxel unit and the international system of units is performed by obtaining the photographing magnification M using the photographing parameters as described above and converting using Equation 4.

Next, the reference three-dimensional shape is transformed so that the size of the reference three-dimensional shape and the detected target object are the same. After that, position matching processing is performed between the surface shape of the target object and the size-converted reference three-dimensional shape. It is determined which surface in the reference three-dimensional shape the surface shape measured by the matching process corresponds to. At the same time, the opposite face of the target object that has not been measured is identified in the reference 3D shape. The three-dimensional shape of the target object is estimated by synthesizing the opposite surface of the specified reference three-dimensional shape with the measured surface shape of the target object. When a plurality of different reference three-dimensional shapes are stored, the opposite surface is estimated from the reference three-dimensional shape that best matches the measured shape.

In order to further improve the accuracy of 3D shape estimation, the difference between the measured surface shape and the matching surface of the reference 3D shape is calculated, and the calculated difference is added to or subtracted from the opposite surface of the reference 3D shape. Correct the shape and assume the opposite side. Alternatively, the amount of the difference with respect to the thickness of the shape may be calculated, and the correction amount may be changed according to the thickness of the shape of the opposite surface.

If the size of one pixel in the object space is larger than the size of the target object, the estimated three-dimensional shape becomes an inaccurate shape with steps. Therefore, it is desirable that the number of pixels of the image sensor 11 is large, and it is desirable that the object occupies as much of the screen as possible when photographing. In order to reduce the pixel-by-pixel step, interpolation processing may be applied to change the shape to a smoother shape, and polygon data may be used.

Next, volume estimation, which is one of attribute information, will be described. In volume estimation, the volume is calculated using the three-dimensional shape estimated in the shape estimation. When the estimated volume is voxel unit data, the number of voxels in the estimated three-dimensional shape is counted, and the length of one side of the voxel is estimated using Equation 4 to estimate the volume in the object space. If the estimated volume is data already expressed in the International System of Units in the object space, the volume is estimated by integrating within the estimated three-dimensional shape. Note that the estimated volume may be derived based on the voxel standard, which is a unit volume element (regular grid unit) that is the base in image processing, or may be derived based on the actual size standard in the real world. may

Next, mass estimation, which is one of attribute information, will be described. In the mass estimation, the mass of the target object is estimated by multiplying the volume of the target object derived by the volume estimation and the density information of the target object stored in the storage unit 14 and/or the memory 136 . The density information may be uniform for the target object, but it is also possible to store and use different information for each part in order to estimate the mass with higher accuracy. Using skeletal analysis of the target object, the object is divided into parts such as the head, body, arms, and legs, and mass estimation is performed using different density information for each part. In this embodiment, the density information of the target object is used to calculate the mass, but the present invention is not limited to this. The weight may be estimated by multiplying the volume of the three-dimensional shape of the target object.

In S<b>308 , the control unit 12 displays the attribute information estimated in S<b>307 on the display unit 16 and stores it in the storage unit 14 . The attribute information estimated in S307 is preferably recorded in association with the depth image as metadata of the viewing image generated in S303.

As described above, according to this embodiment, it is possible to estimate the attribute information of an object in an image from the depth image generated from the captured image and the conditions under which the image was captured. Specifically, the attribute information of the target object is estimated by using the depth image, the information learned and acquired in advance for object region detection, pose estimation, and pose transformation, the shape measured in advance, the shooting parameters, and the mass ratio. can.

[Embodiment 2] Next, Embodiment 2 will be described.

In the first embodiment, the user selects the object to be measured. In contrast, the second embodiment differs from the first embodiment in that the user does not input the selection of the object to be measured. In the second embodiment, the configuration and functions of the digital camera 100 are the same as those in FIGS. 1 and 3 of the first embodiment, and the differences from the attribute information estimation processing of the first embodiment will be mainly described.

FIG. 8 shows the attribute information estimation process of the second embodiment, and the same step numbers are given to the same processes as the processes of FIG. 3 of the first embodiment.

In S801, as in S302 of FIG. 3, the control unit 12 performs processing to perform imaging with imaging settings such as the set focal position, aperture, and exposure time.

In S802, the image generator 130 generates a viewing image and a depth image, as in S303 of FIG.

In S803, the object detection unit 132 recognizes the subject. Object recognition/region detection identifies an object in an image and extracts its position/contour based on object classification/type information acquired in advance by machine learning, and generates an object extraction image. The extracted position/contour information is also applied to the depth image, and the depth generation unit 131 generates an object extraction depth image. Machine learning is not limited to a specific method, and any method may be used.

In S804, posture estimation section 133 and posture transformation section 134 perform posture estimation and posture transformation of the object identified and extracted in S803. Details of the processing in S804 performed by the orientation estimation unit 133 and the orientation conversion unit 134 of the image processing device 13 will now be described with reference to the flowchart of FIG.

In S8041, the posture estimation unit 133 acquires the shape of the surface of the target object viewed from the shooting direction by subtracting the distance from the digital camera 100 to the reference position of the target object from the object extraction depth image generated in S803. . The reference position may be set at the closest position from the digital camera 100 to the target object, or may be set at the farthest position, and is not particularly limited.

In S8042, the posture estimating unit 133 compares the surface shape of the target object obtained in S8041 with the three-dimensional shape of the target object stored in advance in the storage unit 14, and determines which one of them has the same size. Change the size of one. Considering subsequent attribute information estimation, it is desirable to match the size of the three-dimensional shape stored in the storage unit 14 and/or the memory 136 in advance with the size of the surface shape of the target object, and the transform coefficients are stored in advance. Preferably stored in unit 14 and/or memory 136 .

In S8043, the posture estimation unit 133 performs matching processing between the surface shape of the target object acquired in S8041 and the three-dimensional shape stored in advance in the storage unit 14 and/or the memory 136, and shoots the target object. identify the direction This method is effective when the posture of the photographed object is similar to the posture in which the attribute information stored in advance in the storage unit 14 and/or the memory 136 is easy to obtain, and the photographing direction is different. On the other hand, when the orientation of the target object is significantly different from the orientation stored in advance in the storage unit 14 and/or the memory 136, the evaluation value (matching score) in the matching process is lowered. Therefore, the matching score is compared with the threshold in S8044. If the matching score is higher than the threshold in S8044, the imaging plane representing the orientation of the target object is specified in S8045. If the matching score is lower than the threshold, the joint parts and skeleton of the target object are specified in S8046. This identification is also performed using information acquired by learning in advance.

In S8047, the posture transforming unit 134 calculates the difference from the skeletal position in the three-dimensional shape prepared in advance, and rotates the part in the distal direction from the joint position with the joint position as the fulcrum to obtain a posture similar to the reference posture. transform the surface shape to For example, when photographing a cow in a sitting position, the position, length, and rotation angle of the leg joints, such as the thigh, hock, and front knee, are estimated, and the joint is rotated as the fulcrum of rotation to stand up. generate an estimated image of The converted surface shape is again input to the matching processing in S8043, and the imaging plane representing the orientation of the target object is specified again.

In S8048, based on the imaging plane information specified in S8045, the posture transformation unit 134 geometrically transforms the object extraction image and the surface shape as if the target object were photographed from the side, as in S306 of FIG. Use it to change your posture. The converted object extraction image and surface shape are stored in memory 136 and used for subsequent processing.

Returning to the description of FIG. 8, in S805, the attribute information estimation unit 135 estimates the attribute information of the target object in the same manner as in S307 of FIG. The attribute information estimated in S<b>805 is displayed on the display unit 16 and stored in the storage unit 14 . The estimated attribute information is desirably stored together with the depth image as metadata of the viewing image generated in S802.

Here, the difference between attribute information estimation in S805 and S307 will be described.

Regarding dimension estimation, which is one of the attribute information, in the first embodiment, the user designates the measurement position and measures the dimension at the designated position. On the other hand, in the second embodiment, the image processing apparatus 13 identifies the type of the object based on the identification result of the object obtained in S803, and determines the size from necessary dimensional information (full length, head-body length, body height, etc.). Determine the measurement position. Then, similar to S8046, skeleton recognition is performed using the information acquired by learning in advance, and the measurement position is specified using the contour information obtained by subject recognition in S803.

For shape estimation, which is one of the attribute information, transformation for matching the size of the surface shape generated by posture estimation/transformation in S804 and the three-dimensional shape stored in advance in the storage unit 14 and/or the memory 136 is performed. Use coefficients. The conversion coefficient is used to convert the size of the three-dimensional shape stored in the storage unit 14, and the back surface portion of the object to be measured other than the surface shape is obtained from the three-dimensional shape whose size has been converted. A three-dimensional shape of the object to be measured is generated by synthesizing the surface shape of the object to be measured and the back surface shape obtained from the three-dimensional shape. In synthesizing, a smoothing process is performed so that the connection part becomes smooth.

The estimation of the volume and mass, which are one of the attribute information, is made in consideration of the body hair of the target object. In the first embodiment, body hair is not taken into account and the mass is calculated with the same density as the body hair. Therefore, the error between the actual mass and the estimated mass may increase. Also, in wool production, etc., it may be necessary to estimate only the volume of body hair.

As described above, according to the present embodiment, in addition to the processing of the first embodiment, the amount of body hair is estimated and corrected. In estimating the amount of body hair, matching processing is first performed using a joint bilateral filter, a guided filter, or the like, and an alpha matte is calculated. A region with an alpha matte of 1 or less is treated as a hair region, and its thickness is calculated, and a volume obtained by excluding the hair region from the estimated three-dimensional shape is calculated. Similarly, for mass estimation, the volume excluding the hair region is used and the density information stored for each type of object is multiplied to estimate the mass excluding the hair region. Alternatively, by calculating the volume of only the hair region from the volume including the hair and the volume excluding the hair, estimating the hair mass using the density information of the hair, and taking the sum of the mass excluding the hair, Perform mass estimation considering differences in hair density.

[Other embodiments]
In the present embodiment, the imaging element 11 has a photoelectric conversion element of the imaging surface phase difference ranging method, and is capable of acquiring an appreciation image and a depth image. is not limited to this. The depth information may be acquired by a stereo ranging method, for example, based on a plurality of captured images obtained by binocular imaging devices or a plurality of different imaging devices. Alternatively, for example, a stereo ranging method using a light irradiation unit and an imaging device, or a method using a combination of a TOF (Time of Flight) method and an imaging device may be used.

The attribute information estimation processing of Embodiments 1 and 2 is not limited to each embodiment, and can be implemented by replacing the processing using the same information.

Image processing apparatuses that can be applied as the present embodiment include digital still cameras, digital video cameras, vehicle-mounted cameras, mobile phones, smart phones, and the like.

Further, the present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device executes the program. It can also be realized by a process of reading and executing. It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

The present invention is a non-contact object attribute information estimation using an imaging device. It is useful in situations where it is difficult to measure with a mass meter.

The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the claims are appended to make public the scope of the invention.

DESCRIPTION OF SYMBOLS 100... Digital camera 12... Control part 13... Image processing apparatus 130... Image generation part 131... Depth generation part 132... Object detection part 133... Posture estimation part 134... Posture conversion part 135... Attribute information estimation part

Claims (24)

Depth generation means for generating depth information indicating the distance distribution of the subject in the depth direction from a captured image of the subject;
an object detection means for detecting an area of a specific object from the captured image;
posture estimation means for estimating the posture of the specific object;
attitude transformation means for transforming the attitude of the specific object in the captured image and the depth information into a specific attitude suitable for estimating attribute information of the specific object ;
and attribute information estimation means for estimating attribute information of the specific object from the captured image whose attitude has been changed, depth information, and image capturing conditions. 2. The object detection means detects the type and area of the specific object by extracting the area of the specific object using information about the object learned and acquired in advance. The image processing device according to . 2. The image processing apparatus according to claim 1, wherein the orientation estimation means estimates the orientation of the specific object using object information obtained by learning in advance. 4. The apparatus according to claim 3, wherein the posture estimation means estimates the posture of the specific object by using preliminarily acquired three-dimensional shape data of the object and part data of the specific object. Image processing device. The attitude transforming means uses the attitude of the specific object in the captured image and the depth information estimated by the attitude estimation means and the attitude of the specific object acquired in advance to convert the captured image and the depth information. 5. The image processing apparatus according to any one of claims 1 to 4, wherein the geometric transformation is performed on the . The pose transforming means uses the information of the part of the specific object estimated by the pose estimating means and the information of the pose and part of the object obtained in advance to convert the specified part in the captured image and the depth information. 6. The image processing apparatus according to claim 5, wherein the geometric transformation is performed for each part of the object. The posture estimating means calculates a main normal direction of the body of the specific object and obtains a plane perpendicular to the normal direction,
7. The image processing apparatus according to claim 6, wherein said attitude transformation means performs geometric transformation on said captured image and said depth information with reference to said plane. 8. The image processing apparatus according to claim 1, wherein said attribute information is at least one of size, shape, volume and mass. 3. The attribute information estimating means specifies a measurement position corresponding to the type of the specific object, and estimates the dimension at the measurement position using object information acquired by learning in advance. 9. The image processing device according to 8. The attribute information estimating means estimates the shape of the part of the object that cannot be obtained from the captured image by using the three-dimensional shape of the object obtained in advance, and synthesizes the shape with the shape of the part of the object obtained from the depth information of the object. 9. The image processing apparatus according to claim 8, wherein the three-dimensional shape of said specific object is estimated by: 9. The attribute information estimation means according to claim 8, wherein said attribute information estimation means estimates a volume from the three-dimensional shape of said specific object, and estimates a mass of said specific object using said volume and density of said object. Image processing device. 12. The image processing apparatus according to claim 11, wherein the attribute information estimating means estimates the mass of the specific object by using the parts of the object obtained from the captured image and the density of each part of the object. . When the specific object is an animal, the attribute information estimating means calculates a volume of the region of the specific object excluding the hair region, and calculates the volume of the specific object using the volume excluding the hair region and the density of the object. 12. The image processing apparatus according to claim 11, wherein the mass of is estimated. 13. The image processing apparatus according to claim 1, further comprising input means for inputting information regarding said specific object. The object detection means determines the type of the specific object by using information for identifying the type of the object acquired by learning in advance and information on the object input by the input means. 15. The image processing apparatus according to claim 14. further comprising designating means for designating the measurement position of the specific object;
10. The image processing apparatus according to claim 9, wherein said attribute information estimating means calculates dimensions of the measurement position of said specific object specified by said specifying means. 16. The designating means designates the measurement positions in accordance with an operation by a user of designating two locations, an operation of designating three or more locations, or a tracing operation with respect to the specific object. The image processing device according to . 18. The image processing apparatus according to any one of claims 1 to 17, wherein said specific object is an animal other than human. The image processing device is an imaging device having an imaging element that captures an image,
18. The image processing apparatus according to any one of claims 1 to 17, wherein said depth generating means generates depth information from images having different parallaxes captured by said imaging element. 20. The image processing apparatus according to any one of claims 1 to 19, further comprising recording means for recording the captured image, the depth information, and the attribute information in association with each other. 20. The image processing apparatus according to claim 19, wherein said image sensor has a plurality of photoelectric conversion units in one pixel. a step in which the depth generating means generates depth information indicating a distance distribution of the subject in the depth direction from the captured image of the subject;
an object detection means detecting an area of a specific object from the captured image;
a pose estimating means estimating the pose of the particular object;
a step of transforming the pose of the specific object in the captured image and the depth information into a specific pose suitable for estimating attribute information of the specific object ;
An image processing method, wherein attribute information estimating means estimates attribute information of the specific object from the captured image whose attitude has been changed, depth information, and photographing conditions of the image. A program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 21. A storage medium storing a program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 21.

Images