JP5958082B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method.

  Conventionally, there is a TV conference system that can hold a conference at a remote place. In the TV conference system, a conference participant is imaged by an imaging device and transmitted to a remote video conference terminal. There is also a video conference system that uses a wide-angle lens for photographing a large number of people. In this case, the image captured by the image capturing apparatus is often distorted, and various techniques for removing the distortion have been proposed.

  In the technique described in Patent Document 1, the size of a conference participant is increased by stretching the image in the vertical direction based on a function that connects the edges of the conference desk and two functions that represent a curve connecting the heads of the conference participants. First correction is performed to bring the thickness closer. Further, the second correction for adjusting the horizontal size is performed according to the position on the screen. In the technique described in Patent Document 1, distortion is corrected by the first correction and the second correction.

  However, the technique described in Patent Document 1 is originally considered on the premise of an image obtained by combining videos of a plurality of imaging devices with little lens distortion. Therefore, although the vertical distortion caused by the first correction is taken into account, the horizontal distortion caused by the first correction is not taken into consideration. Therefore, by performing the second correction, there is a problem that the video becomes horizontally long, the person image displayed in one image becomes small, and the image after the second correction becomes an uncomfortable image for the user. is there.

  In view of the above-described problems, the present invention provides an image processing apparatus and an image processing method for determining a correction rate for correcting an image captured by an imaging apparatus into an image that does not cause a sense of incongruity to the user. For the purpose.

  In order to achieve the above object, the determination unit that determines whether or not a person image exists in the image captured by the imaging device, and the determination unit determines that the person image exists in the image, Measuring means for measuring the size or contrast amount of a person image, and when the determining means determines that the person image is present in the image, a recognizing means for recognizing the position of the person image in the image; Position information indicating the position of the person related to the person image from the imaging device based on the position of the person image recognized by the recognition unit and the size or contrast amount of the person image measured by the measurement unit And generating means for determining a correction factor for correcting an image picked up by the image pickup device based on the position information generated by the generating means. To provide an image processing apparatus according to claim and.

  According to the image processing device and the image processing method of the present invention, it is possible to determine a correction rate for correcting an image captured by the imaging device into an image that does not cause a sense of incongruity for the user.

FIG. 3 is a diagram illustrating a functional configuration example of an image processing apparatus according to the first embodiment. FIG. 3 is a perspective view of the image pickup apparatus according to the first embodiment. 2 is a diagram illustrating a hardware configuration of an imaging apparatus and an image processing apparatus in Embodiment 1. FIG. 3 is a diagram illustrating an example of information used by each functional unit of the imaging apparatus and the image processing apparatus in Embodiment 1. FIG. FIG. 6 is a diagram for explaining correction in the first embodiment. FIG. 3 is a diagram illustrating an example of an image before correction and an image after correction in the first embodiment. FIG. 3 is a diagram illustrating a functional configuration example of a CPU according to the first embodiment. FIG. 3 is a diagram illustrating a processing flow of the image processing apparatus according to the first embodiment. FIG. 3 is a diagram illustrating an example of a frame image and the like in the first embodiment. The figure which shows an example of a magnification table. The figure which shows an example of a distance table. The figure which shows an example of an angle table. The figure which shows a predetermined area | region. The figure which shows an example of a parameter table. FIG. 4 is a diagram showing experimental results in Example 1. 2 is a diagram illustrating a hardware configuration of an imaging apparatus and an image processing apparatus in Embodiment 2. FIG. FIG. 10 is a diagram illustrating an example of information used by each functional unit of the imaging apparatus and the image processing apparatus according to the second embodiment. FIG. 10 is a diagram illustrating a functional configuration example of a CPU according to the second embodiment. The figure which shows an example of a lens position-distance table. FIG. 10 is a diagram illustrating a processing flow of the image processing apparatus according to the second embodiment. FIG. 10 is a diagram illustrating an example of a distance measurement flow in the second embodiment. The figure which shows the procedure (the 1) of the distance measurement in Example 2. FIG. The figure which shows the procedure (the 2) of the distance measurement in Example 2. FIG. The figure which shows the procedure (the 3) of the distance measurement in Example 2. FIG. The figure which shows the procedure (the 4) of the distance measurement in Example 2. FIG.

  Hereinafter, embodiments for carrying out the present invention will be described. The image processing apparatus according to the present embodiment is preferably used for a video conference, for example. In the following description, it is assumed that the image processing apparatus of this embodiment is used for a video conference.

[Example 1]
<About the video conference system>
FIG. 1 shows a schematic configuration diagram of a video conference system according to the first embodiment. This video conference system includes a plurality of video conference terminals 3000 and a network 4000 to which these video conference terminals 3000 are connected. The video conference terminal 3000 includes an imaging device 3010 and an image processing device 3020.

  A video captured by the imaging device 3010 of an arbitrary video conference terminal 3000 is corrected by the image processing device 3020, which will be described later. The corrected image is displayed on the display device 120 (see FIG. 2) in the video conference terminal 3000, and is also transmitted to other video conference terminals 3000 connected to the network 4000. 120.

  When a user presses the zoom-in / zoom-out button on any video conference terminal 3000, a digital zoom image reflecting the zoom-in / zoom-out is displayed on the display of the video conference terminal 3000. The data is transmitted to other video conference terminals 3000 connected to the network 4000 and displayed on the display device 120.

  According to this video conference system, it is possible to display a video subjected to correction and digital zoom in real time on a plurality of video conference terminals connected to the network.

<About video conference terminals>
Next, a functional configuration example of the video conference terminal will be described. FIG. 2 shows an example of a specific external view of the video conference terminal 3000. Hereinafter, the longitudinal direction of the video conference terminal 3000 is the X-axis direction, the direction orthogonal to the X-axis direction in the horizontal plane is the Y-axis direction (width direction), and the X-axis direction and the Y-axis direction are orthogonal (vertical direction, high direction). (Direction) is described as the Z-axis direction.

  The video conference terminal 3000 includes a housing 1100, an arm 1200, and a housing 1300. Among these, a sound collecting hole 1131 is provided in the right wall surface 1130 of the housing 1100. Sound from the outside that has passed through the sound collecting hole 1131 is collected by the sound input means 14 provided inside.

  Further, the upper surface means 1150 is provided with a power switch 109 and a sound output hole 1151. The user can activate the video conference terminal 3000 by turning on the power switch 109. The sound output from the sound output means 12 passes through the sound output hole 1151 and is output to the outside.

  Further, on the left wall surface 1140 side of the housing 1100, a concave means-shaped accommodation means 1160 for accommodating the arm 1200 and the camera housing 1300 is formed. Further, a connection port (not shown) is provided on the left wall surface 1140 of the housing 1100. The connection port is connected to a cable 120c for connecting the video output means 30 and the display device 120 (display).

  The arm 1200 is attached to the housing 1100 by a torque hinge 1210. The arm 1200 is configured to be vertically rotatable with respect to the housing 1100 within a tilt angle ω1 of 135 degrees. FIG. 2 shows that the tilt angle ω1 is 90 degrees. By setting the tilt angle ω1 to 0 degree, the arm 1200 and the camera housing 1300 can be accommodated in the accommodating means 1160.

  The camera housing 1300 houses the built-in photographing means 10. The photographing means 10 can photograph a person (for example, a participant in a video conference), characters or symbols written on a sheet, a room, or the like. A torque hinge 1310 is formed in the camera housing 1300. The camera housing 1300 is attached to the arm 1200 via a torque hinge 1310. With respect to the arm 1200, the camera housing 1300 is in the up / down / left / right direction within a range of a pan angle ω2 of ± 180 ° and a tilt angle ω3 of ± 45 °, assuming the state shown in FIG. It is configured to be rotatable.

  Further, the video conference terminal 3000 according to the first embodiment is not described in FIG. 2 and may have another configuration. For example, a PC (Personal Computer) with the sound output means 12 and the sound input means 14 externally connected may be used. Moreover, you may apply the video conference terminal 3000 of Example 1 to portable terminals, such as a smart phone.

<Example of Hardware Configuration of Imaging Device and Image Processing Device>
FIG. 3 illustrates a hardware configuration example of the imaging device 3010 and the image processing device 3020. The imaging device 3010 and the image processing device 3020 are connected by wire (such as USB) or wirelessly.

  First, the imaging device 3010 will be described. The sensor 12 converts the optical image formed by the lens 11 into a frame image of an electric signal. Examples of the sensor include a charge coupled device image sensor (CCD) and a complementary metal oxide semiconductor (CMOS).

  The image processing unit 13 performs predetermined image processing on the frame image. The image processing unit 13 is, for example, an ISP (Image Signal Processor). The I / F unit 14 transmits and receives a frame image, a converted frame image, other data, a control signal, and the like with the image processing device 3020.

  Next, the image processing apparatus 3020 will be described. The I / F unit 21 transmits and receives a frame image, a converted frame image, other data, a control signal, and the like with the imaging device 3010.

  The CPU 22 executes various processes. The storage unit 23 stores various software and data necessary for the processing of the CPU 22, frame images and converted frame images, various table tables and functions (calculation formulas) described later, and the like. The storage means 23 is a general term for RAM, ROM, and HDD, for example. The video output unit 24 sends a video signal to the monitor 120 (see FIG. 2).

  The communication unit 25 transmits a video signal or the like to another video conference terminal 3000 connected to the network 4000. The control unit 26 controls the entire image processing apparatus 3020. The bus 27 connects each unit in the image processing apparatus 3020.

<Information Used by Imaging Device and Image Processing Device>
Next, details of information used by the functional units of the imaging device 3010 and the image processing device 3020 will be described. FIG. 4 shows an example of information used by each device.

  First, the imaging device 3010 will be described. The image acquisition unit 121 generates a frame image. Then, the image acquisition unit 121 transmits a frame image to the first correction unit 131 and the frame image transmission unit 142.

  The first correction unit 131 performs correction on the frame image transmitted from the image acquisition unit 121 using the correction rate set by the correction rate setting unit 132, and generates a corrected frame image. The correction rate setting unit 132 sets the correction rate transmitted from the image processing apparatus 3020 in a memory in the image processing unit (ISP or the like) 13.

  The corrected frame image transmission unit 141 sends the corrected frame image to the image processing apparatus 3020. Actually, the first correction unit 131 and the corrected frame image transmission unit 141 operate in parallel. Further, the transmission of the corrected frame image is performed at a higher speed than the frame image sent for calculating the correction rate. The frame image transmission unit 142 sends the frame image to the image processing apparatus 3020.

  Then, the video output unit 24 causes the display device 120 to display the corrected frame image transmitted from the imaging device 3010. Further, the communication unit 25 transmits the corrected frame image transmitted from the imaging device 3010 to another video conference terminal 3000 via the network 4000.

  The CPU 22 calculates a correction rate from the frame image transmitted from the frame image transmission unit 142. Detailed processing of the CPU 22 will be described later. The correction rate transmission unit 211 sends the correction rate to the imaging device 3010. The correction rate setting means 132 sets the transmitted correction rate.

<Regarding correction by the first correction means>
Next, correction by the first correction unit 131 will be described. Hereinafter, the vertical direction of the image (image y-axis direction) means the depth direction of the conference when the image is shown as a plane, and the horizontal direction of the image (image x-axis direction) Is a direction orthogonal to the vertical direction.

  In general, it is not preferable for a conference participant (user) that a human face or body in a conference image looks bent due to distortion. Accordingly, in the first embodiment, the first correction unit 131 corrects the horizontal component of distortion so as to eliminate all the distortion. Further, the first correction unit 131 corrects the distortion in the vertical direction component so that a part of the distortion (predetermined amount) remains. By correcting in this way, it is possible to suppress deformation of the image due to perspective emphasis and correction in the conference image.

  FIG. 5 shows the relationship between the pixels of the image before conversion and the pixels of the image after conversion. The processing contents of the first correction unit 131 will be described with reference to FIG. First, terms will be explained. One square in FIG. 5 is defined as one pixel. The light source center C (see also FIG. 5) is the origin (coordinates are (0, 0)). The pixel of interest refers to any pixel of interest among all the pixels of the image (real image) before correction. Let the coordinate of the pixel of interest be P1 (x ′, y ′).

  Also, let P2 (x ″, y ″) be the coordinates after conversion of the pixel of interest when the image is corrected so as to leave a predetermined amount of distortion. Further, the coordinate after conversion of the pixel of interest when correction is performed so as not to leave image distortion is P0 (x, y). When the first correcting unit 131 corrects the image, P1 (x ′, y ′) is converted to P2 (x ″, y ″).

  FIG. 6 shows a pixel of an image before correction (a pixel before conversion, referred to as “pixel before conversion”), and a pixel of an image after correction (a pixel after conversion, referred to as “pixel after conversion”). Is shown schematically. The process of the 1st correction | amendment means 131 is demonstrated easily using FIG. The first correction unit 131 obtains a pre-conversion pixel corresponding to the post-conversion pixel. Also, the pixel before conversion at coordinates (a, b) is referred to as “pixel before conversion (a, b)”, and the pixel after conversion at coordinates (c, d) is referred to as “post-conversion pixel (c, d)”.

In the example of FIG. 6, the number of post-conversion pixels in the X-axis direction is N _x , and the number of pre-conversion pixels in the Y-axis direction is N _y . That is, there are N _x × N _y pixels after conversion. In the example of FIG. 6, when attention is paid to the post-conversion pixel (1, 1), the post-conversion pixel (1, 1) corresponds to the pre-conversion pixel (3, 3). A method for obtaining the pre-conversion pixel corresponding to the post-conversion pixel will be described later. Then, the converted pixels of interest are (2, 1), (3, 1)... (N _x , 1), (1, 2) (1, 3)... (1, N _y ). -Change to (N _x , N _y ) to obtain the coordinates of the pre-conversion pixels corresponding to these post-conversion pixels.

  Here, the following expression (1) is established between the pre-conversion pixel P1 (x ′, y ′) and the post-conversion pixel P2 (x ″, y ″).

Here, h is the ideal image height, that is, the distance from the optical axis center C (0, 0) to P0 (x, y), and h = (x ² + y ² ) ^1/2 . The value of h for each pixel of interest is a value measured in advance by calibration or the like. The conversion coefficient _cm is obtained in advance from (x, y) (x ′, y ′). The constant M is determined in advance from the type of camera unit of the photographing means 10 and the like.

  The correction factors α and β are values that can control the degree of reduction of distortion, and are 0 ≦ α ≦ 1 and 0 ≦ β ≦ 1, and are determined based on human arrangement data calculated from image information. is there. Further, in this case, α = 1 is set because the distortion in the horizontal direction component (X-axis direction component) is corrected so as to be completely eliminated. θ is an angle formed by the horizontal line A and the target pixel P1 (x ′, y ′), and the value of θ is measured every time the target converted pixel changes. Note that the origins (0, 0), P1 (x ′, y ′), and P0 (x, y) are positioned on a straight line (the arrow of the ideal image height h) due to distortion characteristics.

From the equation (1), the coordinate P1 (x ′, y ′) before conversion corresponding to the coordinate P2 (x ″, y ″) after conversion can be obtained. In this way, the coordinates of the pre-conversion pixels corresponding to all N _x × N _y pixels after conversion are obtained.

  Then, the first correction unit 131 obtains the luminance value for all the pre-conversion pixels P1 (x ′, y ′) calculated by Expression (1). A known technique may be used as a method for obtaining the luminance value. The first correction unit 131 sets the obtained luminance value of the pre-conversion pixel as the luminance value of the corresponding post-conversion pixel.

  For example, from the above equation (1), when the noticed converted pixel is (1, 1), the pre-conversion pixel (3, 3) corresponding to the noticed converted pixel (1, 1) is obtained. The luminance value of the pre-conversion pixel (3, 3) is obtained. The obtained luminance value of the pre-conversion pixel (3, 3) is set to the post-conversion pixel (1, 1). In this way, the first correction unit 131 can generate a corrected image.

  The first correcting unit 131 may obtain the luminance values of all the pre-conversion pixels after obtaining the pre-conversion pixels corresponding to all the post-conversion pixels. Further, after obtaining one pre-conversion pixel corresponding to the post-conversion pixel, the luminance value of the obtained pre-conversion pixel may be obtained. Further, a predetermined number of pre-conversion pixels corresponding to the post-conversion pixels are obtained, all the pre-conversion pixel luminance values obtained are obtained, and this process is repeated to obtain the pre-conversion pixel luminance values. You may do it.

<About processing of image processing apparatus>
Next, processing of the image processing apparatus according to the first embodiment will be described. FIG. 7 shows a functional configuration example of the CPU 22. FIG. 8 shows a main processing flow of the image processing apparatus.

  First, in step S <b> 2, the control unit 26 develops the image data transmitted from the frame image transmission unit 142 of the imaging device 3010 in the storage unit 23. Next, in step S4, the determination unit 102 detects a person image from the developed image data. Here, the person image is a person's face image (hereinafter simply referred to as “face image”). The detection process of the determination unit 102 may be performed by a user operation or may be performed at predetermined time intervals. FIG. 9A shows an example of an image for the developed image data. In the example of FIG. 9A, since the lens 11 is a wide-angle lens, the image is distorted.

  For example, template matching may be used as the face image detection method. Hereinafter, a case where template matching is used will be described. Details of the template matching process are described in, for example, Japanese Patent No. 4219521.

  The template matching process will be briefly described. When performing template matching processing, a facial feature data group (template) is stored in the storage means 23 in advance. The feature data group is data in which feature quantities such as face contours and face components (for example, chin, mouth, eye, nose, contour, etc.) are determined for each type of face. The types of faces are types of faces such as adults, children, babies, elderly people, women, men, yellow races, white races, and black races. In the following description, it is assumed that the person participating in the conference is a yellow race, that is, the face color is skin color.

  Then, the determination unit 102 divides the image into a plurality of areas. Here, the shape of the region may be any shape such as a rectangular shape, a circular shape, or an elliptical shape. Here, the shape of the region is assumed to be rectangular. The determination unit 102 calculates the similarity between the feature amount of the image in the rectangular area and the feature data group. The rectangular area for which the similarity is calculated is called an inspection area. The determination unit 102 determines whether the calculated similarity is equal to or greater than a predetermined threshold. When the determination unit 102 determines that the threshold value is equal to or greater than the threshold value, the determination unit 102 determines that the inspection area is a face image area (hereinafter referred to as “face image area”), that is, detects a face image. to decide.

  Further, the determination unit 102 changes the size of the face image area so as to make the area other than the skin color area in the face image area as small as possible. By making the change, the determination unit 102 can generate a face image area corresponding to the size of the face image.

  In step S6, the determination unit 102 determines whether the inspection area is a face image area. If the determination unit 102 determines that the inspection area is a face image area (Yes in step S6), the processes of steps S7, S8, S9, and S10 (described later) are performed. If the determination unit 102 determines that the inspection area is not a face image area (No in step S6), the process proceeds to step S12.

  In step S12, the determination unit 102 determines whether or not the entire area of the image has been inspected. When the determination unit 102 determines that the entire area of the image has been inspected (Yes in step S12), the process proceeds to step S14. If the determination unit 102 determines that the entire area of the image has not yet been inspected, the process returns to step S4.

  In this way, the determination unit 102 determines whether or not a person image (here, a face image) exists in the image of the developed image data (see FIG. 9A). When the determination unit 102 uses the template matching process, the person image is a face image. When performing human image detection by a process other than the template matching process, or when the image processing apparatus is used for other purposes other than the video conference system, the human image may be an image other than the human face. .

  FIG. 9A illustrates an example of an image captured by the imaging device 3010. In the example of FIG. 9A, since the lens 11 is a wide-angle lens, the image is distorted. Next, in step S4, the determination unit 102 performs face image detection using the facial feature data group.

  In step S7, the measuring unit 106 measures the size of the person image. Here, the size of the person image is the length of the face image area in the y-axis direction. That is, the measuring unit 106 measures the length of the face image area in the y-axis direction. The length of the face image area in the y-axis direction is, for example, the number of pixels in the y-axis direction of the face image area. The size of the person image may be, for example, the area of the face image, that is, the area of the skin color portion in the outline of the face.

In FIG. 9A, two face images A and B exist. Further, the areas for the face images A and B are set as face image areas A and B, respectively. Further, the y-axis direction length of the face image region R _A and h _A, and y-axis direction length h _B of the face image area R _B. The measuring means 106 measures the length h _A in the y-axis direction of the face image region _RA . The measurement means 106 measures the y-axis direction length _{h B} of the face image area _{R B.} Hereinafter, the size of the person image is referred to as “the length of the face image”.

In step S8, the recognition unit 108 recognizes the position of the face image area with respect to the entire area of the image. Here, the recognition of the position of the face image area by the recognition means 108 is, for example, “from the center C (see FIG. 9A) of the image captured by the imaging device 3010 to the center (center of gravity) of the face image area. It means calculating the “distance L”. In the example of FIG. 9A, the distance from the center C to the center of the face image area R _A is L _A, and the distance from the center C to the center of the face image area R _B is L _B.

  Either of the processes of steps S7 and S8 may be performed first or in parallel.

  By the way, ideally, the measuring unit 106 preferably measures the height of the face image area for the person so as to correspond to the distance from the imaging device 3010 to the person. For example, for a plurality of face image areas that are equidistant from the lens 11 of the imaging device 3010, the heights of the face image areas are measured assuming that all the face image areas have the same height. Is preferred.

In other words, as shown in FIG. 9A, since the image captured by the imaging device 3010 is distorted, the heights h _A and h _{B of} the face image measured by the measuring unit 106 and the actual person It does not correspond to the height of the face. Therefore, in step S9, the second correction unit 104 preferably corrects the height of the face image.

  In step S9, correction based on the position of the person image recognized by the recognition unit 108 is performed on the size of the person image (height (number of pixels) of the face image) measured by the measurement unit 106. Here, a correction method of the second correction unit 104 will be described. For example, “a distance L from the center C of the image captured by the imaging device 3010 (see FIG. 9A) to the face image (face image region)” and “a magnification g by which the size h of the person image is multiplied”. Are used in correspondence with each other. The magnification table is obtained experimentally and is stored in the storage means 23 in advance.

FIG. 10 shows an example of the magnification table. In the example of FIG. 10, the distance L and the magnification g are associated with each other. In the example of FIG. 10, for example, the corresponding ratio to the distance _{L 1} becomes _{g 1.} Second correction means 104 refers to the ratio table, obtains the ratio _{g A} corresponding to the distance _{L A.} Further, the second correction unit 104 refers to the magnification table to obtain the magnification g _B corresponding to the distance L _B.

Then, the second correcting unit 104 multiplies the height h of the face image area by the obtained magnification g. In the example of FIG. 9A, the height h of the face image region is corrected by calculating h _A ′ = h _A · g _A and h _B ′ = h _B · g _B. Here, h _A ′ and h _B ′ are the heights of the face image areas after correction. FIG. 9B shows h _A ′ and h _B ′.

Further, without using a magnification table, a function F ₁ (•) that satisfies g = F ₁ (L) is obtained in advance, and a magnification g _A corresponding to the distance L _A is obtained using the function. May be.

In step S <b> 10, the generation unit 110 generates position information based on the position of the person image recognized by the recognition unit 108 and the size of the person image measured (after correction) by the measurement unit 106. Here, the position information is information indicating the position of the person related to the person image from the imaging device 3010. FIGS. 9C and 9D show H _A and H _B as persons for the person images A and B, respectively.

Here, as illustrated in FIG. 9D, it is assumed that the person position information includes a distance P from the imaging device 3010 to the person and an angle θ of the person in the horizontal direction from the imaging device 3010. The horizontal direction angle θ from the imaging device 3010 is a straight line connecting the center of the lens 11 of the imaging device 3010 and a person (for example, H _A , H _B ) and the optical axis of the lens 11 when viewed from directly above. Is the angle formed by the y-axis.

  First, the generation means 110 and the distance P calculation method will be described. A distance table indicating the correspondence between the height h ′ of the face image area corrected by the second correction unit 104 and the distance P is obtained in advance and stored in the storage unit 23. FIG. 11 shows an example of the distance table. In the example of FIG. 11, when the height h ′ of the face image area is 50 pixels, the distance P from the imaging device 3010 to the person of the face image is 50 cm.

The generation unit 110 refers to the distance table to determine the distance P corresponding to the height h ′ of the face image area determined by the second correction unit 104. Further, without using the distance table, a function F ₂ (·) that satisfies P = F ₂ (h ′) may be calculated in advance, and the distance P may be obtained using the function.

Next, how to obtain the horizontal angle θ from the imaging device 3010 will be described. An angle table indicating the correspondence between the distance L and the horizontal angle θ is obtained in advance and stored in the storage unit 23. FIG. 12 shows an example of the angle table. In FIG. 12, for example, the horizontal angle corresponding to the distance L ₁ is θ ₁ .

The second correcting unit 104 refers to the angle table to obtain the horizontal direction angle θ corresponding to the distance L. Further, the function F ₃ (·) that satisfies θ = F ₃ (L) may be calculated in advance without using the distance table, and the horizontal direction angle θ may be obtained using the function.

  In general, the characteristics of an image picked up by the image pickup device 3010 differ depending on the design of the lens 11. Also, except for directional lenses, ordinary lenses have the same characteristics on concentric circles that are far from the center of the lens. From this, if the characteristics of the lens design are known, the horizontal direction angle θ from the imaging device 3010 is determined by the position of the person image in the image.

FIG. 9D shows position information about the persons H _A and H _B. For the person _{H A,} the distance from the imaging device 3010 becomes _{L A,} the horizontal angle becomes theta _A. Also, the person _{H B,} the distance from the imaging device 3010 becomes _{L B,} the horizontal angle becomes theta _B.

In this way, the generation unit 110 generates position information including the distance L from the imaging device 3010 and the horizontal direction angle θ. As another example of the position information, the coordinates (x _A , y _A ) of each person H _A , H _B when the imaging device 3010 is viewed from above and the imaging device 3010 is the origin (0, 0). ), (X _A , y _B ).

  In step S11, it is preferable that the detection unit 112 detects whether or not the horizontal angle θ is within a predetermined range. In other words, the detecting means 112 detects whether or not there is a person within a predetermined area. The predetermined area refers to an area within a predetermined range of the horizontal angle. FIG. 13 shows the predetermined region M (hatched portion). An area other than the predetermined area is referred to as a predetermined outside area N. In the example of FIG. 13, the predetermined area M is an area having a horizontal angle (view angle) of 100 degrees, that is, −50 degrees to +50 degrees.

  If YES is determined in step S12, in step S14, the determination unit 114 determines the correction factor β (see the above formula (1)) based on the position information generated by the generation unit 110 and the detection result of the detection unit 112. ). A determination method of the determination unit 114 will be described. The determination unit 114 determines the correction rate β (correction data) using the correction rate table.

  First, a case where there are two or more person images will be described. In this case, the determination unit 114 obtains the distance difference D in the depth direction between the person farthest from the imaging device 3010 and the person closest to the imaging device 3010. To determine the farthest person and the closest person, the determining unit 114 may compare the distances P. The depth direction refers to the y-axis direction that is the center line of the angle of view of the imaging device 3010.

In the example of FIG. 9 (D), the farthest person from the image pickup device 3010 is a person _{H A,} closest to the person from the image pickup device 3010 is a person _{H B.} And the determination means 114 can obtain | require the distance difference D by the following formula | equation (2).
D = P _A cos θ _A −P _B cos θ _B (2)
Furthermore, the determination unit 114 analyzes the detection result of the detection unit 112 to determine whether or not the horizontal angle of one or more persons is within a predetermined angle (see FIG. 13). In other words, it is determined whether or not a person exists in the non-predetermined area N.

Then, the determination unit 114 obtains a correction rate β corresponding to the difference D in the depth direction and the detection result. For example, the person _H A, and the depth direction of the distance difference D = 60cm for _{H B,} the person _H A, is _{H B} together, does not exist in the predetermined area outside N (person _H A, is _{H B} exists in a predetermined region M ), The determination unit 114 determines the correction rate β to be 90% (= 0.9). Also, a person _H A, the depth direction of the distance difference D = 60cm for _{H B,} when the person _H A, one of _{H B} is present in a predetermined area outside N is determining unit 114, the correction factor β is determined to be 80%. Further, when the distance difference D in the depth direction for the persons H _A and H _B is 100 cm, regardless of whether or not the persons H _A and H _B exist in the predetermined region M, the determining unit 114 The correction rate β is determined to be 80%.

  When the number of person images is three or more, the distance difference D in the depth direction between the person farthest from the imaging device 3010 and the person closest to the imaging device 3010 among the three or more people is determined. Ask. The determination unit 114 may compare the distance P of each of the three or more person images with respect to how to determine the person farthest from the imaging device 3010 and the person closest to the imaging device 3010.

  When the person image is 1, the determination unit 114 determines the correction rate β assuming that the distance difference D = 0 in the depth direction.

  In this way, the determination unit 102 measures the number of person images, and the correction rate β differs between the case where the person image is “1” and the case where the person image is “2 or more”.

Further, the function F ₄ (·) that satisfies β = F ₄ (D) may be determined and the function may be used without storing the correction rate table in the storage unit 23.

  The correction rate transmission unit 211 (see FIG. 4) transmits the correction rates α and β determined by the determination unit 114 to the correction rate setting unit 132. Then, the first correction unit 131 corrects the frame image using the correction rates α and β set by the correction rate setting unit 132.

  Further, in the description of the processing of steps S7 to S11 in FIG. 8, the face images A and B are described simultaneously. However, in FIG. 8, every time it is determined in step S6 that there is one face image, steps S7 to S11 are performed. Processing is performed.

<Experimental result>
Next, experimental results will be described. FIG. 15A illustrates an original image captured by the imaging device 3010. FIG. 15B shows an image when the original image is corrected with the correction factors α and β being both “100%”. FIG. 15C shows an image when the original image is corrected with the correction factor β determined by the processing flow shown in the example of FIG. 8 (α = 100%).

  In FIG. 15A, distortion is caused by the lens 11, the posture of the person at both ends is bent, and the fluorescent lamp is distorted vertically, resulting in an unnatural image. In FIG. 15B, when the correction factors α and β are both set to “100%”, the image is stretched as the distance from the center increases. Therefore, although the vertical and horizontal portions that are originally straight lines are straight lines, they are particularly elongated due to the influence of the stretching of the persons at both ends, resulting in an unnatural image. Perspective is also very emphasized.

  On the other hand, in FIG. 15C, since the vertical correction factor is adjusted in accordance with the arrangement of the person, a slight distortion remains in the horizontal direction, but the straight line in the vertical direction is maintained, and the person also has a natural image. It has become. As is clear from this experiment, the image can be corrected to an image that does not feel uncomfortable for the user by performing correction using the correction factor determined by the image processing apparatus according to the first embodiment.

  In the correction rate table illustrated in FIG. 14, when the distance D in the depth direction is long in two or more persons, the correction rate β is set smaller than in the case where the distance D in the depth direction is short. This can reduce the emphasis on perspective in the depth direction.

  Even when the distance D in the depth direction is small, the correction rate β when the person is present in the region N outside the predetermined range is set lower than the correction rate β when the person is not present in the region N outside the predetermined range. doing. Thereby, as shown in FIG. 15C, which will be described later, it is possible to mitigate the phenomenon that the persons at both ends of the image are stretched. In other words, the determination unit 114 can further correct the image without a sense of incongruity by using the detection result of the detection unit 112.

[Example 2]
Next, the video conference system in Example 2 will be described. In the second embodiment, the contrast amount of the person image is used to obtain the distance from the imaging device to the person. Thereby, the precision of distance measurement can be improved.

  The focusing distance of the camera varies depending on the lens position. By knowing the position of the lens when the subject is in focus using this variation, the positions of the subject and the camera can be obtained. In Example 2, the distance is measured using this method.

  Further, in order to determine when the image is in focus, the image contrast amount is used in the second embodiment. In general, the amount of contrast is maximized when the subject is in focus.

  The schematic configuration of the video conference system in the second embodiment is the same as the configuration shown in FIG. 1, and the functional configuration of the video conference terminal in the second embodiment is the same as the configuration shown in FIG. In the second embodiment, an imaging apparatus and an image processing apparatus will be described using reference numerals 5010 and 5020, respectively.

<Example of Hardware Configuration of Imaging Device and Image Processing Device>
FIG. 16 illustrates a hardware configuration example of the imaging device 5010 and the image processing device 5020 according to the second embodiment. Regarding the hardware of the imaging device 5010 and the image processing device 5020 in the second embodiment, the same components as those in FIG. Hereinafter, hardware that performs processing different from that of the first embodiment will be mainly described.

  The imaging device 5010 according to the second embodiment includes a focus control unit 15. The focus control unit 15 adjusts the position of the lens 11.

  The CPU 28 of the image processing apparatus 5020 according to the second embodiment executes various processes. Further, unlike the first embodiment, the CPU 28 measures the distance from the imaging measure 5010 to the person based on the contrast amount of the person image.

<Information Used by Imaging Device and Image Processing Device>
Next, details of information used by the functional units of the imaging device 5010 and the image processing device 5020 will be described. FIG. 17 shows an example of information used by each device. The functional units that perform the same processing as in the first embodiment are denoted by the same reference numerals as the functions illustrated in FIG. In FIG. 17, functions and information different from those in the first embodiment will be mainly described.

  The focus control unit 15 changes the position of the lens 11 by a signal from the CPU 28 of the image processing device 5020. The image acquisition unit 121 acquires frame images taken at each position of the lens 11. The subsequent processing is performed on each frame image. The correction by the first correction unit 131 is the same as that in the first embodiment.

  The CPU 28 calculates a contrast amount from each frame image transmitted from the frame image transmission unit 142. The CPU 28 calculates a distance based on the contrast amount and calculates a correction rate. Detailed processing of the CPU 28 will be described later.

<About processing of image processing apparatus>
Next, processing of the image processing apparatus 5020 in the second embodiment will be described. FIG. 18 is a diagram illustrating an example of a functional configuration of the CPU 28 according to the second embodiment.

  In the functions shown in FIG. 18, the same functions as those shown in FIG. The CPU 28 in the second embodiment includes a contrast measurement unit 202 and a generation unit 204.

  The contrast measuring unit 202 calculates the contrast amount of the person image detected by the determining unit 102 for each frame image. Hereinafter, for example, an image of a human face area (face image) is used as the person image, but the present invention is not limited to this. The contrast measuring unit 202 may use the difference between the maximum luminance and the minimum luminance in the face image as the contrast amount of the face image, or may use the variance of the luminance in the face image.

  Unlike the first embodiment, the generation unit 204 calculates the distance from the position of the lens 11 where the contrast amount of the face image is maximized to a person (for example, a face). The generation unit 204 creates in advance a lens position-distance table in which the position of the lens 11 is associated with the actual distance. This lens position-distance table is stored in, for example, the storage unit 23 and is appropriately read out by the generation unit 204.

FIG. 19 shows an example of a lens position-distance table. In the example shown in FIG. 19, the lens position S and the distance P are associated with each other. The distance P is a distance from the imaging device 5010 to a person, for example. In the example shown in FIG. 19, for example, the distance to the lens position _{S 11 is P 11.} The generation unit 204 refers to a lens position-distance table as shown in FIG. 19 to obtain a distance P corresponding to the lens position.

In addition, the generation unit 204 calculates a function F ₃ (·) that satisfies P = F ₃ (S) in advance without using the lens position-distance table, and obtains the distance P using the function. Also good.

  FIG. 20 shows a main processing flow of the image processing apparatus 5020. In step S101 shown in FIG. 20, the determination unit 102 performs face recognition processing. The processing in step S101 is the same as the processing in steps S2 to S4 shown in FIG.

  In step S102, the contrast measurement unit 202, the generation unit 204, and the like calculate the distance P from the recognized face image. Detailed processing in step S102 will be described later with reference to FIG.

  In step S103, the determination unit 114 determines the correction rate β using the difference D in the depth direction. The process of step S103 is the same as the process of step S14 shown in FIG.

  Next, the process of step S102 will be described. FIG. 21 shows an example of a distance measurement flow in the second embodiment. In step S201 shown in FIG. 21, the contrast measuring unit 202 determines whether or not the number of face images recognized by the determining unit 102 in the target frame image is 1 or more. If the number of face images is 1 or more (step S201—Yes), the process proceeds to step S202. If the number of face images is not 1 or more (step S201—No), the distance measurement process is terminated.

  In step S202, the contrast measuring unit 202 acquires a frame image including a face image.

  In step S203, the contrast measuring unit 202 calculates a contrast amount for each face image in the acquired frame image. As the contrast amount, the difference between the maximum luminance and the minimum luminance in the face image may be used as the contrast amount of the face image, or the luminance dispersion in the face image may be used.

  In step S204, the contrast measurement unit 202 stores the current lens position, face image ID (for example, face images A and B), and the contrast amount in the memory. The memory is, for example, the storage means 23.

  In step S205, the contrast measuring unit 202 determines whether or not processing has been performed at all lens positions. In order to determine whether or not the lens positions are all, the contrast measuring unit 202 only needs to know the movable range of the lens 11. If all the lens positions are finished (step S205—Yes), the process proceeds to step S207. If all the lens positions are not finished (step S205—No), the process proceeds to step S206.

  In step S <b> 206, the CPU 28 transmits a signal for shifting the position of the lens 11 to the focus control unit 15. The CPU 28 controls the focus control unit 15 of the imaging device 5010 to adjust the position of the lens 11. After step S206, the process returns to step S202, and the processes of steps S202 to S204 are performed at another lens position.

  In step S207, the generation unit 204 obtains the position of the lens 11 at which the contrast amount is maximized for each face image. The generation unit 204 can obtain the lens position of the maximum contrast amount by comparing the stored contrast amount for each face image.

  In step S208, the generation unit 204 obtains the distance P by referring to the lens position-distance table from the maximum lens position for each face image.

  After obtaining the distance P, the generation unit 204 generates position information including the position of the face image recognized by the recognition unit 108 and the distance P.

  When the position information is obtained by the generation unit 204, the same processing as in the first embodiment is performed by the detection unit 112 and the determination unit 114. Therefore, the correction rate β is determined by the determination unit 114.

<Distance Measurement Procedure in Example 2>
Next, a distance measurement procedure according to the second embodiment will be described with reference to FIGS. FIG. 22 shows a distance measurement procedure (part 1) in the second embodiment. As illustrated in FIG. 22, the determination unit 102 performs (1) face recognition, for example, on the frame image from the image acquisition unit 121. The determination unit 102 stores the recognition result information in the memory. The memory may be the storage unit 23 or another storage unit. For example, in the case illustrated in FIG. 22, the determination unit 102 stores the coordinates and size of the rectangles of the faces A and B in the memory.

  FIG. 23 shows a distance measurement procedure (part 2) in the second embodiment. As illustrated in FIG. 23, the imaging apparatus 5010 (2) continuously captures images while moving the position of the lens 11. In this case, the CPU 28 controls the focus control unit 15 to move the lens position. Here, after the photographing at all lens positions is completed, the following process (3) may be performed, or the process (3) may be performed every time one image is photographed.

  In this example, for the sake of simplicity, a single lens will be described, but a plurality of lenses may be used. When the imaging device 5010 includes a plurality of lenses, the contrast amount may be measured for each of the plurality of lenses.

  The contrast measuring means 202 (3) measures the amount of contrast for each face image of each image. The contrast measuring unit 202 stores the measured contrast amount, the lens position, and the face image in a memory in association with each other. The memory is, for example, the storage unit 23 and may be another storage unit.

  FIG. 24 shows a distance measurement procedure (part 3) in the second embodiment. As shown in FIG. 24, the generation unit 204 (4) obtains the lens position where the contrast amount is maximum for each face image. For example, the generation unit 204 obtains the lens position lensA (mm) from the image with the maximum contrast amount of the face image A, and obtains the lens position lensB (mm) from the image with the maximum contrast amount of the face image B.

FIG. 25 shows a distance measurement procedure (part 4) in the second embodiment. As shown in FIG. 25, the generation unit 204 obtains the actual distance from the relation F ₃ which has been determined in advance and (5) the lens position S. Relation of the actual distance between the lens position F ₃ is previously obtained by such pre-experiment. In order to improve the actual accuracy of the distance, for each angle from the front of the camera relational expression F ₃ may be prepared. Also, the relational expression F ₃ is not necessarily become linear as shown in FIG. 25 may be represented by such a quadratic function.

  As described above, according to the second embodiment, the distance can be obtained by using the focus adjustment, and the correction can be performed with the correction factor determined by using the distance, so that the image can be corrected without any sense of incongruity for the user.

  Further, according to the image processing apparatus of the present embodiment, the correction factor of the vertical component is determined according to the position information indicating the position of the person from the imaging apparatus. In other words, the correction factor for the vertical component is determined according to the arrangement of the person, in other words, the correction factor for performing the correction that partially retains the distortion. Therefore, it is possible to determine a correction rate that can be corrected to an image that does not feel uncomfortable for the user.

  4 and 18, the imaging devices 3010 and 5010 have the first correction unit 131. However, the image processing devices 3020 and 5020 have the first correction unit 131 so that images can be displayed. The processing devices 3020 and 5020 may perform the correction of the above formula (1).

  In addition, the above-described embodiments may be combined so that any processing is possible, and the user may be able to switch between performing the processing in the first embodiment and performing the processing in the second embodiment.

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. Various inventions can be formed by appropriately combining a plurality of disclosed constituent elements. For example, some components may be deleted from all the components shown in the embodiment.

DESCRIPTION OF SYMBOLS 11 ... Lens 12 ... Sensor 13 ... Image processing unit 14 ... I / F means 21 ... I / F means 22, 28 CPU
23 ... Storage means 24 ... Video output unit 25 ... Communication unit 26 ... Control unit 27 ... Bus 102 ... Determination means 104 ... Second correction means 106 ... Measuring means 108 ... Recognizing means 110 ... Generating means 112 ... Detecting means 114 ... Determining means 131 ... First correcting means 132 ... Correction rate setting means 141 ... Conversion Frame image transmission means 142... Frame image transmission means 202... Contrast generation means 3010... Imaging device 3020.

Japanese Patent No. 4279613

Claims (8)

Determination means for determining whether or not a human image exists in an image captured by the imaging device;
When the determination unit determines that the person image is present in the image, a measurement unit that measures the size or contrast amount of the person image;
When the determination unit determines that the person image is present in the image, a recognition unit that recognizes a position of the person image in the image;
A position indicating the position of the person related to the person image from the imaging device based on the position of the person image recognized by the recognition unit and the size or contrast amount of the person image measured by the measurement unit. Generating means for generating information;
Determining means for determining a correction rate for correcting the image based on the position information generated by the generating means;
Correction means for performing correction based on the position of the person image recognized by the recognition means on the size of the person image measured by the measurement means;
The image processing apparatus generates the position information based on the position of the person image recognized by the recognition means and the size of the person image corrected by the correction means . The determination means counts the number of person images present in the image,
When the number of person images counted by the determination means is 2 or more,
The determining means includes
The image processing according to claim 1, wherein a correction factor is determined based on a distance difference in a depth direction between a person farthest from the imaging device and a person closest to the imaging device in a person related to each of the two or more human images. apparatus. When the number of person images counted by the determination means is 1,
The image processing apparatus according to claim 2 , wherein the determination unit determines a correction rate on the assumption that the distance difference in the depth direction is zero. A straight line connecting the center of the lens of the imaging device and the person related to the person image;
An optical axis of the imaging device;
Having an detecting means for detecting whether or not an angle formed by is within a predetermined range;
It said determining means, on the basis of the detection result of the position information and said detection means, the image processing apparatus according to claim 1 to 3 any one of claims to determine the correction factor. The size of the person image is an image processing apparatus according to claim 1-4 any one wherein the length of the y-axis direction of the person of the face image according to the person image. The determining means includes
The image processing apparatus according to claim 1 to 5 any one of claims to determine the correction factor for correcting so as to leave the distortion in the y-axis direction component of the image. The measuring means includes
Measure the contrast amount of the person image in a plurality of images taken by changing the position of the lens,
The generating means includes
The image processing apparatus according to claim 1, wherein a distance from the imaging device to a person is obtained based on a lens position of an image with the maximum contrast amount. A determination step of determining whether a human image is present in an image captured by the imaging device;
When the determination step determines that the person image is present in the image, a measurement step of measuring the size or contrast amount of the person image;
When the determination step determines that the person image is present in the image, a recognition step of recognizing the position of the person image in the image;
A correction step for performing correction based on the position of the person image recognized by the recognition step on the size of the person image measured by the measurement step;
Based on the position of the person image recognized in the recognition step and the size or contrast amount of the person image measured in the measurement step, the position of the person related to the person image from the imaging device is determined. A generating step for generating position information to be shown;
Determining a correction rate for correcting the image based on the position information generated by the generating step, and
An image processing method in which, in the generation step, the position information is generated based on the position of the person image recognized in the recognition step and the size of the person image corrected in the correction step .