JP2012226645A - Image processing apparatus, image processing method, recording medium, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable three-dimensional coordinates to be detected correctly.SOLUTION: An acquisition unit acquires a fisheye image that is an image captured through a fisheye lens, a conversion unit converts the fisheye image and generates a converted image, and a recognition unit recognizes an image to be recognized from the converted image. The present invention is applicable to an image processing apparatus.

Description

  The present technology relates to an image processing apparatus and method, a recording medium, and a program, and more particularly, to an image processing apparatus and method, a recording medium, and a program that can accurately detect three-dimensional coordinates.

  Fisheye lenses may be used in cameras. Since the fisheye lens can capture an image in a range of 360 degrees, it is suitable for monitoring the entire surroundings, for example.

  It has been proposed to arrange a camera having such a fisheye lens on the robot's head and observe the surroundings (for example, Patent Document 1).

JP 2008-210403 A

  However, since an image captured by a camera using a fisheye lens has a large distortion, it is difficult to accurately estimate the three-dimensional position of the subject imaged from the image.

  The present technology has been made in view of such a situation, and enables accurate detection of three-dimensional coordinates.

  One aspect of the present technology is an acquisition unit that acquires a fisheye image that is an image captured through a fisheye lens, a conversion unit that converts the fisheye image to generate a converted image, and a recognition target from the converted image It is an image processing apparatus provided with the recognition part which recognizes this image.

  The converted image may be a developed image obtained by developing the fisheye image on a cylindrical surface.

  The image processing apparatus may further include a detection unit that detects a size of the recognition target image on the developed image.

  The image processing apparatus may further include a calculation unit that calculates a distance to the recognition target from a size of the recognition target image.

  The calculation unit can calculate the distance to the recognition target corresponding to the size of the recognition target image from a relational expression that defines the relationship between the distance and the size.

  The information processing apparatus may further include a selection unit that selects one from the plurality of relational expressions prepared for the plurality of recognition targets having different sizes.

  In order to select one of the relational expressions, a measuring unit that measures a distance may be further provided.

  In order to select one said relational expression, the specific part which specifies the specific said recognition object can further be provided.

  The calculation unit can further calculate and add a new relational expression.

  An updating unit for updating the relational expression can be further provided.

  The image processing apparatus may further include a coordinate conversion unit that converts a coordinate system of the fisheye image into a world coordinate system using a distance to the recognition target detected from the relational expression as a parameter.

  An image processing method, a recording medium, and a program according to one aspect of the present technology are an image processing method, a recording medium, and a program corresponding to the image processing device according to one aspect of the present technology.

  In one aspect of the present technology, a fish-eye image that is an image captured through a fish-eye lens is acquired, the fish-eye image is converted, a converted image is generated, and a recognition target image is detected from the converted image.

  As described above, according to one aspect of the present technology, three-dimensional coordinates can be accurately detected.

It is a block diagram which shows the structure of the image processing apparatus of this technique. It is a flowchart explaining the three-dimensional position estimation process of the image processing apparatus of this technique. It is a figure explaining the three-dimensional position estimation process of the image processing apparatus of this technique. It is a figure explaining the relationship between the magnitude | size of a to-be-photographed object's image, and distance. It is a graph which shows the relationship between the magnitude | size and distance of a to-be-photographed object. It is a figure explaining the coordinate system of a fisheye image. It is a figure explaining a world coordinate system. It is a flowchart explaining a learning process. It is a figure explaining the formula showing the relationship between the magnitude | size of a subject image, and distance. It is a flowchart explaining the three-dimensional position estimation process of the image processing apparatus of this technique. It is a flowchart explaining the three-dimensional position estimation process of the image processing apparatus of this technique. It is a flowchart explaining a relational expression selection process. It is a figure explaining the principle which calculates | requires distance from the magnitude | size of the image of a to-be-photographed object. It is a flowchart explaining a relational expression selection process. It is a flowchart explaining the three-dimensional position estimation process of the image processing apparatus of this technique. It is a flowchart explaining the three-dimensional position estimation process of the image processing apparatus of this technique. It is a flowchart explaining the three-dimensional position estimation process of the image processing apparatus of this technique. It is a flowchart explaining the three-dimensional position estimation process of the image processing apparatus of this technique. It is a figure explaining expansion | deployment of an image. It is a block diagram which shows the structure of a personal computer.

Hereinafter, modes for carrying out the technology (hereinafter referred to as embodiments) will be described. The description will be given in the following order.
1. 1. First embodiment 2. Second embodiment 3. Third embodiment 4. Fourth embodiment Modification 6 6. Application of this technology to programs Other

<1. First Embodiment>
[Configuration of image processing apparatus]

  FIG. 1 is a block diagram illustrating a configuration of an image processing apparatus 1 according to the present technology. The image processing apparatus 1 includes a camera 11, a motor 12, a microprocessor 13, an input unit 14, an output unit 15, a communication unit 16, a memory 17, and a stereo camera 18.

  The camera 11 has a fisheye lens 11A and images a subject in a monitoring area within a predetermined range. The motor 12 drives the camera 11 and directs its direction in a predetermined direction. The microprocessor 13 controls each part. The input unit 14 includes a keyboard, a mouse, a switch, and the like. The input unit 14 is operated by a user and outputs a signal corresponding to the operation to the microprocessor 13. The output unit 15 includes an LCD (Liquid Crystal Display), a speaker, and the like, and displays a predetermined image or message or outputs a sound.

  The communication unit 16 communicates with other devices via the Internet or other networks. The memory 17 stores various information and programs. The stereo camera 18 is added as necessary to capture an image for distance measurement.

  The microprocessor 13 includes an imaging unit 31, an image conversion unit 32, a face recognition unit 33, a determination unit 34, a detection unit 35, a calculation unit 36, a coordinate conversion unit 37, an instruction unit 38, a selection unit 39, a measurement unit 40, and a storage unit. 41, an update unit 42, and a control unit 43.

  The imaging unit 31 images the subject with the camera 11 and the stereo camera 18. The image conversion unit 32 converts the image by the fisheye lens 11A into a converted image. The face recognition unit 33 recognizes a human face as a recognition target from the converted image. The determination unit 34 executes various determination processes. The detection unit 35 performs detection processing such as a face and a face size. The calculation unit 36 performs calculations for specifying distances, relational expressions, and faces.

  The coordinate conversion unit 37 performs a coordinate conversion calculation. The instruction unit 38 gives instructions such as changing the distance and changing the size of the face. The selection unit 39 selects a relational expression. The measurement unit 40 measures the distance. The storage unit 41 stores the distance and the face size. The update unit 42 updates the relational expression between the distance and the face size. The control unit 43 drives the motor 12 and controls the orientation of the camera 11 and the stereo camera 18.

<1. First Embodiment>
[Three-dimensional position estimation process 1]

  Next, the three-dimensional position estimation process of the image processing apparatus 1 will be described. FIG. 2 is a flowchart illustrating the three-dimensional position estimation process of the image processing apparatus 1 according to the present technology. This process is started when the image processing apparatus 1 is turned on.

  In step S <b> 1, the imaging unit 31 captures an image with the camera 11. A surrounding object is photographed by the camera 11 via the fisheye lens 11A, and the image is acquired as a fisheye image.

  FIG. 3 is a diagram illustrating a three-dimensional position estimation process of the image processing apparatus 1 according to the present technology. The fisheye image P1 shown in the upper left of FIG.

  In step S2, the image conversion unit 32 converts the image. Specifically, the fisheye image P1 acquired in step S1 is converted into a converted image P2. The converted image P2 in this embodiment is a developed image in which the fisheye image P1 is developed on the cylindrical surface. This converted image P2 is shown in the upper right of FIG.

  In step S3, the face recognition unit 33 performs face recognition processing. That is, as shown in the lower right diagram of FIG. 3, the human face image 51 included in the converted image obtained in step S2 is recognized from the converted image P2. This recognition process can be performed, for example, by searching for a region having characteristics such as skin color, eyes, mouth, ears, and nose.

  In the fisheye image P1 obtained by the imaging process in step S1, the upward direction in the three-dimensional space is the center of the image. Therefore, it is difficult to detect the face with a detector that assumes that the upward direction in the three-dimensional space is the upward direction of the image, and to detect the size quickly and accurately. Therefore, in the present embodiment, a converted image P2 is generated. This facilitates the upward detection in the three-dimensional space, and as a result, the face can be easily detected. That is, as shown in the lower right of FIG. 3, a face image 51 is detected from the converted image P2.

  In step S4, the determination unit 34 determines whether a face is recognized. If the face is not recognized, the process returns to step S1, and the processes of steps S1 to S4 are repeated until the face is recognized.

  If it is determined in step S4 that the face has been recognized, in step S5, the detection unit 35 detects the recognized face from the coordinate system of the fisheye image. That is, as shown in the left center of FIG. 3, a face image 51 on the fisheye image P1 is detected.

  In step S6, the detection unit 35 detects the size of the face, that is, the size. Specifically, the size of the face image 51 on the fish-eye image P1 shown at the left center of FIG. 3 is detected.

  FIG. 4 is a diagram for explaining the relationship between the size of the subject image and the distance. As shown in FIG. 4, in the XYZ world coordinate system, the face image 51-1 exists at a position at a distance R1 farther from the camera 11, and the face image 51-2 exists at a position at a closer distance R2. Then. In this case, when comparing the size of the face image 51-1 on the fish-eye image P <b> 1 and the face image 51-2, the face image 51-2 closer to the distance is the face image 51 farther away. Greater than -1.

FIG. 5 is a graph showing the relationship between the size of the subject image and the distance. In FIG. 5, the horizontal axis represents the size of the face image 51 (that is, the face size), and the vertical axis represents the distance R between the camera 11 and the face. As indicated by a curve L in FIG. 5, there is a negative correlation between the face size S and the distance R. That is, as the distance R increases, the face size S decreases. The curve L has a shape that protrudes downward. The curve L is represented by the following formula.
R = fs (S) (1)

  In step S7, the calculation unit 36 obtains the distance from the face size based on the relational expression. Here, the relational expression is an expression represented by a curve L in FIG. 5, that is, an expression (1) representing a relationship between the face size S and the distance R. A distance R corresponding to the face size S detected in the process of step S6 is obtained from the equation (1). Expression (1) is stored in the memory 17 in advance.

  In step S8, the coordinate conversion unit 37 converts the coordinate system to the world coordinate system.

FIG. 6 is a diagram for explaining a coordinate system of a fish-eye image captured by the fish-eye lens 11A. In the fisheye image coordinate system xy of FIG. 6, the coordinates (x _C , y _C ) are coordinates where the camera 11 exists. r _C is the field radius of the fisheye lens 11A. Assume that a face image 51 located at a distance R from the camera 11 in the world coordinate system is located at coordinates (x, y) in the coordinate system of the fisheye image. The angle in the clockwise direction with respect to the y coordinate axis of the straight line connecting the coordinates (x _C , y _C ) and the coordinates (x, y) is defined as θ. The elevation angle of the face image 51 from the camera 11 on the XY plane of the world coordinate system, that is, the elevation angle from the horizontal plane is defined as φ. At this time, the following equation holds.

x = x _C −r _C × cos φ × sin θ (2)
y = y _C + r _C × cosφ × cosθ (3)

From the above equations (2) and (3), the following equation is obtained.
r _C × cos φ = (x _C −x) / sin θ (4)
y = y _C + ((x _C −x) / sin θ) × cos θ (5)
y = y _C + (x _C −x) / tan θ (6)
θ = tan ⁻¹ ((x _C −x) / (y−y _C )) (7)
r _C × cos φ × sin θ = x _C −x (8)
cosφ = (x _C −x) / (r _C × sin θ) (9)
φ = cos ⁻¹ ((x _C −x) / (r _C × sin θ)) (10)

  Based on the above equation, the calculation unit 36 can obtain the angles θ and φ from the coordinates (x, y) of the face image 51.

FIG. 7 is a diagram illustrating the world coordinate system. As shown in FIG. 7 (and the lower diagram on the left side of FIG. 3), the coordinates (X, Y, Z) in the world coordinate system of the face image 51 are expressed by the following equations.
X = Rcosφ × cosθ (11)
Y = Rcosφ × sinθ (12)
Z = Rsinφ (13)

  The coordinate conversion unit 37 can calculate the coordinates (X, Y, Z) of the face image 51 based on the above formulas (11) to (13), using the distance R obtained in step S7 as a parameter. . The angles θ and φ can be calculated by the above formulas (7) and (10).

  In step S9, the determination unit 34 determines whether to end the estimation process. If the end of the process is not instructed by the user, the process returns to step S1 and the subsequent processes are repeated.

  If it is determined in step S9 that the user has instructed the end of the process, the estimation process ends.

  As described above, the world coordinates can be obtained quickly and accurately from the image of one camera 11 having one fisheye lens 11A. As a result, the movement trajectory of the face image 51 in the world coordinate system can be predicted, or the camera 11 can be pointed in a direction in which the face image 51 can be more easily recognized.

<1. First Embodiment>
[Learning process]

  As described above, when the three-dimensional position estimation process of FIG. 2 is executed, the relationship between the distance and the face size needs to be known in advance. That is, it is necessary to obtain a relational expression in advance. Therefore, a method for learning in advance and obtaining a relational expression between the distance and the face size will be described with reference to FIGS. In the embodiment shown in FIGS. 8 and 9, a plurality of relational expressions for a plurality of faces having different sizes are learned.

  FIG. 8 is a flowchart for explaining the learning process. Note that the processes in steps S41, S42, S44, and S45 in FIG. 8 are the same as the processes in steps S1, S2, S3, and S6 in FIG. 2, and these processes are repeated and will be briefly described. .

  In step S <b> 41, the imaging unit 31 captures an image with the camera 11. A surrounding object is photographed by the camera 11 via the fisheye lens 11A, and the image is acquired as a fisheye image.

  In step S42, the image conversion unit 32 converts the image. That is, the fisheye image P1 acquired in step S41 is converted into the converted image P2.

  In step S43, the detection unit 35 detects the distance at which a face having a predetermined size is arranged. That is, the distance to the face is measured by the stereo camera 18 or a distance measuring device provided separately, and the result is acquired.

  In step S44, the face recognition unit 33 executes face recognition processing. That is, the human face image included in the converted image obtained in step S42 is recognized.

  In step S45, the detection unit 35 detects the size of the face.

  Next, in step S46, the storage unit 41 stores the relationship between the distance and the face size. That is, the distance detected in step S43 and the face size detected in step S45 are associated with each other and stored in the memory 17.

  In step S47, the determination unit 34 determines whether the necessary range has been measured. That is, it is determined whether the measurement for the monitored range is completed.

  If the measurement of the necessary range has not been completed, the instruction unit 38 instructs to change the distance in step S48. For example, a message instructing the change of the distance is displayed on the output unit 15. The user changes the position of the face according to this instruction.

  Thereafter, the processing returns to step S41, and the same processing is executed for the face newly placed at the position.

  If it is determined in step S47 that the measurement of the necessary range has been completed, the determination unit 34 determines in step S49 whether all the sizes have been measured. If the faces of all sizes have not been measured yet, the instruction unit 38 instructs to change the size of the face in step S50. For example, a message instructing to change the face size is displayed on the output unit 15. The user changes the size of the face according to this instruction.

  Thereafter, the process returns to step S41, and the same process is executed for the face of a new size.

  When it is determined in step S49 that faces of all sizes have been measured, in step S61, the calculation unit 36 calculates a relational expression between the face of each size and the distance.

FIG. 9 is a diagram for explaining an expression representing the relationship between the size of the subject image and the distance. As described above, for example, the curve L1 representing the relationship between the distance and the face size for the large face image 51L is obtained by the least square method or the like, and the relational expression R = f _L ( S) is required.

Similarly, a curve L2 representing the relationship between the distance and the face size for the face image 51M having a medium size is obtained by the least square method or the like, and a relational expression R = f _M (S) representing it. Is required. Further, a curve L3 representing the relationship between the distance and the face size for the face image 51S having a smaller size is obtained by the least square method or the like, and a relational expression R = f _S (S) representing it is obtained.

  The obtained relational expression is stored in the memory 17.

<2. Second Embodiment>
[Three-dimensional position estimation process 2]

  The three-dimensional position estimation process of FIG. 2 is a process when there is one relational expression, but a process when there are a plurality of relational expressions will be described with reference to FIGS. 10 to 14.

  10 and 11 are flowcharts illustrating the three-dimensional position estimation process of the image processing apparatus 1 according to the present technology. Note that the processes in steps S81 to S86 and steps S88 to S90 in FIGS. 10 and 11 are the same as the processes in steps S1 to S9 in FIG. 2, and the processes in steps S87 and S91 to S96 are as follows. This is different from the three-dimensional position estimation process of FIG. In the following, a description will be given mainly of processing different from the case in FIG.

  That is, as in the case of FIG. 2, in steps S81 to S86, the fisheye image obtained by photographing with the camera 11 is converted into a converted image, the face is recognized from the converted image, and the fisheye image of the recognized face is recognized. The upper size is detected.

  After the face size on the fisheye image is detected, the relational expression selection process is executed in step S87. Details of this relational expression selection processing will be described with reference to the flowchart of FIG. In the case of this embodiment, a stereo camera 18 is provided in addition to the camera 11 having the fisheye lens 11A.

  FIG. 12 is a flowchart for explaining the relational expression selection process. In step S <b> 111, the measurement unit 40 measures the distance using the stereo camera 18. That is, the stereo camera 18 images the face, and the measurement unit 40 calculates the distance from the image to the face.

  In step S112, the selection unit 39 selects a relational expression from the distance and the face size.

FIG. 13 is a diagram for explaining the principle of obtaining the distance from the size of the subject image. Assume that three relational expressions R = f _L (S), R = f _M (S), and R = f _S (S) are stored as shown in FIG. When the size of the detected face in the processing of step S85 and _{S 1,} on the relationship R _{= f} L (S), a distance corresponding to the size _{S 1} of the face becomes _{R 1.} On the relational expression R = f _M (S), the distance corresponding to the face size S ₁ is R ₂ , and on the relational expression R = f _S (S), the distance corresponding to the face size S ₁ is R _3. It becomes.

Selecting unit 39, among the distances _{R 1} to _{R 3,} selects a relation corresponding to the value closest to the distance _{R 0} measured in step S111. For example, when the value of the distance R ₁ is closest to the value of the distance R ₀ , the relational expression R = f _L (S) is selected. That is, in this case, the size of the face of the person being photographed is determined to be a large size.

  When one relational expression is selected as described above, processing similar to the processing in steps S7 to S9 and S1 to S6 in FIG. 2 is thereafter performed in steps S88 to S96. That is, in steps S88 and S89, the distance is obtained from the face size based on the selected relational expression, and the coordinate system of the fisheye image is converted into the world coordinate system.

  If it is determined in step S90 that the estimation process has not yet been instructed, the same processes as in steps S81 to S86 are executed in steps S91 to S96. That is, a fisheye image newly taken by the camera 11 is converted into a converted image. Then, face recognition processing is performed until the face is recognized from the converted image, and when the face is recognized, the size of the face is detected. Further, the distance is obtained from the relational expression selected in step S87 based on the detected face size. Thereafter, the same processing is repeated.

  Thus, once a relational expression is selected, the selected relational expression is used thereafter. If there are multiple faces, the detected face is tracked and the same relation is used for the same face.

  As described above, in the case of this embodiment, one relational expression is selected based on the measurement result by the stereo camera 18, and the distance is obtained based on the selected relational expression.

  When the stereo camera 18 is used, the measurement result obtained by the stereo camera 18 can be used as it is in the process of obtaining the distance in step S88. However, there may be a range that can be captured by the camera 11 but cannot be captured by the stereo camera 18. Therefore, if the measurement result obtained by the stereo camera 18 is used only for selection of the relational expression and the distance is detected from the relational expression, it becomes possible to perform uniform detection with less variation as a whole. .

  In the case of the embodiment of FIG. 12, one relational expression is selected based on the measurement result by the stereo camera 18, but it is also possible to select the relational expression by another method. An example of selecting a relational expression by another method will be described with reference to FIG.

  FIG. 14 is a flowchart for explaining the relational expression selection process. This process represents another example of the relational expression selection process in step S87 of FIG. In the case of this embodiment, the person can be specified from the face, and the size of the face of the specified person is registered in the memory 17 in advance.

  In the same manner as described above, in steps S81 to S86, the fisheye image obtained by photographing with the camera 11 is converted into a converted image, the face is recognized from the converted image, and the recognized face on the fisheye image is displayed. The size is detected.

  After the face size on the fisheye image is detected, the relational expression selection process of FIG. 14 is executed in step S87.

  In step S131 in FIG. 14, the detection unit 35 functions as a specifying unit and specifies a face. That is, the face of the person to be monitored and the size of the face are registered in the memory 17 in advance, and the face recognized in the process of step S83 is compared with the face registered in advance and specified.

In step S132, the selection unit 39 selects a relational expression corresponding to the specified face size. For example, the face size of each person is classified into one of the large face image 51L, the medium face image 51M, and the small face image 51S shown in FIG. A relational expression corresponding to the size of the person's face is selected. When the identified face image 51S has a small face size, the relational expression R = f _S (S) is selected.

  When one relational expression is selected as described above, processing similar to that described above is executed in steps S88 to S96 in FIG.

  In this embodiment, since the stereo camera 18 is unnecessary, the configuration can be simplified and the cost can be reduced as compared with the case where the stereo camera 18 is provided. Further, the space for installing the device may be narrow.

<3. Third Embodiment>
[Three-dimensional position estimation process 3]

  In the embodiment of FIG. 2 and the embodiments of FIGS. 10 and 11, it is necessary to prepare the relational expression in advance, but it is possible to add the relational expression as appropriate. Hereinafter, an embodiment in this case will be described with reference to FIGS. 15 and 16. Also in this embodiment, a stereo camera 18 is provided in addition to the camera 11 having the fisheye lens 11A.

  FIGS. 15 and 16 are flowcharts illustrating the three-dimensional position estimation process of the image processing apparatus 1 of the present technology. The processes in steps S161 to S166 and steps S170 to S172 in FIGS. 15 and 16 are the same as the processes in steps S1 to S9 in FIG. The processes in steps S167 to S169 and steps S173 to S181 are different from those in FIG. Hereinafter, mainly different processes will be described.

  In steps S161 to S166, the fisheye image obtained by photographing with the camera 11 is converted into a converted image, the face is recognized from the converted image, and the size of the face on the recognized fisheye image is detected.

  After the face size on the fisheye image is detected, the same processing as in steps S111 and S112 in FIG. 12 is executed in steps S167 and S168. That is, in step S167, the measurement unit 40 measures the distance by the stereo camera 18. At this time, the stereo camera 18 images the face, and the measurement unit 40 calculates the distance from the image to the face.

In step S168, the selection unit 39 selects a relational expression from the distance and the face size. That is, as described above with reference to FIG. 13, the selection unit 39 selects a relational expression corresponding to the value closest to the distance R ₀ measured in step S 166 among the distances R _{1 to} R ₃ . For example, when the value of the distance R ₁ is closest to the value of the distance R ₀ , the relational expression R = f _L (S) is selected.

  As described above, when one relational expression is selected, in step S169, the determination unit 34 determines whether there is an appropriate relational expression. That is, it is determined in step S168 whether an appropriate relational expression has been selected. For example, it is determined whether the difference between the face size detected in step S166 and the face size of each relational expression is within a predetermined threshold. If there is a relational expression in which the difference is within the threshold, an appropriate relation exists. It is determined that the expression exists.

  When it is determined in step S169 that an appropriate relational expression exists, in steps S170 to S172, processing similar to the processing in steps S7 to S9 in FIG. 2 is executed. That is, in steps S170 and S171, the distance is obtained from the face size based on the selected relational expression, and the coordinate system of the fisheye image is converted into the world coordinate system.

  In step S172, the determination unit 34 determines whether the end of the estimation process is instructed. If it is determined in step S172 that the estimation process has not been instructed yet, the processes of steps S173 to S178 and steps S170 to S172 are executed. The processes in steps S173 through S178 are the same as the processes in steps S161 through S166.

  That is, a new fisheye image is acquired and converted into a converted image. Face recognition processing is performed until a face is recognized from the converted image. When the face is recognized, the size of the face on the fisheye image is detected.

  In step S170, the calculation unit 36 obtains a distance corresponding to the size of the face detected in step S178 based on the relational expression selected in step S168. In step S171, the coordinate conversion unit 37 converts the coordinate system to the world coordinate system.

  In step S172, it is determined whether or not the estimation process is to be terminated. If the termination has not been instructed yet, the process returns to step S173, and the subsequent processes are repeated.

  If it is determined in step S169 that an appropriate relational expression does not exist, that is, if the difference between the face size detected in step S166 and the face size of each relational expression is not within the threshold value, step S179 and the subsequent steps are performed. Processing is executed. In step S179, the storage unit 41 stores the distance and the face size. That is, the face size detected in step S166 and the distance measured by the stereo camera 18 in the process of step S167 are stored in the memory 17 in association with each other.

In step S180, the determination unit 34 determines whether data is sufficiently stored. Whether the distance and face size data is sufficient to calculate the relational expression can be determined, for example, as follows. For example, the curves L _{1 to} L ₃ representing the relational expressions of the face sizes shown in FIG. 13 can be considered to be almost similar to each other. In this case, if the coordinates of one point defined by the distance of a predetermined value and the size of the face are determined, a similar curve passing there can be uniquely identified. When thinking in this way, one piece of data is sufficient data.

  It is also possible to calculate an average value of a plurality of coordinates and obtain a similar curve passing through one point represented by the average value. In this case, an arbitrary number of data for obtaining the average value is sufficient data.

  Further, when one relational expression is calculated from a plurality of data by the least square method, the number of data necessary for calculating one relational expression by the least square method is sufficient data.

  If it is determined in step S180 that the data has been sufficiently stored, in step S181, the calculator 36 calculates and adds a relational expression between the predetermined distance and the face size. That is, a new relational expression is stored in the memory 17.

  After the process of step S181, it is determined in step S172 whether or not the end of the estimation process has been instructed. If the end has not been instructed yet, the process proceeds to step S173, and the subsequent processes are executed. If it is determined in step S172 that the end of the estimation process has been instructed, the process ends.

  On the other hand, when it is determined in step S180 that the data has not been sufficiently stored, the process of step S181 may be skipped and the process may proceed to step S172. However, in this embodiment, when it is determined in step S180 that the data is not yet sufficiently stored, in step S182, processing for selecting the closest relational expression is executed. That is, in this case, the difference between the face size detected in step S166 and the face size of each relational expression exceeds the threshold value (determined as such in step S169). The relational expression with the smallest is selected. Thereafter, similarly to the case where it is determined in step S169 that an appropriate relational expression exists, the processing proceeds to step S170, and the subsequent processing is executed.

  As described above, in the case of this embodiment, new relational expressions are added as appropriate.

<4. Fourth Embodiment>
[Three-dimensional position estimation process 4]

  Next, an embodiment for updating the relational expression will be described with reference to FIGS. 17 and 18. Also in this embodiment, a stereo camera 18 is provided in addition to the camera 11 having the fisheye lens 11A. In this embodiment, a specific face whose face size is known is stored in the memory 17 in advance.

  FIGS. 17 and 18 are flowcharts illustrating the three-dimensional position estimation process of the image processing apparatus 1 according to the present technology. The processes in steps S201 to S206 and steps S212 to S214 in FIGS. 17 and 18 are the same as the processes in steps S1 to S9 in FIG. The processes in steps S207 to S211 and steps S215 to S222 are different from those in FIG. Hereinafter, mainly different processes will be described.

  In steps S201 to S206, the fisheye image obtained by photographing with the camera 11 is converted into a converted image, the face is recognized from the converted image, and the size of the face on the recognized fisheye image is detected.

  After the face size on the fisheye image is detected, in step S207, the same processing as in step S111 in FIG. 12 is executed. That is, in step S207, the measurement unit 40 measures the distance using the stereo camera 18. At this time, the stereo camera 18 images the face, and the measurement unit 40 calculates the distance from the image to the face.

In step S208, the determination unit 34 determines whether the face recognized in the process of step S203 is a specific face. If the face recognized in the process of step S203 is not a specific face, the same process as in step S112 of FIG. 12 is executed in step S211. That is, the selection unit 39 selects a relational expression from the distance and the face size. As described above with reference to FIG. 13, the selection unit 39 selects a relational expression corresponding to the value closest to the distance R ₀ measured in step S 207 among the distances R _{1 to} R ₃ . For example, when the value of the distance R ₁ is closest to the value of the distance R ₀ , the relational expression R = f _L (S) is selected.

  As described above, when one relational expression is selected, in steps S212 to S214, processing similar to the processing in steps S7 to S9 in FIG. 2 is executed. That is, in steps S212 to S214, the distance is obtained from the face size based on the selected relational expression, and the coordinate system of the fisheye image is converted into the world coordinate system.

  In step S214, the determination unit 34 determines whether the end of the estimation process is instructed. If it is determined in step S214 that the end of the estimation process has not been instructed yet, the processes of steps S215 to S220 and steps S212 to S214 are executed. The processes in steps S215 to S220 are the same as the processes in steps S201 to S206.

  That is, a new fisheye image is acquired and converted into a converted image. Face recognition processing is performed until a face is recognized from the converted image, and when the face is recognized, the size of the face is detected.

  In step S212, the calculation unit 36 obtains a distance corresponding to the face size detected in the process of step S220, based on the relational expression selected in the process of step S211. In step S213, the coordinate conversion unit 37 converts the coordinate system to the world coordinate system.

  In step S214, it is determined whether or not the estimation process is to be terminated, and if the termination has not been instructed yet, the process returns to step S215 and the subsequent processes are repeated.

  On the other hand, when it is determined in step S208 that the recognized face is a specific face registered in advance, in step S209, the storage unit 41 stores the distance and the size of the face. That is, the face size detected in step S206 and the distance measured in step S207 are stored in the memory 17 in association with each other.

  In step S210, the determination unit 34 determines whether data is sufficiently stored. Whether or not the distance and face size data are sufficient to calculate the relational expression can be determined in the same manner as in step S180 of FIG. 15, for example.

That is, for example, when the curves L _{1 to} L ₃ representing the relational expressions of the face sizes shown in FIG. 13 are substantially similar to each other, the coordinates of one point defined by the predetermined distance and the face size are used. If it is determined, a similar curve passing there can be uniquely identified. When thinking in this way, one piece of data is sufficient data.

  It is also possible to calculate an average value of a plurality of coordinates and obtain a similar curve passing through one point represented by the average value. In this case, an arbitrary number of data for obtaining the average value is sufficient data.

  Further, when one relational expression is calculated from a plurality of data by the least square method, the number of data necessary for calculating one relational expression by the least square method is sufficient data.

  If it is determined in step S210 that the data is sufficient, in step S221, the update unit 42 calculates and updates the relational expression between the specific distance and the face size. The exact value of the specific face size is registered in the memory 17 in advance. Therefore, the face size detected in step S206 is associated with a pre-registered face size. An accurate relational expression can be obtained by performing calculations based on the size of the face and the distance measured by the stereo camera 18 in step S207.

  When the relational expression is updated, in step S222, the selection unit 39 selects the updated relational expression. That is, the relational expression just updated in step S221 is selected.

  Thereafter, the process proceeds to step S212, and the subsequent processes are executed.

  If it is determined in step S210 that the data is not yet sufficiently stored, the processes in steps S221 and S222 are skipped. In this case, in step S211, a relational expression is selected from the distance and the face size. That is, in this case, the relational expression is not updated, and the previous relational expression specified based on the face size detected in step S206 and the distance measured in step S207 is selected. Thereafter, the process proceeds to step S212, and the subsequent processes are executed.

  Thus, in this embodiment, when a face with a known size is detected, the relational expression is updated, so that the relational expression can be made more accurate and an accurate position can be detected based on the relational expression. become.

[5. Modified example]

  Examples of other converted images of fisheye images will be described. FIG. 19 is a diagram for explaining image development. In the example of FIG. 19, one fisheye image is divided into four regions A1 to A1, an upper left, an upper right, a lower left, and a lower right by a horizontal virtual line and a vertical virtual line passing through the center. It is divided into A4. Then, one fisheye image is converted into a conversion image to be processed that includes two regions that are adjacent to each other on the left and right or top and bottom. That is, the image is converted into a converted image of region A3 and region A4, a converted image of region A4 and region A2, a converted image of region A2 and region A1, and a converted image of region A1 and region A3.

  Even when the face is recognized from the converted image thus converted, the face can be easily recognized as compared with the case where the face is recognized from one fisheye image itself.

  However, in this case, in each converted image, the face is inclined up to ± 45 degrees. Therefore, the face recognizer needs to have a performance capable of recognizing a face inclined at ± 45 degrees.

  In the above, the distance is measured by the stereo camera 18, but other distance measuring means can be used.

  It is also conceivable to develop a fisheye image on a plane.

  The present technology can be applied to, for example, a monitoring device that monitors a predetermined space, a device that visually recognizes the surroundings by a robot, and other image processing devices. The recognition target is not limited to a human face. Animals, various objects, etc. can be recognized.

 [6. Application of this technology to programs]

  The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software is installed in the computer. Here, the computer includes, for example, a general-purpose personal computer capable of executing various functions by installing various programs by installing a computer incorporated in dedicated hardware.

  FIG. 20 is a block diagram showing a configuration of a personal computer that executes the above-described series of processing by a program.

  In the computer, a CPU (Central Processing Unit) 221, a ROM (Read Only Memory) 222, and a RAM (Random Access Memory) 223 are connected to each other via a bus 224.

  An input / output interface 225 is further connected to the bus 224. An input unit 226, an output unit 227, a storage unit 228, a communication unit 229, and a drive 230 are connected to the input / output interface 225.

  The input unit 226 includes a keyboard, a mouse, a microphone, and the like. The output unit 227 includes a display, a speaker, and the like. The storage unit 228 includes a hard disk, a nonvolatile memory, and the like. The communication unit 229 includes a network interface. The drive 230 drives a removable medium 231 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

  In the computer configured as described above, the CPU 221 loads the program stored in the storage unit 228 to the RAM 223 via the input / output interface 225 and the bus 224 and executes the program, for example. Is performed.

  The program executed by the computer (CPU 221) can be provided by being recorded on a removable medium 231 as a package medium, for example. The program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

  In the computer, the program can be installed in the storage unit 228 via the input / output interface 225 by attaching the removable medium 231 to the drive 230. The program can be received by the communication unit 229 via a wired or wireless transmission medium and installed in the storage unit 228. In addition, the program can be installed in the ROM 222 or the storage unit 228 in advance.

  The program executed by the computer may be a program that is processed in time series in the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program for processing.

  The embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present technology.

[7. Others]

The present technology can be configured as follows.
(1)
An acquisition unit that acquires a fisheye image that is an image taken through a fisheye lens;
A conversion unit that converts the fisheye image and generates a converted image;
An image processing apparatus comprising: a recognition unit that recognizes an image to be recognized from the converted image.
(2)
The image processing apparatus according to (1), wherein the converted image is a developed image obtained by developing the fisheye image on a cylindrical surface.
(3)
The image processing apparatus according to (1) or (2), further including a detection unit configured to detect a size of the recognition target image on the developed image.
(4)
The image processing apparatus according to (3), further including a calculation unit that calculates a distance to the recognition target from a size of the image of the recognition target.
(5)
The image processing apparatus according to (4), wherein the calculation unit calculates a distance to the recognition target corresponding to a size of the image to be recognized, from a relational expression that defines a relationship between the distance and the size.
(6)
The image processing apparatus according to (5), further including a selection unit that selects one of the plurality of relational expressions prepared for the plurality of recognition targets having different sizes.
(7)
The image processing apparatus according to (6), further including a measurement unit that measures a distance in order to select one of the relational expressions.
(8)
The image processing apparatus according to (6), further including a specifying unit that specifies the specific recognition target in order to select one of the relational expressions.
(9)
The image processing apparatus according to any one of (5) to (8), wherein the calculation unit further calculates and adds a new relational expression.
(10)
The image processing apparatus according to any one of (5) to (9), further including an update unit that updates the relational expression.
(11)
The coordinate system according to any one of (5) to (10), further including a coordinate conversion unit that converts the coordinate system of the fisheye image into a world coordinate system using the distance to the recognition target detected from the relational expression as a parameter. The image processing apparatus described.
(12)
An acquisition step of acquiring a fisheye image, which is an image taken through a fisheye lens;
Converting the fisheye image to generate a converted image;
A recognition step of recognizing an image to be recognized from the converted image.
(13)
On the computer,
An acquisition step of acquiring a fisheye image, which is an image taken through a fisheye lens;
Converting the fisheye image to generate a converted image;
A computer-readable recording medium storing a program for executing a recognition step of recognizing an image to be recognized from the converted image.
(14)
On the computer,
An acquisition step of acquiring a fisheye image, which is an image taken through a fisheye lens;
Converting the fisheye image to generate a converted image;
A recognition step of recognizing an image to be recognized from the converted image.

DESCRIPTION OF SYMBOLS 1 Image processing apparatus, 11 Camera, 11A Fisheye lens, 13 Microprocessor, 31 Image pick-up part, 32 Image conversion part, 33 Face recognition part, 34 Determination part, 35 Detection part, 36 Calculation part, 37 Coordinate conversion part, 38 Instruction part, 39 selection unit, 40 measurement unit, 41 storage unit, 42 update unit, 43 control unit

Claims (14)

An acquisition unit that acquires a fisheye image that is an image taken through a fisheye lens;
A conversion unit that converts the fisheye image and generates a converted image;
An image processing apparatus comprising: a recognition unit that recognizes an image to be recognized from the converted image. The image processing apparatus according to claim 1, wherein the converted image is a developed image obtained by developing the fisheye image on a cylindrical surface. The image processing apparatus according to claim 2, further comprising a detection unit that detects a size of the image to be recognized on the developed image. The image processing apparatus according to claim 3, further comprising a calculation unit that calculates a distance to the recognition target from a size of the image of the recognition target. The image processing apparatus according to claim 4, wherein the calculation unit calculates a distance to the recognition target corresponding to a size of the image to be recognized, from a relational expression that defines a relationship between the distance and the size. The image processing apparatus according to claim 5, further comprising a selection unit that selects one of the plurality of relational expressions prepared for the plurality of recognition targets having different sizes. The image processing apparatus according to claim 6, further comprising a measurement unit that measures a distance in order to select one of the relational expressions. The image processing apparatus according to claim 6, further comprising a specifying unit that specifies a specific recognition target in order to select one of the relational expressions. The image processing apparatus according to claim 6, wherein the calculation unit further calculates and adds the new relational expression. The image processing apparatus according to claim 6, further comprising an update unit that updates the relational expression. The image processing apparatus according to claim 5, further comprising a coordinate conversion unit that converts a coordinate system of the fisheye image into a world coordinate system using a distance to the recognition target detected from the relational expression as a parameter. An acquisition step of acquiring a fisheye image, which is an image taken through a fisheye lens;
Converting the fisheye image to generate a converted image;
A recognition step of recognizing an image to be recognized from the converted image. On the computer,
An acquisition step of acquiring a fisheye image, which is an image taken through a fisheye lens;
Converting the fisheye image to generate a converted image;
A computer-readable recording medium storing a program for executing a recognition step of recognizing an image to be recognized from the converted image. On the computer,
An acquisition step of acquiring a fisheye image, which is an image taken through a fisheye lens;
Converting the fisheye image to generate a converted image;
A recognition step of recognizing an image to be recognized from the converted image.

Images