JP2019175283A - Recognition apparatus, recognition system, program, and position coordinate detecting method - Google Patents

Abstract

To measure positional information of an object in a predetermined area from a motion picture shot by a camera having a fisheye lens.SOLUTION: A recognition apparatus has conversion means for converting a fisheye motion picture obtained by shooting a predetermined area by a camera with a fisheye lens into a plurality of distortion correction element images, recognition means for carrying out of recognition processing for an object in the distortion correction element images, first position coordinate detecting means for calculating positional coordinates of the recognized object in the distortion correction element images, and second position coordinate detecting means for calculating position coordinates of the recognized object in the predetermined area from the position coordinates in the distortion correction element images.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a recognition device, a recognition system, a program, and a position coordinate detection method.

  2. Description of the Related Art Conventionally, in order to visualize where and where an object is, it has been performed to measure the position of the object and analyze the position information of the object by analyzing a moving image of a predetermined area.

  Patent Document 1 discloses a technique for performing object recognition processing using a low-resolution image and a high-resolution image in detection of an object using a camera capable of capturing 360-degree omnidirectional images. ing.

  However, according to the conventional technology, distortion occurs in a moving image captured by a camera capable of capturing 360-degree omnidirectional images, and position information of an object in a predetermined region is measured from the distorted image. There was a problem that was difficult.

  The present invention has been made in view of the above, and an object thereof is to measure position information of an object in a predetermined region from a moving image captured by a camera having a fisheye lens.

  In order to solve the above-described problems and achieve the object, the present invention provides a conversion means for converting a fish-eye moving image obtained by photographing a predetermined area by a camera equipped with a fish-eye lens into a plurality of distortion correction element images, and the distortion correction element image. A recognition means for performing recognition processing of the object, first position coordinate detection means for obtaining a position coordinate of the recognized object in the distortion correction element image, and position coordinates in the distortion correction element image. And second position coordinate detection means for obtaining position coordinates of the object in the predetermined area.

  According to the present invention, there is an effect that position information of an object in a predetermined region can be measured from a moving image captured by a camera having a fisheye lens.

FIG. 1 is a diagram illustrating a hardware configuration of the recognition system according to the embodiment. FIG. 2 is a block diagram illustrating a hardware configuration of the fisheye camera. FIG. 3 is a block diagram illustrating a hardware configuration of the recognition apparatus. FIG. 4 is a block diagram illustrating a functional configuration of the recognition device. FIG. 5 is a diagram illustrating an example of an image input by a fisheye camera. FIG. 6 is a diagram for explaining the unit sphere. FIG. 7 is a diagram for explaining the coordinate system of the equirectangular projection. FIG. 8 is a diagram for explaining a coordinate system for perspective projection. FIG. 9 is a diagram illustrating a setting example of a plurality of azimuth angles and a plurality of field angles. FIG. 10 is a diagram illustrating an example of creating a distortion correction element image in three directions. FIG. 11 is a diagram exemplarily showing a coordinate system and position coordinates in the work area. FIG. 12 is a diagram exemplarily showing a coordinate system and position coordinates in a distortion correction element image. FIG. 13 is a diagram illustrating block scanning of the human recognition process. FIG. 14 is a diagram illustrating a feature amount for human recognition. FIG. 15 is a diagram illustrating a hierarchical structure of human recognition processing. FIG. 16 is a diagram illustrating an example of the result of person recognition. FIG. 17 is a flowchart schematically showing the flow of processing in the recognition apparatus.

  Hereinafter, embodiments of a recognition device, a recognition system, a program, and a position coordinate detection method will be described in detail with reference to the accompanying drawings.

  Here, FIG. 1 is a diagram illustrating a hardware configuration of the recognition system 100 according to the embodiment. As shown in FIG. 1, the recognition system 100 includes a fisheye camera 200 and a recognition device 300. The recognition apparatus 300 includes a recognition processing unit 321 and an interface unit 322 that connects the recognition processing unit 321 and the fisheye camera 200.

  First, the hardware configuration of the fisheye camera 200 will be described.

  Here, FIG. 2 is a block diagram showing a hardware configuration of the fisheye camera 200. As shown in FIG. 2, the fish-eye camera 200 includes a fish-eye lens 201 and a CCD (Charge Coupled Device) 203 having a field angle with a diagonal field angle of 180 degrees or more. The fisheye camera 200 causes subject light to enter the CCD 203 through the fisheye lens 201. Further, the fisheye camera 200 includes a mechanical shutter 202 between the fisheye lens 201 and the CCD 203. The mechanical shutter 202 blocks incident light to the CCD 203. The fisheye lens 201 and the mechanical shutter 202 are driven by a motor driver 206.

  The CCD 203 converts an optical image formed on the imaging surface into an electrical signal and outputs it as analog image data. The image information output from the CCD 203 has its noise component removed by a CDS (Correlated Double Sampling) circuit 204, converted into a digital value by an A / D converter 205, and then sent to an image processing circuit 208. Is output.

  The image processing circuit 208 performs various image processing such as YCrCb conversion processing, white balance control processing, contrast correction processing, edge enhancement processing, and color conversion processing using an SDRAM (Synchronous DRAM) 212 that temporarily stores image data. The white balance process is an image process that adjusts the color density of image information, and the contrast correction process is an image process that adjusts the contrast of image information. The edge enhancement process adjusts the sharpness of image information, and the color conversion process is an image process that adjusts the hue of image information. The image processing circuit 208 displays image information subjected to signal processing and image processing on a liquid crystal display 216 (hereinafter referred to as LCD 16).

  Further, the image processing circuit 208 generates an equirectangular image obtained by changing the fisheye image input from the fisheye lens 201 using equirectangular projection.

  Image information subjected to signal processing and image processing in the image processing circuit 208 is recorded in the memory card 214 via the compression / decompression circuit 213. The compression / decompression circuit 213 compresses the image information output from the image processing circuit 208 and outputs the compressed image information to the memory card 214 according to the instruction acquired from the operation unit 215, and decompresses the image information read from the memory card 214 to generate an image. The data is output to the processing circuit 208.

  The fisheye camera 200 includes a CPU (Central Processing Unit) 209 that performs various arithmetic processes according to a program. The CPU 209 includes a ROM (Read Only Memory) 211 which is a read only memory storing programs and the like, and a RAM (Random Access) which has a work area used in various processing processes, various data storage areas, and the like. Memory) 10 and a bus line.

  The timing of the CCD 203, the CDS circuit 204, and the A / D converter 205 is controlled by the CPU 209 via a timing signal generator 207 that generates a timing signal. Further, the image processing circuit 208, the compression / decompression circuit 213, and the memory card 214 are also controlled by the CPU 209.

  The output of the fisheye camera 200 is input to an interface unit 322 that is a signal processing board of the recognition apparatus 300 shown in FIG.

  Next, the hardware configuration of the recognition apparatus 300 will be described.

  Here, FIG. 3 is a block diagram showing a hardware configuration of the recognition apparatus 300. As illustrated in FIG. 3, the recognition apparatus 300 includes a CPU (Central Processing Unit) 301 that controls the operation of the entire recognition apparatus 300, a ROM (Read Only Memory) 302 that stores a program used to drive the CPU 801, and a work of the CPU 301. A RAM (Random Access Memory) 303 used as an area is included. Also, an HD (Hard Disk) 304 for storing various data such as programs and the like, and an HDD (Hard Disk Drive) 305 for controlling reading or writing of various data with respect to the HD 304 according to the control of the CPU 301 are provided.

  The recognition apparatus 300 includes a media I / F 307, a display 308, and a network I / F 309. The media I / F 307 controls reading or writing (storage) of data with respect to the medium 306 such as a flash memory. The display 308 displays various information such as a cursor, menu, window, character, or image. A network I / F 309 performs data communication using a communication network.

  The recognition apparatus 300 includes a keyboard 311, a mouse 312, a CD-ROM (Compact Disk Read Only Memory) drive 314, and a bus line 310. The keyboard 311 includes a plurality of keys for inputting characters, numerical values, various instructions, and the like. The mouse 312 performs selection and execution of various instructions, selection of a processing target, movement of a cursor, and the like. The CD-ROM drive 314 controls reading or writing of various data with respect to a CD-ROM 313 as an example of a removable recording medium. The bus line 310 is an address bus, a data bus, or the like for electrically connecting the above components.

  The hardware configuration of the illustrated recognition device 300 does not need to be housed in a single casing or provided as a single device, and represents a hardware element that the recognition device 300 preferably includes. . Further, in order to support cloud computing, the physical configuration of the recognition apparatus 300 according to the present embodiment may not be fixed, and hardware resources may be dynamically connected / disconnected according to the load. May be configured.

  The program is distributed in an executable format, a compressed format, or the like stored in a storage medium such as the medium 306 or the CD-ROM 313, or distributed from a server that distributes the program.

  The program executed by the recognition apparatus 300 according to the present embodiment has a module configuration including the following functions. The CPU 301 of the recognition device 300 reads out a program from a storage medium such as the ROM 302 and the HD 304 and executes the program, whereby each module is loaded onto the RAM 303 and performs each function.

  FIG. 4 is a block diagram illustrating a functional configuration of the recognition apparatus 300. As shown in FIG. 4, the recognition processing unit 321 of the recognition apparatus 300 according to the present embodiment includes a fisheye camera moving image input unit 101, a distortion correction element image generation unit 102 that functions as a conversion unit, and a distortion correction element image generation parameter. Input unit 103, human recognition dictionary input unit 104, human recognition processing unit 105 functioning as a recognition unit, work area human position calculation unit 106 functioning as a first position coordinate detection unit and a second position coordinate detection unit, work region human position A measurement result output unit 107 is provided.

  The fisheye camera moving image input unit 101 inputs a fisheye moving image obtained by photographing a work area (predetermined area) such as an office or a factory with the fisheye camera 200.

  Here, FIG. 5 is a diagram illustrating an example of an image input by the fisheye camera 200. As shown in FIG. 5, the image input by the fisheye camera 200 is an equirectangular image obtained by changing the fisheye image using equirectangular projection.

  FIG. 6 is a diagram for explaining the unit sphere. In FIG. 6, a unit sphere having a radius of 1 is shown. The imaging of the fisheye camera 200 will be described using this unit sphere. As shown in FIG. 6, the light that enters the fisheye camera 200 has an incident angle with respect to the equatorial plane of the unit sphere, that is, the latitude indicating the elevation angle is latitude. The longitude indicating the azimuth angle of the incident light is longitude.

  FIG. 7 is a diagram for explaining the coordinate system of the equirectangular projection. As shown in FIG. 7, the horizontal axis is longitude longitude, and the vertical axis is latitude latitude. The angle range of longitude on the horizontal axis is (−180 degrees to 180 degrees), and the angle range of longitudinal latitude is (−90 degrees to 90 degrees). The vertical direction and the horizontal direction intersect at equal intervals. That is, the vertical direction and the horizontal direction are equally spaced angles.

  The distortion correction element image generation unit 102 generates an element image in a plurality of directions by a perspective projection method, and generates a distortion correction element image. The distortion correction element image generation unit 102 covers the work area with element images in a plurality of directions. The distortion correction element image generation parameter input unit 103 inputs parameters for creating a distortion correction element image in order to create element images in a plurality of directions. As input parameters, the number of element images, the image size of each element image, the field angle, the azimuth angle, the elevation angle, and the rotation angle are input.

  FIG. 8 is a diagram for explaining a coordinate system for perspective projection. As shown in FIG. 8, in order to correct fisheye distortion and create a distortion correction element image, a distortion correction element image is created by perspective projection. As shown in FIG. 8, the perspective projection method is a projection method for drawing on a two-dimensional plane as if a three-dimensional object was seen.

  The distortion correction element image generation unit 102 generates a distortion correction element image by the perspective projection method shown in FIG. 8 in order to generate a distortion correction element image. For this reason, as the one-pixel coordinates (xp, yp) of the distortion correction element image, the corresponding coordinates (longitude, latitude) in the input fisheye image shown in FIG. 5 may be obtained.

  As shown in FIG. 8, the distortion correction element image generation unit 102 calculates (longitude, latitude) longitude and latitude from the element image pixel (xp, yp) according to the following expressions (1) and (2). To do.

  Expressions (1) and (2) are calculation expressions when the azimuth angle, elevation angle, and rotation angle of the distortion correction element image are all zero. When generating a plurality of distortion correction element images, the distortion correction element image generation unit 102 calculates coordinates and longitude / latitude (longitude, latitude) in the fisheye image at each azimuth angle, elevation angle, and rotation angle. The distortion correction element image generation unit 102 performs coordinate conversion in the fisheye image at each azimuth, elevation, and rotation angle based on the results of Expression (1) and Expression (2) by the rotation conversion of the unit sphere illustrated in FIG. Calculate latitude (longitude, latitude).

  Equation (3) is a unit ball rotation conversion equation. Here, α is an azimuth angle, β is an elevation angle, and γ is a rotation angle. These parameters are input from the distortion correction element image generation parameter input unit 103. The distortion correction element image generation unit 102 calculates (x, y, z) coordinates on the unit sphere from the obtained longitude, latitude (longitude, latitude).

  The distortion correction element image generation unit 102 obtains coordinates on the unit sphere of (x ′, y ′, x ′) by the unit sphere rotation matrix calculation of Expression (3) based on the azimuth angle, the elevation angle, and the rotation angle. From these coordinates, longitude and latitude coordinates (longitude ', latitude') are obtained. That is, the coordinate value on the fisheye image.

  The distortion correction element image generation unit 102 creates a pixel value at coordinates (longitude ′, latitude ′) on the fisheye image, and gives the pixel value to the (xp, yp) pixel value of the distortion correction element image. In this way, the distortion correction element image generation unit 102 creates a distortion correction element image.

  Here, a method for generating distortion correction element images in a plurality of directions will be described. It is necessary to set the azimuth and angle of view of each of the plurality of distortion correction element images. Therefore, a method for setting each azimuth and angle of view will be described. When the number of images to be divided is set, if the number is set large, the angle of view of each corrected image becomes small. That is, the number of divisions can be set as a parameter.

  Here, FIG. 9 is a diagram illustrating an example of setting a plurality of azimuth angles and a plurality of field angles. The example shown in FIG. 9 shows an example of the division number 3. As illustrated in FIG. 9, the distortion correction element image generation unit 102 divides the image into three parts within a range of 180 degrees, and sets the respective azimuth angles and field angles. The distortion correction element image generation unit 102 has a boundary between the images, and therefore provides a region that partially overlaps each other and sets each angle of view a little wider. The distortion correction element image generation unit 102 performs distortion correction at each set azimuth angle and field angle.

  The distortion correction element image generation unit 102 creates a plurality of distortion correction element images using a plurality of azimuth angles, elevation angles, and rotation angles. An example of creating a distortion correction element image in three directions is shown in FIG.

  It is necessary to recognize a person from the distortion correction element image and calculate the position coordinates in the work area from the position coordinates of the recognition result. Therefore, pre-calibration is performed to obtain a conversion formula.

  Here, FIG. 11 is a diagram exemplarily showing a coordinate system and position coordinates in the work area. In the present embodiment, calibration is performed by providing a marker on the ground of the work area. As shown in FIG. 11, for example, a black dot marker M is set in the work area. The coordinates (X, Y) of the marker M are measured in advance and set to known values. In FIG. 5, a white marker is provided.

  Here, FIG. 12 is a diagram exemplarily showing a coordinate system and position coordinates in the distortion correction element image. As shown in FIG. 12, the corresponding distortion correction element image forms an image of the marker M provided on the ground of the work area. The position of the marker M is detected from the distortion correction element image, and the coordinates of the marker are (x, y). Since the ground of the work area and the ground of the distortion correction element image created by the perspective projection method are flat, they have a projective transformation relationship. Conversion is performed by the following equation (4).

Since there are eight m _ij as unknowns in equation (4), if there are four or more corresponding points (X, Y) and (x, y), the coefficient m _ij can be obtained.

  The human recognition processing unit 105 performs human recognition processing using the distortion correction element image. More specifically, the human recognition processing unit 105 performs human recognition processing on the distortion correction element image created by the distortion correction element image generation unit 102.

  The human recognition dictionary input unit 104 inputs a human recognition dictionary used for the human recognition processing in the human recognition processing unit 105. More specifically, the human recognition dictionary input unit 104 creates a human recognition dictionary using human learning data in advance by a machine learning method.

  Here, FIG. 13 is a diagram showing block scanning of the human recognition processing. As shown in FIG. 13, in order to recognize a person, the person recognition processing unit 105 first cuts out a rectangular block 1 from the image within the range of the distortion correction element image. The upper left coordinates (Xs, Ys) and the lower right coordinates (Xe, Ye) in the rectangular block 1 are determined by the position of the block 1 in the image and the size of the block 1.

  The rectangular blocks are selected in order from the largest size to the smallest size. The reason is that in this method, normalization of the block is performed, so that the processing time of the large block and the small block is the same. Since the number of candidates for a large block in the image is small and the number of small blocks is large, an object can be detected earlier when selected from large blocks. When a large image is detected, the sensory speed increases when it is output.

  The human recognition processing unit 105 calculates a feature amount for human recognition. Here, FIG. 14 is a diagram showing a feature amount of human recognition. As shown in FIG. 14, the human recognition processing unit 105 calculates a black and white rectangular feature amount in the block. That is, the human recognition processing unit 105 adds the pixel value in the white area to the black and white rectangular area in the block, and the difference from the total pixel value in the black pixel area is the feature amount h (x) in the block area. And In FIG. 14, examples of four characteristics A, B, C, and D are given.

The calculated rectangular feature value is used to calculate a feature value weighted evaluation value f (x) as shown in Equation (5). As shown in Expression (5), the human recognition processing unit 105 calculates the feature value h _t (x) in the block, adds the weight coefficient α _t , and calculates the evaluation value f (x).

As shown in Expression (5), the evaluation function has a feature quantity h _t (x) and a weight coefficient α _t . The human recognition dictionary input unit 104 calculates feature amounts and weighting factors in advance by a machine learning method. The human recognition dictionary input unit 104 collects and learns learning data for a human recognition target, and obtains a feature amount and a weight coefficient.

  Here, FIG. 15 is a diagram showing a hierarchical structure of human recognition processing. As shown in FIG. 15, the human recognition process has a hierarchical structure. Each hierarchy has an evaluation function shown in Expression (5). If the value of the evaluation function is smaller than a preset threshold value, the person recognition processing unit 105 determines that the person is not a person and stops evaluating the block (non-human block). The person recognition processing unit 105 calculates an evaluation value at each layer. The person recognition processing unit 105 determines that the block that is not determined to be a person in the last hierarchy is a person.

  The human recognition dictionary input unit 104 learns the feature value and weighting coefficient for calculating the evaluation value at each level of the human recognition processing unit 105 and the evaluation value threshold value at each level by using learning images that are not human and human. To create a dictionary for human recognition.

  Here, FIG. 16 is a diagram illustrating an example of a result of person recognition. Frames 500A and 500B shown in FIG. 16 are obtained by surrounding a result of human recognition with a rectangle to form a human area. The coordinate of the center point of the bottom of the rectangle is the position coordinate for human recognition in the distortion correction element image.

  The work area human position calculation unit 106 uses the distortion correction element image and the coordinate conversion formula (4) of the work area to calculate the human position coordinates (x on the distortion correction element image obtained by the human recognition processing in the human recognition processing unit 105. , Y) is converted to obtain the position (X, Y) of the person in the work area.

  The work area person position measurement result output unit 107 outputs the person position (X, Y) in the work area obtained by the work area person position calculation unit 106.

  Here, FIG. 17 is a flowchart schematically showing the flow of processing in the recognition apparatus 300. As shown in FIG. 17, first, the fisheye camera moving image input unit 101 inputs a fisheye moving image obtained by photographing a work area such as an office or a factory with the fisheye camera 200 (step S1).

  Next, the distortion correction element image generation unit 102 generates element images in a plurality of directions by a perspective projection method, and generates a distortion correction element image (step S2).

  Next, the human recognition processing unit 105 performs human recognition processing on the distortion correction element image created in step S2 (step S3).

  Next, the work area human position calculation unit 106 converts the human position coordinates (x, y) on the distortion correction element image obtained in step S3, and obtains the human position (X, Y) in the work area (step). S4).

  Finally, the work area person position measurement result output unit 107 outputs the person position (X, Y) in the work area obtained in step S4 (step S5).

  As described above, according to the present embodiment, it is possible to measure position information of an object in a predetermined region from a moving image captured by a fisheye camera.

  In the present embodiment, the form in which the position information of a person is measured from a fish-eye video obtained by shooting a work area such as an office or a factory has been exemplarily described. However, the present invention is not limited to this. Needless to say, it is applicable.

DESCRIPTION OF SYMBOLS 100 Recognition system 102 Conversion means 105 Recognition means 106 1st position coordinate detection means, 2nd position coordinate detection means 200 Camera 300 Recognition apparatus

JP 2013-009050 A

Claims (9)

Conversion means for converting a fisheye moving image obtained by photographing a predetermined area by a camera having a fisheye lens into a plurality of distortion correction element images;
Recognizing means for performing object recognition processing on the distortion correction element image;
First position coordinate detection means for obtaining position coordinates of the recognized object in the distortion correction element image;
Second position coordinate detection means for obtaining position coordinates in the predetermined area of the recognized object from position coordinates in the distortion correction element image;
A recognition apparatus comprising: The conversion means generates a plurality of element images having different azimuth angles according to perspective projection by inputting a plurality of azimuth angles, elevation angles, and rotation angle parameters.
The recognition apparatus according to claim 1. The recognizing means calculates longitude and latitude in a unit sphere from the coordinates of the distortion correction element image, and converts the unit sphere into an equirectangular fisheye by rotation matrix calculation based on the azimuth angle, elevation angle, and rotation angle of the element image. Find the longitude and latitude coordinates of the image,
The recognition apparatus according to claim 1. The second position coordinate detecting means is provided with four or more markers on the plane of the predetermined area, and the position coordinates of the marker in the predetermined area measured in advance, and the position coordinates of the marker in the distortion correction element image And obtaining a position coordinate conversion formula from the distortion correction element image to the predetermined region,
The recognition apparatus according to claim 1. The recognition means surrounds the result of recognizing the object in a rectangle, and the base center coordinate of the rectangle is the position coordinate of the object in the distortion correction element image.
The recognition apparatus according to claim 1. The converting means provides a region where the plurality of distortion correction element images partially overlap each other when generating the plurality of distortion correction element images.
The recognition apparatus according to claim 1. A camera with a fisheye lens;
A recognition device according to any one of claims 1 to 6;
A recognition system comprising: A computer that controls the recognition device,
Conversion means for converting a fisheye moving image obtained by photographing a predetermined area by a camera having a fisheye lens into a plurality of distortion correction element images;
Recognizing means for performing object recognition processing on the distortion correction element image;
First position coordinate detection means for obtaining position coordinates of the recognized object in the distortion correction element image;
Second position coordinate detection means for obtaining position coordinates in the predetermined area of the recognized object from position coordinates in the distortion correction element image;
Program to function as. A method for detecting position coordinates of an object recognized by a recognition device,
A conversion step of converting a fish-eye moving image obtained by photographing a predetermined area with a camera equipped with a fish-eye lens into a plurality of distortion correction element images;
A recognition step of performing recognition processing of the object on the distortion correction element image;
A first position coordinate detection step of obtaining a position coordinate of the recognized object in the distortion correction element image;
A second position coordinate detection step for obtaining a position coordinate of the recognized object in the predetermined area from a position coordinate in the distortion correction element image;
The position coordinate detection method characterized by including.

Images