JP2020067593A - Imaging apparatus - Google Patents

Abstract

To provide an image that focuses on a height of a subject such as a face of a human from front to back even when the subject is not on the center of the image.SOLUTION: An imaging device includes: image recognition means for detecting a predetermined subject in an image by image recognition; acquisition means for acquiring focusing position information of the predetermined subject until the number of the predetermined subjects detected by the image recognition means meets a predetermined condition; and tilt angle control means for controlling a relative tilt angle between a lens and an imaging element based on the focusing position information of the predetermined subject acquired by the acquisition means. This can facilitate reliable focusing on the predetermined subject at the same height.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image pickup device using a tilt technique.

  Conventionally, when identifying and tracking a suspicious person or the like using a surveillance camera at night, the brightness of an image is often increased by opening a diaphragm. However, when the aperture is opened, there is a problem that the depth of field becomes shallow and it becomes difficult to focus. On the other hand, Patent Document 1 discloses a tilting technique for changing a relative angle of an image sensor or a lens, particularly a relative tilt angle. By using this tilting technique, it is possible to widen the range of focusing at a predetermined height.

  However, manually adjusting the tilt angle is very troublesome, and there is a demand for easily determining the tilt angle. Furthermore, the tilt angle determination depends on the focus position (the position of the focus lens or the distance to the subject). Therefore, in order to easily determine the tilt angle, it is necessary to search while changing the combination of both parameters of the tilt angle and the focus position, which is very time-consuming. Further, for example, Patent Document 2 discloses a technique for controlling the tilt angle based on the tilt angle obtained by image analysis.

Japanese Patent Laid-Open No. 11-271599 JP, 2017-093904, A

However, in the prior art disclosed in the above-mentioned patent document, when determining the focus position, it is necessary to perform a complicated work for adjusting in advance so that the subject is imaged around the optical axis at the center of the image. There was a problem.
Therefore, it is an object of the present invention to provide an imaging device in which it is easy to focus on the height of a face or the like from the near side to the back even when a person's face or the like does not exist in the center of an image or the like.

In order to achieve the above object, the present invention provides an imaging device,
Image recognition means for detecting a predetermined subject in the image by image recognition,
An acquisition unit that acquires focus position information of the predetermined subject until the number of the predetermined subjects detected by the image recognition unit satisfies a predetermined condition,
A tilt angle control means for controlling a relative tilt angle between the lens and the image pickup device based on the focus position information of the predetermined subject acquired by the acquisition means.

  According to the present invention, it is possible to provide an imaging device in which it is easy to easily focus on the height of a face or the like from the front side to the back side even when the face of the subject does not exist in the center of the image or the like.

The system block diagram of an Example. The hardware block diagram of the surveillance camera of an Example. The software block diagram of the surveillance camera of an Example. The figure explaining the principle of the tilt control of an Example. The figure explaining the tilt control of an Example. The figure explaining the certainty of the face detection of an Example. The flowchart of the focus search of an Example. The detailed flowchart of the focus search of an Example. The flowchart of the tilt control of an Example. The figure which shows the example of UI of an Example. 6 is a flowchart for displaying the detection positions in an overlapping manner according to the embodiment.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. FIG. 1 is a system configuration diagram of an embodiment of the present invention. The surveillance camera 100 and the client device 200 are connected to each other via a network 300.
The surveillance camera 100 is provided with a camera, and delivers the captured image via the network 300.

  The client device 200 accesses the surveillance camera 100 and acquires an image. Then, it is possible to change the setting of the surveillance camera 100 and control the focus, the tilt angle, and the like.

  For the sake of simplification of explanation, the number of surveillance cameras is one, but there may be two or more. In addition to the client device 200, there may be another client device that accesses the surveillance camera 100 to receive and store images.

The network 300 is composed of a plurality of routers, switches, cables, etc. that comply with communication standards such as Ethernet (registered trademark). In the present invention, a LAN (Local Area Network) or the like may be used regardless of the communication standard, scale, and configuration as long as communication between each surveillance camera and client is possible.
FIG. 2 is a block diagram of the surveillance camera 100. A CPU 1101 that functions as a computer, a primary storage device 1102, a secondary storage device 1103, an image capture I / F (Interface, hereinafter abbreviated as I / F) 1105, a camera control I / F 1108, an image detection I / F 1109, a network. The I / F 1107 is connected to each other via the internal bus 1104.

The primary storage device 1102 is a writable high-speed storage device represented by a RAM, and is loaded with an OS, various programs and various data, and is also used as a work area for the OS and various computer programs.
The secondary storage device 1103 is a non-volatile storage device represented by a flash memory, an HDD, an SD card, etc., and is used as a permanent storage area for an OS, various computer programs, and various data, as well as a short-term storage device. It is also used as a storage area for various data. Details of various computer programs stored in the primary storage device 1102 and the secondary storage device 1103 of the surveillance camera 100 will be described later.
The image capture I / F 1105 is connected to a camera unit 1106 including an image sensor such as CCD or CMOS and a lens, and image data acquired from the camera unit 1106 is converted / compressed into a predetermined format and stored in a primary storage device 1102. Transferred.

  The camera unit 1106 is also connected to the camera control I / F 1108, various camera control signals are sent from the camera control I / F 1108 to the camera unit 1106, and the results are returned to the camera control I / F 1108. The camera unit 1106 is provided with a tilt control mechanism for tilting the image sensor or the lens in the tilt direction or the like with respect to the optical axis. In order to measure the distance to the subject, a TOF (Time Of Flight) type distance measurement is performed in which a laser is projected onto the subject, the reflected light is received by a light receiving sensor, and the distance to the subject is measured based on the round-trip time difference. It may have a mechanism. Alternatively, instead of the TOF method, a pattern irradiation method may be adopted in which a laser having a predetermined pattern is irradiated and the distance is measured based on the distortion of the pattern of the reflected light.

The image detection I / F 1109 performs object detection based on the image generated by the image capture I / F 1105. As a method of image detection, in addition to the object detection described in this embodiment, a predetermined subject can be detected by image recognition, and the predetermined subject may be, for example, face detection, human body detection, or license plate detection.
The network I / F 1107 is an I / F for connecting to the above-described network 300, and is responsible for communication with the client device 200 and the like via a communication medium such as Ethernet.

FIG. 3 is a software configuration diagram of the surveillance camera 100. An OS 1000, a protocol processing unit 1001, a camera control unit 1002, an image generation unit 1003, and an image detection unit 1004 are loaded on the primary storage device 1102.
The OS 1000 is a basic program that controls the entire surveillance camera. Typical functions of the OS 1000 include a secondary storage device 1103 management function such as a file system, a memory management function such as virtual storage, a process management function such as multitasking, and a network I / F 1107 management function.

  The protocol processing unit 1001 executes distribution of image data generated by the image generation unit 1003, control information included in the request, information acquisition, and the like in response to a request from the client device 200. The protocol processing unit 1001 relays the request to the image generation unit 1003, the camera control unit 1002, and the image detection unit 1004, and transmits a corresponding response to the client device 200 based on the result from the relay destination. The protocol processing unit 1001 communicates with an external device via the network I / F 1107 and the communication medium 300. The control information includes a focus search command and relays it to the camera control unit 1002.

  The camera control unit 1002 controls the camera function based on the control information received from the protocol processing unit 1001. Typical examples include control of the camera unit 1106 and notification of control results. The control information includes focus control that determines the position of the lens and tilt control that determines the tilt angle. When the camera control unit receives the focus search command, the camera control unit transmits the focus search command to the image detection unit 1004 and starts the focus operation accordingly. The focus search result is received from the image detection unit 1004, and the focus is moved to that position. The camera control unit 1002 also performs distance measurement for measuring the distance to the subject. The distance measurement may be controlled by the TOF method of the camera unit 1106 or by the pattern irradiation method. Even if the camera unit 1106 does not have these mechanisms, the distance may be measured by moving the lens focus position (corresponding to the lens position in focus = the distance to the subject) using the contrast method. good. Alternatively, the distance may be measured using an image plane phase difference method in which pixels for distance measurement are arranged in the pixel array of the image sensor to perform autofocus. Other controls may include zoom control for changing the zoom magnification of the camera.

  The image generation unit 1003 generates the image data generated by the camera unit 1106 via the image capture I / F 1105. Then, the generated image data is encoded. The encoding method may be jpeg, or an encoding method for image compression such as H264 / AVC or H265 / HEVC can be applied.

  The image detection unit 1004 controls the image detection I / F 1109 based on the video generated by the image generation unit 1003 to detect an object in the image. Various parameters used for detection are relayed from the client device 200 to the protocol processing unit 1001 and received. The parameters used for detection include a threshold value for object detection and the like. The image detection unit 1004 notifies the client device 200 of the detection result via the protocol processing unit 1001. As an image detection algorithm, an existing algorithm such as subject contour extraction or background difference is used. Further, the image detection unit 1004 receives the focus search command received from the camera control unit 1002, and searches for a focus position with a large number of image detection results while using image detection according to the focus search command.

The focus control and tilt control of the camera will be described below.
FIG. 4 is a diagram for explaining the principle of tilt control. Reference numeral 100 denotes a surveillance camera, and this surveillance camera 100 shows a range of a vertical field angle 1306. The optical axis of the lens of the surveillance camera 100 is 1303. Reference numeral 1304 denotes a focus surface (focus position) of focus. The focus plane 1304 is perpendicular to the optical axis 1303. FIG. 4 shows an example in which the focus position is on the ground 1305. Let D be the subject distance, f be the focal length, and a be the installation angle of the surveillance camera. Reference numeral 1302 denotes the angle of the sensor surface when the sensor surface (light receiving surface) of the image pickup element is not covered (not tilted) with respect to the optical axis 1303 of the lens. A sensor surface 1201 is obtained by inclining the sensor surface 1302 with respect to the optical axis 1303 of the lens by a tilt angle b. Then, the focus surface 1304 is inclined to 1308. The tilt angle b at this time is expressed by the following equation (1) according to the Scheimpflug principle.
b = arctan (f / (D · tan a)) ... (1)
If this equation is transformed into the equation of the installation angle a, it can be expressed as the equation (2).
a = arctan (f / (D · tan b)) (2)

When the focus is made parallel to the ground as shown in FIG. 4, the lens position in the contrast method and the subject distance D can be acquired from the camera unit 1106 by the image plane phase difference method. Since f is also the focal length, it can be acquired from the camera unit 1106. Furthermore, since the tilt angle b can also be acquired from the camera unit 1106, all the parameters on the right side of Expression (2) can be obtained, and the installation angle a can be acquired. The installation angle a means an installation angle for making the focus position parallel to the ground.
Further, the installation angle a may be obtained by utilizing the angle of view information and the position between two points.

  If the installation angle a can be obtained, the object distance D and the focal length f can be acquired even when the focus position is moved upward from the ground, and thus all the right sides of the equation (1) are known. The tilt angle b can be obtained. That is, even if the focus position is shifted upward from the ground, for example, the tilt angle b can be easily corrected. That is, it means that at any arbitrary focus position, a tilt angle b that can focus on the same height position can be obtained.

  In this embodiment, for simplification of description, an example in which the image pickup device is vertically rotated (tilt-rotated) about the center line in the long side direction of the image pickup device as an axis is described. However, the present invention can also be applied to two-axis tilt control in which the image sensor is placed on the left and right (pan rotation) about the center line in the short side direction of the image sensor. Further, in the present embodiment, the angle of the sensor surface of the image sensor is changed with respect to the optical axis of the lens, but conversely, the optical axis of the lens may be changed with respect to the sensor surface to perform tilt control. Needless to say.

FIG. 5 is an explanatory diagram of the tilt control of the embodiment. The focus surface is moved upward from the ground from the state of FIG. 4, and the focus surface is moved to the height position of 1402. The focus surface 1402 is the height of the faces of the people 1400 and 1401. Even with the focus surface of 1402, the installation angle a of the camera does not change, and as described above, the focal length is f and the subject distance is D2, which can be sequentially acquired. Regarding the tilt angle b, since the focus surface moves to 1402, the tilt angle changes from b in FIG. 4 to b2 in FIG. 5 according to the Scheimpflug principle. b2 can be calculated by the equation (1).
In this way, in the present embodiment, the focus position of 1402 is searched by image detection with the installation angle a once determined. Although the ground is taken as an example of the surface having the same height in FIG. 5, the ceiling, floor, desk, etc. may be used as long as the surfaces have the same height regardless of the type. .

  FIG. 6 is a diagram for explaining the certainty of face detection. In the surveillance camera of this embodiment, for example, when detecting a suspicious person's face, it is more problematic that a face is not detected (missing detection) than an erroneous detection of a non-face (erroneous detection). Therefore, even an object that is not a target object can be detected. When it is detected as a face, a face frame or the like is superimposed and displayed on the face image of the person displayed on the display unit of the client device. At this time, the certainty of face detection as shown in FIG. 6 is used as a criterion for recognizing a face and superimposing a face frame. 1500 is an axis that numerically represents the probability of face detection, and the probability increases as it goes to the left. Reference numerals 1501, 1502, 1503 and 1504 are examples of human face models. As you move to the left, you can see that the face is distinct. Reference numeral 1505 represents the range of face detection under normal conditions, and the probability is 60% or more. If the certainty is 60% or more, a face frame or the like is superimposed on the face image and displayed.

  On the other hand, when the focus position is searched for and obtained in order to calculate the tilt angle in the present embodiment, the focus position is searched by detecting the face in focus. Therefore, the likelihood of the face having a threshold value higher than the criterion of the likelihood of the face at the time of displaying the face frame is used as the criterion. That is, in the present embodiment, an example of the face detection range in the case of performing the focus search while performing the face detection is set to 1506 (80% or more in certainty). If the range of 1505, which is the reference for face detection in normal times, is used, even if the subject is out of the depth of field and the subject is out of focus, it is detected as a face. Therefore, in the present embodiment, when searching the focus position to calculate the angle, the face detection threshold value is changed to a value higher than that in the normal time. As a result, it is possible to control the corner with high accuracy.

  Next, a method for searching for and acquiring the focus position will be described with reference to FIG. FIG. 7 is an overall flowchart when searching for a focus position. S1700 starts when a focus search command is received from the camera control unit 1701, when the number of objects detected by the image detection unit 1004 has changed by a predetermined number or more, or when a certain period of time has passed periodically. To do. In step S1701, it is determined whether a focus search command has been received. If the focus search command is received (YES), the process branches to S1702 to increase the threshold value for image detection (threshold value for face detection probability in FIG. 6). If not received (NO), the processing ends in S1706.

In S1702, the threshold value for image detection is increased as described in the face detection probability in FIG. Then, in step S1703, a detailed flow of focus position search described later with reference to FIG. 8 is executed. Next, in step S1704, the face detection threshold value is returned to the original value, and in step S1705, the focus position information (information regarding the in-focus lens position or the distance information to the subject) obtained in step S1703 is sent to the camera controller 1002. Then, the camera control unit 1002 moves to the focus position. Finally, the processing ends in step S1706.
It should be noted that in the above embodiment, the face detection threshold value is raised during the face search, but the search may be performed by expanding the aperture of the camera parameter to make the depth of field shallow.

  Next, the method of searching and acquiring the focus position in S1703 of FIG. 7 will be specifically described with reference to FIG. FIG. 8 is a detailed flowchart when the focus position is searched by detecting an image. S1600 starts in S1703 of FIG. The current position is substituted for the detected position in S1601, and 0 is substituted for the "maximum detected number" in S1602 for initialization. Object detection is performed in S1603, and it is determined whether an object is detected in S1604. If detected (YES) in S1604, the process branches to S1605 to acquire the number of detections. If an object is not detected in S1604 (NO), the process branches to S1609, and it is determined whether the search end condition is reached.

  If the number of detections is larger than the “maximum number of detections” in S1606 (YES), the process branches to S1607. If not (NO), the process branches to S1609. Since the "maximum number of detections" is set to 0 in S1602, if even one is detected, the process proceeds to S1607. In S1607, the obtained number of detections is substituted for the "maximum number of detections", and the "maximum number of detections" is updated. Next, in step S1608, the detection position is saved, and the focus position at the time of "maximum number of detections" is stored.

  In step S1609, it is determined whether the search termination condition has been reached. The condition for ending the search is to end when the moving end of the focus lens is reached. However, since it takes time to search until the moving end of the focus lens is reached, a large number of object detections should occur at one time in a state where the image sensor is covered. Therefore, search is performed when the maximum number of detections reaches a predetermined value, when the number of detections decreases from the number of detections up to that time, when the number of detections is zero until then, or when a certain time has elapsed. It may be a termination condition. If the end condition is not reached in S1609 (NO), the process branches to S1603 and the object detection is performed again. When the end condition is reached (YES), the process branches to S1610, the search is ended, and the detected focus position is stored in the primary storage device 1102. If there are a plurality of focus positions detected at that time, the median value may be stored as the focus position, for example. Alternatively, the average value may be stored. Then, in S1611, the flow of FIG. 8 ends.

  FIG. 9 is a flowchart of the camera control unit 1002 for executing the tilt control. S1800 starts when a tilt command is received from the protocol control unit 1001 or the image detection unit 1004, or when a certain period of time elapses periodically. In S1801, it is determined whether the installation angle a has been decided. If the installation angle is determined in 1801 (YES), the process branches to S1802, and it is determined whether or not the new focus position has changed from the previous focus position by a predetermined value or more. If the installation angle is not determined in S1801 (NO), the process ends in S1804. If the focus changes by a predetermined value or more in S1802 (YES), the position at the same height cannot be maintained unless the tilt angle is corrected. Therefore, the flow branches to S1803 to control the tilt angle. The tilt angle at this time is the position obtained by substituting the new D into the equation (1). If the focus does not change in S1802 (NO), the process ends in S1804.

Next, a focus control and tilt control user interface (UI) of the surveillance camera will be described.
FIG. 10 is a diagram showing an example of a display screen during the search of the UI of this embodiment. Reference numeral 1900 denotes a screen of the display unit of the client device 200. An image acquisition request is transmitted from the client device 200 to the surveillance camera 100, and the protocol processing unit 1001 distributes the image generated by the image generating unit 1003. Reference numerals 1905, 1903, and 1901 represent people. Reference numeral 1906 is an area showing the current search state. In FIG. 10, “searching” is displayed because it is being searched. During the search, the faces of people 1901 and 1903 are in focus, but the face of 1905 is out of focus. The face detection position detected by the image detection unit 1004 of the surveillance camera 100 is relayed by the protocol control unit 1001 and transmitted to the client device 200, and the detected position is indicated by 1902 and 1904, and a square frame is displayed. In the person 1905, a face cannot be detected, so a square frame is not displayed. When the face detection position is displayed as a frame as described above, the frame is displayed at the face detection position based on the certainty of the range 1506 in FIG. 6 during the search of the focus position as described above. . However, face detection may be performed even when the search for obtaining the focus position is not in progress, and the criterion for face detection when displaying a frame at the face detection position is within the range of 1505 in FIG. . Therefore, when a face position of a suspicious person or the like is displayed as a frame during normal monitoring, the face frame is displayed even if the probability of face detection is about 60%.

  FIG. 11 is a flowchart for superimposing the face detection position on the screen of the display unit of the client device 200. S2000 starts when a response is received from the protocol processing unit 1001. Next, in S2001, it is determined whether or not the detected position is received. When the detection position is received (YES), the process branches to S2002. In step S2002, the face detection position is superimposed as a face frame on the image on the screen of the display unit of the client device 200. When the detection position is not received (NO), the process ends in S2003.

  Note that the determination of the completion of the search in S1609 may be performed by displaying the detection frames on the faces of all the persons 1905, 1903, and 1901 in FIG. 10, for example. Further, when it is determined that the search is completed in S1609, the expression such as "search completed" may be displayed in the area representing the search state of 1906 in FIG. When the search state is displayed as in 1906, it can be determined at a glance whether or not the search is currently in progress, so that a user-friendly UI can be provided.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the gist thereof.
Moreover, you may make it supply a computer program which implement | achieves a part or all of the control in this invention to the function of the Example mentioned above to an imaging device or an information processing apparatus via a network or various storage media. Then, the computer (or the CPU, MPU, or the like) in the imaging device or the information processing device may read out and execute the program. In that case, the program and the storage medium storing the program constitute the present invention.

100: surveillance camera 200: client device 1001: protocol processing unit 1002: camera control unit 1003: image generation unit 1004: image detection unit

Claims (16)

Image recognition means for detecting a predetermined subject in the image by image recognition,
An acquisition unit that acquires focus position information of the predetermined subject until the number of the predetermined subjects detected by the image recognition unit satisfies a predetermined condition,
An image pickup apparatus comprising: a tilt angle control unit that controls a relative tilt angle between a lens and an image sensor based on the focus position information of the predetermined subject acquired by the acquisition unit.   The imaging apparatus according to claim 1, further comprising an installation angle acquisition unit that acquires an installation angle of the camera.   The image pickup apparatus according to claim 2, wherein the installation angle acquisition unit acquires an installation angle when the focus positions are set to be the same for subjects having the same height. .   The image pickup apparatus according to claim 3, wherein the position having the same height is a position having the same height based on at least one of the ground, the ceiling, and the floor.   The image pickup apparatus according to claim 1, wherein the predetermined subject is at least one of a face, a part of a human body, and a license plate.   The acquisition means may be any one of a case where the number of the predetermined subjects detected by the image recognition means reaches a predetermined number, a case where the number of the predetermined subjects decreases, and a case where the number of the predetermined subjects becomes 0. The imaging apparatus according to claim 1, wherein the focus position information of the predetermined subject is acquired until that time.   7. The acquisition unit ends acquisition of the focus position information of the predetermined subject when the moving end of the focus lens is reached or when a predetermined time elapses, according to any one of claims 1 to 6. The imaging device described.   A client device for acquiring and displaying an image from the imaging device according to claim 1, further comprising a user interface for superimposing the position of the predetermined subject detected by the image recognition means on the image. Client device.   The image recognition reference used when the position of the predetermined subject is detected by the image recognition unit is different depending on whether or not the focus position information of the predetermined subject is acquired by the acquisition unit. The image pickup apparatus according to claim 1.   When the focus position information of the predetermined subject is acquired by the acquisition unit, the criterion of image recognition when detecting the position of the predetermined subject by the image recognition unit is stricter than when the focus position information is not acquired. The imaging device according to claim 9.   The imaging apparatus according to claim 1, wherein the aperture of the lens is changed depending on whether the acquisition unit acquires the focus position information of the predetermined subject or not.   10. The image pickup apparatus according to claim 9, wherein the aperture is opened when the focus position information of the predetermined subject is acquired by the acquisition unit more than when the focus position information is not acquired.   The image pickup apparatus according to claim 2, wherein the tilt angle control means controls the tilt angle based on the installation angle. An image recognition step of detecting a predetermined subject in the image by image recognition,
An acquisition step of acquiring focus position information of the predetermined subject until the number of the predetermined subjects detected by the image recognition step satisfies a predetermined condition;
A tilt angle control step of controlling a relative tilt angle between a lens and an image sensor based on the focus position information of the predetermined subject acquired in the acquisition step.   A computer program for causing a computer to function as each unit of the image pickup apparatus according to claim 1. A computer-readable storage medium that stores the computer program according to claim 15.

Images