WO2015184978A1

WO2015184978A1 - Camera control method and device, and camera

Info

Publication number: WO2015184978A1
Application number: PCT/CN2015/080612
Authority: WO
Inventors: 刘源
Original assignee: 华为技术有限公司
Priority date: 2014-06-04
Filing date: 2015-06-02
Publication date: 2015-12-10
Also published as: CN104023177A

Abstract

Disclosed are a camera control method and device, and a camera. The control method comprises: firstly, determining a pixel point coordinate of a target object in a scenario image and a pixel point coordinate of the target object in a depth image within each detection period; then, based on the pixel point coordinate of the target object in the depth image, acquiring the distance between the target object and a camera; and adjusting the focal length of the camera using a first distance obtained within the first detection period and a second distance obtained within the second detection period. The camera control method is applied to the camera, and an image sensor for generating a scenario image and a depth sensor for generating a depth image of a corresponding scenario are arranged in the camera. By means of the method, the distance between the target object and the camera within each detection period can be acquired, and it is achieved to adjust the focal length of the camera through the distance, so that the algorithm is simple, and the zooming accuracy is improved.

Description

Camera control method, device and camera

The present application claims priority to Chinese Patent Application No. 201410244369.7, entitled "Camera Control Method, Apparatus, and Camera", which is incorporated herein by reference. .

Technical field

The present invention relates to the field of information processing technologies, and in particular, to a camera control method, apparatus, and video camera.

Background technique

The camera used for video conferencing is the basic equipment of the video conferencing system. The video conferencing system is usually applied to more than two locations. These cameras are respectively placed at various locations of the system and are connected to each other through a communication network for photographing local users. Video and transmit the video to other cameras so that users in different locations can communicate with each other via video.

When performing a video conference, it is often necessary to adjust the focal length of the camera to achieve zooming of the camera. For example, if the speaker is far away from the camera, the speaker's picture is smaller in the video captured by the camera, so that the user in other places cannot see the expression of the speaker when the speaker speaks through the video, and the communication experience is poor. It is necessary to adjust the focal length of the camera so that the size of the speaker in the video is kept within a certain range. In the prior art, a method of manual zooming is provided, and the user adjusts the focal length of the camera by manually adjusting the remote controller of the camera, the zoom adjustment interface, or the zoom ring. However, this method is cumbersome to operate, and manual adjustment takes a long time, and the focus adjustment is poor in real-time. In order to solve this problem, the prior art also provides a method for automatically zooming, in which the camera detects the area of the image to be captured in the video in real time, and then adjusts the focal length of the camera according to the area, so that the camera needs to be photographed. The image of the target in the video is always the right size.

However, the inventor found in the research process that the automatic zoom method provided by the prior art adopts an algorithm which is complicated and inaccurate in obtaining an area to be photographed in the camera.

Summary of the invention

The embodiment of the invention provides a camera control method, device and camera, which solves the problem that the algorithm of the automatic zoom method provided by the prior art has complicated algorithm and low accuracy.

In order to solve the above technical problem, the embodiment of the present invention discloses the following technical solutions:

According to a first aspect of an embodiment of the present disclosure, a camera control method is provided, the method comprising:

Determining, in each detection period, pixel coordinates of the target object in the scene image, and determining pixel coordinates of the target object in the depth image;

Obtaining a distance between the target object and the camera based on pixel coordinates of the target object in the depth image;

The focal length of the camera is adjusted using a first distance obtained by the first detection period and a second distance obtained by the second detection period.

With reference to the first aspect, in a first possible implementation manner of the first aspect, the step of adjusting a focal length of the camera by using the first distance obtained by the first detection period and the second distance obtained by the second detection period include:

Obtaining the target according to a second distance d ₂ between the target object and the camera obtained in the second detection period, and pixel coordinates (u ₂ , v ₂ ) of the target object in the scene image The world coordinates of the object in the second detection period (x _c2 , y _c2 );

When the focal length adjusted by the camera is set to f ₂ , according to the world coordinate (x _c2 , y _c2 ) of the target object in the second detection period, the second distance d ₂ and the target object in the scene the pixel coordinates (u _{_2,} v ₂₎ in the image, the focal length f ₂ acquires the pixel coordinates (u _{_2,} v ₂₎ a corresponding relation;

Corresponding relationship between the focal point f ₂ and the pixel point coordinates (u ₂ , v ₂ ), pixel coordinates (u ₁ , v ₁ ) of the target object in the scene image in the first detection period, Obtaining the pixel point coordinates (u ₁ , v ₁ ) and pixel point coordinates (u ₂ ) by a first distance d ₁ between the target object and the camera in a detection period and a focal length f ₁ of the camera in the first detection period The difference of v ₂ ) is the value of the focal length f ₂ when the difference is within the preset range, and the focal length of the camera is adjusted to the focal length f ₂ .

With reference to the first aspect, in a second possible implementation manner of the first aspect, the first distance acquired by using the first detection period, and the second distance acquired by the second detection period, adjusting the photo The steps of the camera's focal length include:

After obtaining the first distance d ₁ of the target object and the camera in the first detection period, acquiring a first ratio of the focal length f ₁ of the camera and the first distance d ₁ in the first detection period;

Setting a focal length adjusted by the camera to be f ₂ , and acquiring a second ratio d ₂ of the focal length f ₂ and the second distance d ₂ after acquiring the second distance d ₂ of the target object and the camera in the second detection period;

Obtaining a value of the focal length f ₂ when the difference between the first ratio and the second ratio is within a preset ratio range, and adjusting a focal length of the camera to a focal length f ₂ .

With reference to the first aspect, in a third possible implementation manner of the first aspect, the method further includes:

After obtaining the first distance d ₁ and the second distance d ₂ , calculating a difference between the first distance d ₁ and the second distance d ₂ , and determining that the adjustment is needed when the difference is not within a preset threshold range The focal length of the camera.

With reference to the first aspect, in a fourth possible implementation manner of the first aspect, the obtaining, by the pixel coordinates of the target object in the depth image, the distance between the target object and the camera, includes:

Obtaining a gray level of a pixel point corresponding to a pixel point coordinate of the target object in the depth image;

The distance between the target object and the camera is obtained by the correspondence between the gray level and the distance.

In combination with the first aspect, or in combination with the first possible implementation of the first aspect, or with the second possible implementation of the first aspect, or with the third possible implementation of the first aspect, or with the first aspect In a fourth possible implementation manner of the first aspect, the method further includes:

Setting a focus priority of the target object to a highest priority according to the received setting information;

After adjusting the focal length of the camera, the focus position of the camera is adjusted to focus the camera at the position with the highest focus priority.

With reference to the first aspect, in a sixth possible implementation manner of the first aspect, the determining a pixel point coordinate of the target object in the depth image includes:

Determining, according to a correspondence relationship between pixel coordinates of the target object formed in the scene image and the depth image, and pixel coordinates of the target object in the scene image, determining that the target object is Pixel point coordinates in the depth image.

With reference to the sixth possible implementation manner of the first aspect, in a seventh possible implementation manner of the first aspect, the method further includes: determining, in advance, pixel coordinates of the target object formed in the scene image and the depth image Correspondence relationship

The step of predetermining the correspondence relationship between the pixel coordinates formed by the target object in the scene image and the depth image includes:

Obtaining an imaging relationship expression of the scene image and the depth image, where the imaging relationship expression is:

Where x is a homogeneous representation of the pixel coordinates of the object in the scene image; x' is a homogeneous representation of the pixel coordinates of the object in the depth image; H is the scene image and the depth image Perspective transformation matrix;

Obtaining pixel coordinates of the four object points on the scene image and the depth image respectively, and acquiring the value of H in the imaging relationship expression, thereby acquiring a perspective transformation matrix of the same object in the scene image and the depth image And mapping, by the perspective transformation matrix, a correspondence relationship between pixel coordinates of the target object formed in the scene image and the depth image.

According to a second aspect of the embodiments of the present disclosure, there is provided a camera control apparatus, the apparatus comprising:

a determining module, configured to determine, at each detection period, pixel coordinates of the target object in the scene image, and determine pixel coordinates of the target object in the depth image;

An acquiring module, configured to acquire a distance between the target object and a camera based on pixel coordinates of the target object in the depth image;

And an adjustment module, configured to adjust a focal length of the camera by using a first distance obtained by the first detection period and a second distance obtained by the second detection period.

With reference to the second aspect, in a first possible implementation manner of the second aspect, the adjusting module includes:

a first acquiring unit, configured to: according to a second distance d ₂ between the target object and the camera obtained in the second detection period, and a pixel point coordinate of the target object in the scene image (u ₂ , v ₂ ) acquiring world coordinates (x _c2 , y _c2 ) of the target object in the second detection period;

a second acquiring unit, configured to: when the focal length adjusted by the camera is set to f ₂ , according to the world coordinate (x _c2 , y _c2 ) of the target object in the second detection period, the second distance d ₂ and the pixel coordinates of the target object in the scene image in the (u _{_2,} v _2), and acquires the focal length f ₂ of the pixel coordinates (u _{_2,} v ₂₎ a corresponding relation;

a first adjusting unit, configured to: by a correspondence relationship between the focal length f ₂ and the pixel point coordinates (u ₂ , v ₂ ), a pixel point coordinate of the target object in the scene image in the first detection period (u ₁ , v ₁ ), a first distance d ₁ between the target object and the camera in the first detection period and a focal length f ₁ of the camera in the first detection period, and acquiring the pixel point coordinates (u ₁ , v ₁ ) The value of the focal length f ₂ when the difference between the pixel point coordinates (u ₂ , v ₂ ) is within a preset range, and adjusts the focal length of the camera to the focal length f ₂ .

With reference to the second aspect, in a second possible implementation manner of the second aspect, the adjusting module includes:

a third acquiring unit, configured to acquire a focal length f ₁ and a first distance d _{1 of the} camera in the first detection period after acquiring the first distance d ₁ of the target object and the camera in the first detection period First ratio;

a fourth acquiring unit, configured to set a focal length adjusted by the camera to be f ₂ , and obtain a focal length f ₂ and the second distance after acquiring the second distance d ₂ of the target object and the camera in the second detecting period a second ratio of d ₂ ;

a second adjusting unit, configured to acquire a value of a focal length f ₂ when a difference between the first ratio and the second ratio is within a preset proportional range, and adjust a focal length of the camera to a focal length f ₂ .

In conjunction with the second aspect, in a third possible implementation manner of the second aspect, the camera control apparatus further includes:

A determining module, configured to acquire a first distance d ₁ and the second distance d _2, d ₁ calculates the first distance and the second distance d ₂ difference, and when the difference is not within the preset threshold range , determining that the focal length of the camera needs to be adjusted.

With reference to the second aspect, in a fourth possible implementation manner of the second aspect, the acquiring module includes:

a gray level acquiring unit, configured to acquire a pixel point coordinate pair of the target object in the depth image The gray level of the pixel at which it should be;

The distance obtaining unit is configured to acquire a distance between the target object and the camera by the correspondence between the gray level and the distance.

With reference to the second aspect, or in combination with the first possible implementation of the second aspect, or the second possible implementation of the second aspect, or the third possible implementation of the second aspect, or the second aspect In a fourth possible implementation manner of the second aspect, the camera control apparatus further includes:

a setting module, configured to set a focus priority of the target object to a highest priority according to the received setting information;

a focusing module for adjusting a focus position of the camera after adjusting a focal length of the camera to focus the camera at a position with the highest focus priority.

With reference to the second aspect, in a sixth possible implementation manner of the second aspect, the determining module includes:

a determining unit, configured to determine, according to a correspondence relationship between pixel coordinates of the target object formed in the scene image and the depth image, and pixel coordinates of the target object in the scene image, determining the target object in the depth image The coordinates of the pixel points in .

With reference to the sixth possible implementation of the second aspect, in a seventh possible implementation manner of the second aspect, the camera control apparatus further includes: a correspondence determining module, where the corresponding relationship determining module is configured to predetermine the Corresponding relationship between pixel coordinates of the target object formed in the scene image and the depth image;

The correspondence determining module includes:

An imaging relationship expression obtaining unit, configured to acquire an imaging relationship expression of the scene image and the depth image, where the imaging relationship expression is:

a perspective transformation matrix acquiring unit, configured to acquire pixel point coordinates of the four object points on the scene image and the depth image, respectively, and obtain a value of H in the imaging relationship expression, thereby acquiring the same object in the scene A perspective transformation matrix of the image and the depth image, by which the correspondence relationship of the pixel coordinates formed by the target object in the scene image and the depth image is characterized.

According to a third aspect of an embodiment of the present disclosure, a camera is provided, the camera comprising: a processor, a memory, an image sensor, and a depth sensor,

Wherein the image sensor is configured to generate a scene image including a target object;

The depth image sensor is configured to generate a depth image including a target object;

The memory is configured to store a program for controlling a camera;

The processor is configured to read a program stored in the memory, and perform a camera-controlled operation according to the program, where the camera controlled operation comprises:

With reference to the third aspect, in a first possible implementation manner of the third aspect, the image sensor and the depth sensor are integrated in the same sensor;

Alternatively, the image sensor and the depth sensor are both disposed behind the camera lens, the image sensor and the depth sensor are disposed at different horizontal levels, and a half-reverse half is disposed between the camera lens and the image sensor and the depth sensor. lens;

Alternatively, the image sensor is disposed behind the camera lens, and the image sensor is arranged side by side with the depth sensor.

The present application discloses a camera control method and apparatus, and a corresponding camera. In the camera control method, first, at each detection period, determining pixel coordinates of a target object in a scene image, and determining that the target object is at depth Pixel point coordinates in the image; then, based on the pixel point coordinates of the target object in the depth image, acquire the distance between the target object and the camera And reusing the first distance obtained in the first detection period and the second distance obtained in the second detection period to adjust the focal length of the camera.

The camera control method can adjust the focal length of the camera by the distance between the target object and the camera acquired in different detection periods. The change in the distance between the target object and the camera can reflect the size of the area of the target object in the video. Compared with the prior art, the method disclosed in the present application adopts a simple algorithm based on the distance between the target object and the camera to achieve the adjustment of the focus distance, is easy to implement, and has higher accuracy and robustness. .

DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the description of the prior art will be briefly described below. Obviously, the drawings in the following description are only It is a certain embodiment of the present invention, and other drawings can be obtained from those skilled in the art without any inventive labor.

1 is a flow chart of an embodiment of a camera control method according to the present disclosure;

2 is a flow chart of another embodiment of a camera control method according to the present disclosure;

3 is a diagram showing an example of a camera imaging model disclosed in the present invention;

4 is a schematic diagram of a geometric relationship of imaging of a camera disclosed in the present invention;

FIG. 5 is a flowchart of still another embodiment of a camera control method according to the present disclosure;

6 is a flowchart of still another embodiment of a camera control method according to the present disclosure;

7 is a schematic diagram showing the working principle of a depth sensor disclosed in the prior art;

FIG. 8 is a schematic structural diagram of a camera disclosed in the present application; FIG.

FIG. 9 is a schematic structural diagram of still another camera disclosed in the present application; FIG.

FIG. 10 is a schematic structural diagram of a camera control apparatus according to the present disclosure; FIG.

11 is a schematic structural diagram of still another camera control device according to the present disclosure;

12 is a schematic structural diagram of still another camera control device according to the present disclosure;

FIG. 13 is a schematic structural diagram of still another camera control apparatus according to the present invention.

detailed description

The embodiment of the present application provides a camera control method, device, and camera to solve the problem that the target detection algorithm is complicated and the accuracy is low when the camera performs automatic zooming by using the prior art.

The above-mentioned objects, features, and advantages of the embodiments of the present invention will become more apparent and understood. Give further details.

FIG. 1 is a schematic flowchart diagram of a camera control method according to an embodiment of the present application. Referring to FIG. 1, the camera control method includes:

Step 101: Determine, at each detection period, pixel coordinates of the target object in the scene image, and determine pixel coordinates of the target object in the depth image.

The camera control method disclosed in the present application is applied to a camera, and an image sensor and a depth sensor are disposed in the camera. The image sensor usually uses a CCD (Charge-coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) color image sensor, and the color image sensor generates RGB (Red/Green/ A color image in the Blue, Red/Green/Blue format, using the RGB image as a scene image. In addition, a black and white image sensor can also be used, and the generated black and white image is used as a scene image. The depth sensor is configured to generate a depth image, and the gray level of the pixel point in the depth image can represent a distance of the object corresponding to the pixel from the camera.

The target object is a whole or part of a shooting scene, and is a part of the scene that is of interest to the user in the shooting scene. When determining the pixel point coordinates of the target object in the scene image, the attribute characteristics of the target object can be utilized. The attribute characteristics include lines, shapes, areas, and the like. Through the target detection algorithm and attribute features, the pixel points of the target object in the scene image can be obtained. Further, according to the correspondence between the pixel points in the scene image and the pixel points in the depth image, the pixel point coordinates of the target object in the depth image may be determined.

Step 102: Acquire a distance between the target object and a camera based on pixel coordinates of the target object in the depth image.

Step 103: Adjust a focal length of the camera by using a first distance obtained by the first detection period and a second distance obtained by the second detection period.

The two detection periods are respectively set as a first detection period and a second detection period, and the distances acquired in the two detection periods are the first distance d ₁ and the second distance d _{2 , respectively} . The first distance d ₁ and the second distance d ₂ can reflect the change of the distance between the target object and the camera in the two detection periods, thereby reflecting the size of the area of the target object presented in the video. For example, when the second distance d ₂ is greater than the first distance d _1, the target object is present in the video area is reduced, it is necessary to adjust the focal length of the camera is large; when the second distance d ₂ is less than the first distance ₁ d, target The area of the object presented in the video increases, and the focal length of the camera needs to be reduced.

It can be seen from the above embodiment that in the camera control method disclosed in this embodiment, first, the pixel point coordinates of the target object in the scene image and the pixel point coordinates of the target object in the depth image are determined in each detection period; Obtaining a pixel point coordinate of the target object in the depth image, acquiring a distance between the target object and the camera; and using a first distance obtained by the first detection period, and a second distance obtained by the second detection period, adjusting The focal length of the camera.

The camera control method is applied to a camera, and an image sensor for generating a scene image and a depth sensor for generating a depth image are disposed in the camera. By this method, the distance between the target object and the camera in each detection cycle can be acquired, and the change of the distance between the target object and the camera can reflect the size of the area of the target object in the video. Compared with the prior art, the method disclosed in the present application adopts a simple algorithm based on the distance between the target object and the camera to achieve the adjustment of the focus distance, is easy to implement, and has higher accuracy and robustness. .

In step 103, a first distance obtained by using the first detection period and a second distance obtained by the second detection period are disclosed, and a scheme for adjusting a focal length of the camera is disclosed. Referring to FIG. 2, the solution may be implemented in various manners. For the implementation thereof, refer to the following embodiments.

Step 1011: Determine, at each detection period, pixel coordinates of the target object in the scene image, and determine pixel coordinates of the target object in the depth image.

Step 1012: Acquire a distance between the target object and a camera based on pixel coordinates of the target object in the depth image.

The implementation process of the steps 1011 to 1012 is the same as the implementation process of the steps 101 to 102, and can be referred to each other, and details are not described herein again.

Step 1013: Set two detection periods as a first detection period and a second detection period, respectively. Obtaining the target according to a second distance d ₂ between the target object and the camera obtained in the second detection period, and pixel coordinates (u ₂ , v ₂ ) of the target object in the scene image The world coordinates (x _c2 , y _c2 ) of the object in the second detection period.

Since the camera can be placed anywhere in the environment, choose a benchmark in the environment. The coordinate system describes the position of the camera and uses it to describe the position of other objects in the environment, which is called the world coordinate system.

The coordinates of the world coordinate system are difficult to measure, and the coordinates of the pixel points can be obtained by the image. Therefore, in this embodiment, after the coordinates of the pixel points are acquired, the world coordinates of the target object are determined by the coordinates of the pixel points.

Step 1014: When the focal length adjusted by the camera is set to f ₂ , according to the world coordinate (x _c2 , y _c2 ) of the target object in the second detection period, the second distance d ₂ and the target A pixel point coordinate (u ₂ , v ₂ ) of the object in the scene image acquires a correspondence relationship between the focal length f ₂ and the pixel point coordinates (u ₂ , v ₂ ).

Step 1015: The correspondence between the focal point f ₂ and the pixel point coordinates (u ₂ , v ₂ ), the pixel point coordinates of the target object in the scene image in the first detection period (u ₁ , v ₁ a first distance d ₁ between the target object and the camera in the first detection period and a focal length f ₁ of the camera in the first detection period, acquiring the pixel point coordinates (u ₁ , v ₁ ) and pixel coordinates The difference of (u ₂ , v ₂ ) is the value of the focal length f ₂ when the difference is within the preset range, and the focal length of the camera is adjusted to the focal length f ₂ .

Through the operations of steps 1013 to 1015, the adjustment of the focal length of the camera is achieved.

When the world coordinate system uses the optical center of the camera as the origin of the world coordinate system, the camera imaging model is shown in Figure 3, where x _c , y _c , z _c are world coordinate systems, and x, y are scene images or depth images. The imaging plane coordinate system, m, n is the pixel coordinate system of the scene image or the depth image, O _c is the optical center of the camera, O _c z _c is the optical axis of the camera, and O _c p is the focal length f.

The x _c , y _c , and z _c coordinate axes of the world coordinate system respectively represent the coordinates of the three-dimensional space in which the object is located, in meters or millimeters; the imaging plane coordinate system established on the scene image represents the object on the image sensor. The coordinate system in which the imaging plane is located, the imaging plane coordinate system established on the depth image, the coordinate system representing the imaging plane of the object on the depth sensor, and the x, y coordinate axes of the imaging coordinate system, representing the horizontal and vertical coordinate axes, respectively. The unit is m or mm; the pixel coordinate system established on the scene image is the coordinate system of the object after imaging by the image sensor, and the pixel coordinate system established on the depth image, after the object is imaged by the depth sensor, in the pixel The coordinate system at the position, the m, n coordinate axes of the pixel coordinate system, representing the horizontal and vertical coordinate axes, respectively, in pixels.

In addition, between the imaging plane coordinate system and the pixel coordinate system of the same scene image, or between the imaging plane coordinate system and the pixel coordinate system of the same depth image, the conversion may be performed by a certain ratio.

Referring to the geometric relationship diagram of the camera imaging shown in FIG. 4, the world coordinate system and imaging are known. The correspondence between the plane coordinate systems is:

Where f _x and f _y are the equivalent focal lengths of the focal length f of the camera on x and y, respectively; x _c , y _c , z _c are the coordinate axes of the world coordinate system; x, y are the coordinate axes of the imaging coordinate system.

According to formula (1) and formula (2), the transformation relationship between the world coordinate system of a certain object point and the imaging plane coordinate system in three-dimensional space can be obtained as follows:

among them,

Homogeneous representation of plane coordinates;

A homogeneous representation of the world coordinate system. f _x and f _y are the equivalent focal lengths on x and y, respectively; s is the distortion coefficient of the image; u ₀ , v ₀ are the coordinates of the main point of the image. R is the rotation matrix of the camera, and t is the camera translation vector. Where K is called the internal reference of the camera, and R and t are called the external parameters of the camera.

According to formula (3) and formula (4), it can be seen that at a certain moment, the relationship between the pixel point coordinates of a certain object point and the world coordinates in the scene image is:

Where (u, v) is the pixel point coordinate of the object point in the scene image, (u ₀ , v ₀ ) is the image principal point coordinate, and the image main point is the intersection of the camera optical axis and the imaging plane of the scene image, f is the focal length, β is the scale factor between the image coordinate system and the pixel coordinate system in the scene image, and d is the distance from the object point to the image sensor. Since the image sensor is disposed in the camera, it can be considered as d The distance from the object to the camera, which distance can be acquired by the depth image. Generally, the gray level of each pixel in the depth image represents the quantization of the distance between the object corresponding to the pixel and the depth sensor, and the different gray levels respectively represent different distances between the object and the depth sensor. Through the depth image generated by the depth sensor, the quantization level of the gray level, and the quantization algorithm, the actual depth value represented by each gray level can be obtained, and the distance between the object corresponding to each pixel point and the camera can be calculated.

It can be seen from the above that when the world coordinate system takes the optical center of the camera as the origin of the world coordinate system, the first distance d ₁ acquired according to the first detection period and the pixel point coordinates of the target object in the scene image (u ₁ , v ₁ ), the algorithm for acquiring the world coordinates (x _c1 , y _c1 ) of the target object in the first detection period is:

Wherein (u ₁ , v ₁ ) is a pixel point coordinate of the target object in the scene image of the first detection period; (u ₀ , v ₀ ) is a pixel point coordinate of the image main point in the scene image; β is the a scale factor when converting between an image coordinate system and a pixel coordinate system corresponding to the scene image; f ₁ is a current focal length of the camera; d ₁ is a first distance between the target object and the camera; _C1 , y _c1 ) are the world coordinates of the target object in the first detection period. Wherein d ₁ can be acquired by the depth image in the first detection period.

The pixel point coordinates (u ₁ , v ₁ ) of the target object in the scene image of the first detection period may be obtained by querying the scene image based on the characteristics of the target object. Wherein, the feature of the target object may include a line, a shape, and the like of the target object.

Among them, since the focal length of f ₁ is much smaller than d ₁ , it can be ignored and set.

b=βu ₀ , then equation (7) and formula (8) can be simplified as:

Correspondingly, according to the above statement, in step 1013, according to the second distance d ₂ between the target object and the camera in the second detection period, and the pixel point coordinates of the target object in the scene image (u ₂ , v ₂ ), when acquiring the world coordinates (x _c2 , y _c2 ) of the target object in the second detection period, the formula is:

Wherein (u ₂ , v ₂ ) is a pixel point coordinate of the target object in the scene image of the second detection period; (u ₀ , v ₀ ) is a pixel point coordinate of the image main point in the scene image; β is the a scale factor when converting between the image coordinate system and the pixel coordinate system corresponding to the scene image; f ₁ is the current focal length of the camera; d ₂ is a second distance between the target object and the camera; _C2 , y _c2 ) is the world coordinate of the target object in the second detection period. Where d ₂ can be acquired by the depth image of the second detection period.

[01] Since the focal length of the camera has not been adjusted in the second detection period, and the current focal length of the camera is still f ₁ , the world coordinates of the target object in the second detection period can be acquired. And when

When b=βu ₀ , formula (11) and formula (12) can be simplified as:

In step 1014, the camera adjusted focal length is set to f ₂ , and the corresponding relationship between the focal length f ₂ and the pixel point coordinates (u ₂ , v ₂ ) is obtained. In addition, according to the pixel point coordinates (u ₁ , v ₁ ) of the target object in the scene image obtained in the first detection period, the first distance d ₁ between the target object and the camera, and the first detection The focal length f ₁ of the camera during the cycle can be obtained by equations (9) and (10). In addition, according to the formula (13) and the formula (14), it is understood that the correspondence relationship between the focal length f ₂ and the pixel point coordinates (u ₂ , v ₂ ) is:

among them,

b=βu ₀ .

When the distance between the target object and the camera changes, the size of the image of the target object in the video changes. In order to compensate for it, the focal length of the camera needs to be adjusted so that the target object is imaged in the video. The size is kept within a certain range. In step 1015, the difference between the acquired pixel point coordinates (u ₁ , v ₁ ) and the pixel point coordinates (u ₂ , v ₂ ) is disclosed, and the value of the focal length f ₂ when the difference is within the preset range is obtained. Program. In this scheme, in order to keep the size of the image of the target object in the video in the preset range, it should be made

And

among them

For the preset range, the following formula is satisfied:

Solving the formula (17) and the formula (18), since the values of x _c1 , y _c1 , x _c2 , d ₁ , d ₂ , and f ₁ are all known, the value of the focal length f ₂ can be obtained, and the the focal length of the camera can be adjusted to f _2.

The preset range

The value can be set according to the application requirements. If it is necessary to make the size of the image of the target object in the video substantially unchanged, the preset range can be set.

The value is 0.

In addition, referring to FIG. 5, other methods can be used to adjust the focal length of the camera.

Step 1021: Determine, at each detection period, pixel coordinates of the target object in the scene image, and determine pixel coordinates of the target object in the depth image.

Step 1022: Acquire a distance between the target object and the camera based on pixel coordinates of the target object in the depth image.

The implementation process of the steps 1021 to 1022 is the same as the implementation process of the steps 101 to 102, and can be referred to each other, and details are not described herein again.

Step 1023: After obtaining the first distance d ₁ of the target object and the camera in the first detection period, acquiring a first ratio of the focal length f ₁ of the camera and the first distance d ₁ in the first detection period. .

Step 1024: Set a focal length adjusted by the camera to be f ₂ , and obtain a second distance d ₂ between the target object and the camera after the second detection period, and obtain a focal length f ₂ and a second distance d ₂ Two ratios.

Step 1025: Acquire a value of a focal length f ₂ when a difference between the first ratio and the second ratio is within a preset ratio range, and adjust a focal length of the camera to a focal length f ₂ .

The solution disclosed in steps 1021 to 2025 utilizes the distances acquired in the two detection periods to achieve adjustment of the focal length of the camera. In this embodiment, when the ratio between the focal length of the camera and the distance between the camera and the target object is generally considered to be within a certain range, the screen size of the target object in the video is substantially maintained within a certain range. Therefore, in this embodiment, after obtaining the first ratio of f ₁ and d _{1 and} the second ratio of f ₂ and d ₂ , calculating a difference between the first ratio and the second ratio within a preset ratio range The value of f ₂ is adjusted and the focal length of the camera is adjusted to f ₂ to achieve automatic zooming.

The present application discloses a camera control method for realizing automatic zooming of a camera according to a distance between a target object and a camera. Referring to Fig. 6, in order to avoid excessive zooming and image blurring, the present application discloses the following embodiments.

Step 111: Determine, at each detection period, a pixel point coordinate of the target object in the scene image. And determining pixel coordinates of the target object in the depth image.

Step 112: Acquire a distance between the target object and a camera based on pixel coordinates of the target object in the depth image. The distance includes: a first distance d ₁ obtained by the first detection period and a second distance d ₂ obtained by the second detection period.

Step 113: After obtaining the first distance d ₁ and the second distance d ₂ , calculate a difference between the first distance d ₁ and the second distance d ₂ .

Step 114: Determine whether the difference is within a preset threshold range. If not, perform the operation of step 115. If yes, perform the operation of step 116.

Step 115: When it is determined that the difference is not within a preset threshold range, determine that a focal length of the camera needs to be adjusted, and obtain a first distance obtained by using the first detection period, and a second obtained by using the second detection period. Two distances, adjusting the focal length of the camera.

Step 116: When it is determined that the difference is within a preset threshold range, it is determined that it is not currently necessary to adjust the focal length of the camera.

The implementation process of step 111 to step 112 is the same as the implementation process of step 101 to step 102. In step 115, the first distance and the second distance acquired in the two detection periods are utilized to adjust the implementation process of the focal length of the camera, and The implementation process of step 103 is the same and can be referred to each other, and details are not described herein again.

In the above embodiment, a threshold range is set in advance. The difference between the distances obtained in the two detection periods can reflect the change of the position of the target object. If the position change of the target object is less than the threshold range, the focal length of the camera is temporarily not adjusted. When the threshold is larger than the threshold range Only adjust the focal length of the camera. The threshold range can be set based on subjective image effects and empirical values. The method can avoid image blur due to zooming too much, and the image maintains a certain stability.

The present application discloses the step of acquiring the distance between the target object and the camera according to the pixel point coordinates of the target object in the depth image, and the step includes:

First, acquiring a gray level of a pixel point corresponding to a pixel point coordinate of the target object in the depth image; and then acquiring, by the correspondence between the gray level and the distance, between the target object and the camera distance.

Wherein the depth image is generated by a depth sensor disposed in the camera. Depth sensor It is a device capable of generating a depth image about a scene, the basic principle of which is to emit infrared light to a target object, and detect a time difference of the target object reflecting the infrared light, and determine the distance of the target object by the time difference. The depth sensor is capable of acquiring depth images in real time with strong accuracy and reliability.

See the schematic diagram of the working principle of the depth sensor shown in Figure 7. In the figure, the solid line portion of each video frame is a triangularly-intensity-modulated infrared ray emitted by an infrared light source, and the dotted line portion is an infrared ray reflected by the object. The delay is Δt=2d/v, where d is the distance between the target object and the infrared source, and v is the speed of light. Moreover, the vertical line pair of the dotted line in the figure is the light received by the depth sensor when the shutter of the camera is opened. If the shutter is opened for exposure when the brightness of the infrared light source is increased, the intensity of the infrared light received by the depth sensor will become smaller as the distance of the object increases. In this case, the dotted line moves to the right, when the depth sensor is more The reflected light of the target object is received less, indicating that the distance between the target object and the depth sensor becomes far. Conversely, if the shutter is opened for exposure when the brightness of the infrared light source is decreasing, the intensity of the received infrared light increases as the distance of the target object increases. In this case, the dotted line moves to the right, so When the depth sensor receives more of the reflected light of the target object, it indicates that the distance between the target object and the depth sensor becomes far. By comprehensively analyzing the light intensity obtained during the luminance increasing period and the light intensity decreasing period, the influence of the physical reflection characteristics can be eliminated, and the distance between the target object and the depth sensor can be obtained.

When comprehensively analyzing the light intensity obtained during the luminance increasing period and the light intensity decreasing period, set s(t) to the luminance modulated optical power of the triangular wave generated by the infrared light source, I ₊ (t _s , d) and I _- (t _s , d) is the intensity of the light received by the depth sensor in the luminance increment period and the luminance decrement period, respectively, according to which the following expression can be generated:

Where σ is the area of the backscattering cross section of the target object, T is the length of one luminance modulation period, and t _s is the moment when the shutter is opened. According to the above two formulas, the distance d of the target object to the infrared light source can be obtained as:

According to the above calculation method, the depth sensor can determine the distance of the target object from itself according to the received light intensity of the same target object at different times (such as the brightness increasing period and the brightness decreasing period). And, the depth sensor finally converts the calculated distance information into a grayscale or color depth image, and outputs the depth image. Generally, the gray level of each pixel in the depth image represents the quantization of the distance between the object corresponding to the pixel and the depth sensor, and the different gray levels respectively represent different distances between the object and the depth sensor. The depth image may have a frame rate of 30 fps or 60 fps, typically having 256 gray levels. Through the depth image generated by the depth sensor, the quantization level of the gray level, and the quantization algorithm, the actual depth value represented by each gray level can be obtained, and the distance between the object corresponding to each pixel point and the camera can be calculated. For a depth image in grayscale form, the pixel region with higher brightness indicates that the closer the object corresponding to the pixel region is to the depth sensor, the darker the pixel region indicates that the object corresponding to the pixel region is farther from the depth sensor. Wherein, the pixel area with the gray level of 0 represents the object farthest from the depth sensor, and the pixel area with the gray level of 255 represents the object closest to the depth sensor.

Through the introduction of the working principle of the depth sensor, it can be known that the distance between the target object and the depth sensor can be obtained by the gray level of the corresponding pixel point of the target object in the depth image, and, in the present application, A depth sensor is disposed in the camera, and a distance between the target object and the camera is obtained by a distance between the target object and the depth sensor.

In the camera control method disclosed in the present application, the focal length of the camera is adjusted by the distance between the target object and the camera in different detection periods. In addition, in order to achieve focus on the target object, the camera control method disclosed in the present application further includes:

After adjusting the focal length of the camera, adjusting the focus position of the camera to make the camera The machine focuses at the position with the highest focus priority to focus the camera on the target object.

After the adjustment of the focal length of the camera is completed, the focus position can also be adjusted to adjust the focus position to the target object to obtain a video with higher imaging quality. When adjusting the focus position of the camera, it is necessary to set the focus priority of the target object to the highest priority in advance. When the adjustment of the focus position is required, the scene image is usually partitioned, and then the focus value (FV) of each partition is counted, and the focus priority of each partition is obtained by weighting the focus values of the respective partitions. And focusing the camera at the position with the highest focus priority so that the camera is preferentially focused on the target object.

In addition, in the embodiment of the present application, the step of determining the pixel point coordinates of the target object in the scene image and determining the pixel point coordinates of the target object in the depth image is disclosed in each detection period. The determining the pixel point coordinates of the target object in the depth image includes: a correspondence between pixel coordinates formed in the scene image and the depth image based on the target object, and the target object in the scene image The pixel point coordinates in the determination of the pixel point coordinates of the target object in the depth image.

The camera control method disclosed in the present application is applied to a camera, and an image sensor and a depth sensor are provided in the camera, the image sensor is used to generate a scene image, and the depth sensor is used to generate a depth image. Performing target detection on the scene image, that is, acquiring a pixel point generated by the target object on the scene image, and then correspondingly according to the pixel point coordinates formed by the target object in the scene image and the depth image, and the Determining the pixel point coordinates of the target object in the depth image, and determining the pixel point coordinates of the target object in the depth image, so as to determine the pixel coordinates in the depth image according to the target object in a subsequent step. A pixel point of the target object in the depth image, and acquiring a gray level of the pixel point, by which the distance between the target object and the camera is obtained.

Corresponding relationship between pixel coordinates formed by the target object in the scene image and the depth image is determined by the placement position of the image sensor and the depth sensor, and includes the following cases:

If the depth sensor and the image sensor are integrated on one sensor, for example, a pixel unit capable of sensing depth information is added on the basis of an ordinary image sensor, the sensor can simultaneously output a scene image and a depth image, and output the scene image and depth The image is a completely consistent scene. In this case, the pixel coordinates formed by the same object on the scene image are the same as the pixel coordinates formed on the depth image. And, because the depth image is often not needed compared to the scene image High resolution and frame rate, such as scene images, can be achieved by combining a high resolution and frame rate image sensor with a combination of lower resolution and frame rate depth sensors.

In addition, separate image sensors and depth sensors can also be used. Referring to FIG. 8, wherein the image sensor and the depth sensor are both disposed behind the camera lens, and a semi-reverse half lens is disposed between the image sensor and the camera lens, and between the depth sensor and the camera lens. Under the action of the half lens, part of the incident light passing through the camera lens is reflected to the depth sensor and the other is transmitted to the image sensor for imaging. The half-reflex lens is usually at an angle of 45° with respect to the horizontal direction. In addition, the angle may be other angles, which is not limited in this application. In this scheme, the splitting is realized by using a half-reverse half lens, which can ensure that the scenes captured by the image sensor and the depth sensor are consistent, in this case, the pixel coordinates formed by the same object on the scene image, and the pixel points formed on the depth image. The coordinates are the same. In order to ensure the amount of light entering the image sensor to obtain a better image effect, the ratio of the amount of transmitted and reflected light of the half-reflex lens can be controlled. For example, the transmitted light accounts for 70% of the total amount of light, and the reflected light accounts for 30% of the total light. .

In another case, separate image sensors and depth sensors are employed, and the image sensors and depth sensors are used to image different light rays, for example, see FIG. 9, wherein the image sensor is disposed at The rear of the camera lens is described, and the image sensor is arranged side by side with the depth sensor. Wherein the image sensor is generally at the same level as the camera lens. In this case, since the image sensor and the depth sensor are used for different optical paths, the content of the two images is slightly different, causing parallax between the scene image and the depth image, so that the scene image needs to be calibrated to obtain the target object in the scene. Correspondence between pixel coordinates formed in the image and the depth image. Therefore, the camera control method further includes: predetermining a correspondence relationship between pixel coordinates formed by the target object in the scene image and the depth image.

First, an imaging relationship expression of the scene image and the depth image is obtained, and the imaging relationship expression is:

Where x is a homogeneous representation of the pixel coordinates of the object in the scene image; x' is the A homogeneous representation of the coordinates of the pixel points of the object in the depth image; H is a perspective transformation matrix of the scene image and the depth image.

Then, acquiring pixel point coordinates of the four object points on the scene image and the depth image respectively, thereby acquiring the value of H in the imaging relationship expression, thereby acquiring the perspective of the same object in the scene image and the depth image Transforming a matrix, and characterizing, by the perspective transformation matrix, a correspondence relationship between pixel coordinates of the target object formed in the scene image and the depth image.

The H in the imaging relational expression is usually a 3×3 matrix with a degree of freedom of 8, representing a transformation relationship between the scene image and the depth image, which is called a perspective transformation matrix. It is assumed that the coordinates of the pixel point of an object in the scene image are (x, y), and the coordinates of the pixel point of the object in the depth image are (x', y'), according to which the following two equations can be obtained:

It can be known from the above two equations that it is necessary to establish eight equations through the object points of four known coordinates, and the value of H can be obtained. The four object points can be pre-selected by the user, and the pixel coordinates of each object point on the scene image and the depth image are respectively obtained, and substituted into the above two equations, the value of H can be obtained, thereby obtaining the target. The correspondence between the pixel coordinates of the object formed in the scene image and the depth image.

Correspondingly, the embodiment of the present application also discloses a camera control device. Referring to FIG. 10, the camera control apparatus includes: a determination module 100, an acquisition module 200, and an adjustment module 300.

The determining module 100 is configured to determine, at each detection period, pixel coordinates of the target object in the scene image, and determine pixel coordinates of the target object in the depth image;

The acquiring module 200 is configured to acquire a distance between the target object and the camera based on pixel coordinates of the target object in the depth image;

The adjustment module 300 is configured to adjust a focal length of the camera by using a first distance obtained by the first detection period and a second distance obtained by the second detection period.

Further, referring to FIG. 11 , the adjustment module 300 includes: a first obtaining unit 301, a second acquiring unit 302, and a first adjusting unit 303.

The first obtaining unit 301 is configured to: according to the second distance d ₂ between the target object and the camera obtained in the second detection period, and the pixel point coordinates of the target object in the scene image (u ₂ , v ₂ ), acquiring world coordinates (x _c 2, y _c 2) of the target object in a second detection period;

The second acquiring unit 302, when the focal length for the setting of the camera was adjusted to ₂ F, according to a second detection period of the target object in world coordinates (x _{_c2,} y _c2), the second a distance d ₂ and a pixel point coordinate (u ₂ , v ₂ ) of the target object in the scene image, and obtaining a correspondence relationship between the focal length f ₂ and the pixel point coordinate (u ₂ , v ₂ );

The first adjusting unit 303 is configured to: by a correspondence between a focal length f ₂ and the pixel point coordinates (u ₂ , v ₂ ), a pixel point of the target object in the scene image in a first detection period Coordinates (u ₁ , v ₁ ), a first distance d ₁ between the target object and the camera in the first detection period, and a focal length f ₁ of the camera in the first detection period, acquiring the pixel point coordinates (u ₁ , v ₁ ) The value of the focal length f ₂ when the difference between the pixel point coordinates (u ₂ , v ₂ ) is within a preset range, and adjusts the focal length of the camera to the focal length f ₂ .

In addition, referring to FIG. 12, the adjustment module 300 may also be in other forms, including: a third acquisition unit 304, a fourth acquisition unit 305, and a second adjustment unit 306.

The third obtaining unit 304 is configured to acquire, after acquiring the first distance d ₁ of the target object and the camera in the first detection period, the focal length f ₁ of the camera in the first detection period, and the a first ratio of the first distance d ₁ ;

a fourth acquiring unit 305, configured to set a focal length adjusted by the camera to be f ₂ , and obtain a focal length f ₂ and the second after acquiring a second distance d ₂ between the target object and the camera in the second detecting period a second ratio of distance d ₂ ;

The second adjusting unit 306 is configured to acquire a value of a focal length f ₂ when a difference between the first ratio and the second ratio is within a preset proportional range, and adjust a focal length of the camera to a focal length f ₂ .

Further, the camera control apparatus further comprising: a determining module, the determining module is configured to acquire a first distance and the second distances d ₁ d ₂ After calculating the difference between the first and second distances d ₁ is a distance d ₂ And determining that the focal length of the camera needs to be adjusted when the difference is not within a preset threshold range.

Further, referring to FIG. 13, the obtaining module 200 includes: a gray level acquiring unit 201 and a distance acquiring unit 202.

The gray level obtaining unit 201 is configured to acquire a gray level of a pixel point corresponding to a pixel point coordinate of the target object in the depth image;

The distance obtaining unit 202 is configured to acquire a distance between the target object and the camera by using a correspondence between the gray level and the distance.

Further, the camera control device further includes: a setting module and a focusing module. The setting module is configured to set a focus priority of the target object to a highest priority according to the received setting information; the focusing module is configured to adjust a focus of the camera after adjusting a focal length of the camera The position is such that the camera is focused at the position with the highest focus priority.

Further, the determining module 100 includes: a determining unit, configured to use, according to a correspondence relationship between pixel coordinates formed by the target object in the scene image and the depth image, and the target object in the scene image Pixel point coordinates, determining pixel point coordinates of the target object in the depth image.

Further, the camera control device further includes: a correspondence determining module, wherein the correspondence determining module is configured to predetermine a correspondence relationship between pixel coordinates formed by the target object in the scene image and the depth image.

The correspondence determining module includes: an imaging relationship expression acquiring unit and a perspective transform matrix acquiring unit. The imaging relationship expression obtaining unit is configured to acquire an imaging relationship expression of the scene image and the depth image, where the imaging relationship expression is:

The perspective transformation matrix acquiring unit is configured to acquire pixel point coordinates of the four object points on the scene image and the depth image, and obtain the value of H in the imaging relationship expression, thereby acquiring the same object in the And a perspective transformation matrix of the scene image and the depth image, and the correspondence relationship between the pixel coordinates formed by the target object in the scene image and the depth image is represented by the perspective transformation matrix.

The present application discloses a camera control device. When performing the camera control operation, the camera control device first determines, by the determining module, pixel coordinates of the target object in the scene image in each detection cycle, and determines the target object. Pixel coordinates in the depth image, then get The module acquires a distance between the target object and the camera based on pixel coordinates of the target object in the depth image, and then uses a first distance obtained by using the first detection period by the adjustment module, and a second detection period The obtained second distance adjusts the focal length of the camera to adjust the focal length of the camera.

Through the device, the distance between the target object and the camera in each detection cycle can be acquired, and the focal length of the camera can be adjusted by using the distance, the algorithm is simple, and the zoom accuracy is improved.

Accordingly, the present application also discloses a video camera. The camera includes a processor, a memory, an image sensor, and a depth sensor.

The memory is configured to store a program for controlling a camera;

In addition, in the camera disclosed in the embodiment of the present application, the image sensor and the depth sensor may be placed in different forms. In one form, the image sensor and the depth sensor are integrated in the same sensor. In addition, the image sensor and the depth sensor may also be two independent sensors.

When the image sensor and the depth sensor are two independent sensors, as shown in FIG. 8, wherein the image sensor and the depth sensor are both disposed behind the camera lens, the image sensor and the depth sensor are set at different levels. In height, a half mirror half lens is disposed between the camera lens and the image sensor and the depth sensor. Wherein the image sensor is generally at the same level as the camera lens and is arranged in a vertical direction with the camera lens, the depth sensor being disposed between the camera lens and the image sensor, and is generally disposed at a level Square The half-reflex lens is disposed above the depth sensor at a certain inclination angle with respect to the horizontal direction. The angle of inclination may be 45° or other angles, which is not limited in this application. In this scheme, the splitting is realized by using the half-reverse half lens, which can ensure that the scenes captured by the image sensor and the depth sensor are consistent, and the coordinates of the pixel points formed by the same object on the scene image are the same as the coordinates of the pixel points formed on the depth image. Moreover, in order to ensure the amount of light entering the image sensor to obtain a better image effect, the ratio of the amount of transmitted and reflected light of the half-reflex lens can be controlled, for example, the transmitted light accounts for 70% of the total amount of light, and the reflected light occupies the total amount of light. 30%.

Alternatively, when the image sensor and the depth sensor are two independent sensors, as shown in FIG. 9, the image sensor and the depth sensor are used to image different light rays, wherein the image sensor is disposed in the camera. The rear of the lens, and the image sensor is placed side by side with the depth sensor. Additionally, the image sensor is typically at the same level as the camera lens. In this case, since the image sensor and the depth sensor are used for different optical paths, the content of the two images is slightly different, causing parallax between the scene image and the depth image, so that the scene image needs to be calibrated to obtain the target object in the scene. Correspondence between pixel coordinates formed in the image and the depth image.

The camera disclosed in the present application is capable of acquiring a distance between a target object and a camera in different detection periods according to a scene image and a depth image, and adjusting the focal length of the camera by the distance. The algorithm adopted by the method is simple, easy to implement, and improves the accuracy and robustness of the focus adjustment.

It will be apparent to those skilled in the art that the techniques in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such understanding, the technical solution in the embodiments of the present invention may be embodied in the form of a software product in essence or in the form of a software product, which may be stored in a storage medium such as a ROM/RAM. , a diskette, an optical disk, etc., comprising instructions for causing a computer device (which may be a personal computer, server, or network device;) to perform the methods described in various embodiments of the present invention or in certain portions of the embodiments.

The various embodiments in the specification are described in a progressive manner, and the same or similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment.

The embodiments of the invention described above are not intended to limit the scope of the invention. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the invention are intended to be included within the scope of the invention.

Claims

A camera control method, the method comprising:

Determining, in each detection period, pixel coordinates of the target object in the scene image, and determining pixel coordinates of the target object in the depth image;

Obtaining a distance between the target object and the camera based on pixel coordinates of the target object in the depth image;

The focal length of the camera is adjusted using a first distance obtained by the first detection period and a second distance obtained by the second detection period.
The method according to claim 1, wherein the step of adjusting the focal length of the camera by using the first distance obtained by the first detection period and the second distance obtained by the second detection period comprises:

Obtaining the target according to a second distance d 2 between the target object and the camera obtained in the second detection period, and pixel coordinates (u 2 , v 2 ) of the target object in the scene image The world coordinates of the object in the second detection period (x c2 , y c2 );

When the focal length adjusted by the camera is set to f 2 , according to the world coordinate (x c2 , y c2 ) of the target object in the second detection period, the second distance d 2 and the target object in the scene the pixel coordinates (u 2, v 2) in the image, the focal length f 2 acquires the pixel coordinates (u 2, v 2) a corresponding relation;

Corresponding relationship between the focal point f 2 and the pixel point coordinates (u 2 , v 2 ), pixel coordinates (u 1 , v 1 ) of the target object in the scene image in the first detection period, Obtaining the pixel point coordinates (u 1 , v 1 ) and pixel point coordinates (u 2 ) by a first distance d 1 between the target object and the camera in a detection period and a focal length f 1 of the camera in the first detection period The difference of v 2 ) is the value of the focal length f 2 when the difference is within the preset range, and the focal length of the camera is adjusted to the focal length f 2 .
The method according to claim 1, wherein the step of adjusting the focal length of the camera by using the first distance acquired by the first detection period and the second distance acquired by the second detection period comprises:

After obtaining the first distance d 1 of the target object and the camera in the first detection period, acquiring a first ratio of the focal length f 1 of the camera and the first distance d 1 in the first detection period;

Setting a focal length adjusted by the camera to be f 2 , and acquiring a second ratio d 2 of the focal length f 2 and the second distance d 2 after acquiring the second distance d 2 of the target object and the camera in the second detection period;

Obtaining a value of the focal length f 2 when the difference between the first ratio and the second ratio is within a preset ratio range, and adjusting a focal length of the camera to a focal length f 2 .
The method of claim 1 further comprising:

After obtaining the first distance d 1 and the second distance d 2 , calculating a difference between the first distance d 1 and the second distance d 2 , and determining that the adjustment is needed when the difference is not within a preset threshold range The focal length of the camera.
The method according to claim 1, wherein the obtaining a distance between the target object and a camera based on pixel coordinates of the target object in the depth image comprises:

Obtaining a gray level of a pixel point corresponding to a pixel point coordinate of the target object in the depth image;

The distance between the target object and the camera is obtained by the correspondence between the gray level and the distance.
The method according to any one of claims 1 to 5, wherein the method further comprises:

Setting a focus priority of the target object to a highest priority according to the received setting information;

After adjusting the focal length of the camera, the focus position of the camera is adjusted to focus the camera at the position with the highest focus priority.
The method according to claim 1, wherein the determining pixel coordinates of the target object in the depth image comprises:

Determining pixel coordinates of the target object in the depth image based on a correspondence relationship between pixel coordinates of the target object formed in the scene image and the depth image, and pixel coordinates of the target object in the scene image .
The method according to claim 7, wherein the method further comprises: predetermining a correspondence relationship between pixel coordinates formed by the target object in the scene image and the depth image;

The step of predetermining the correspondence relationship between the pixel coordinates formed by the target object in the scene image and the depth image includes:

Obtaining an imaging relationship expression of the scene image and the depth image, the imaging relationship expression for:

Where x is a homogeneous representation of the pixel coordinates of the object in the scene image; x' is a homogeneous representation of the pixel coordinates of the object in the depth image; H is the scene image and the depth image Perspective transformation matrix;

Obtaining pixel coordinates of the four object points on the scene image and the depth image respectively, and acquiring the value of H in the imaging relationship expression, thereby acquiring a perspective transformation matrix of the same object in the scene image and the depth image And mapping, by the perspective transformation matrix, a correspondence relationship between pixel coordinates of the target object formed in the scene image and the depth image.
A camera control device, characterized in that the device comprises:

a determining module, configured to determine, at each detection period, pixel coordinates of the target object in the scene image, and determine pixel coordinates of the target object in the depth image;

An acquiring module, configured to acquire a distance between the target object and a camera based on pixel coordinates of the target object in the depth image;

And an adjustment module, configured to adjust a focal length of the camera by using a first distance obtained by the first detection period and a second distance obtained by the second detection period.
The apparatus according to claim 9, wherein the adjustment module comprises:

a first acquiring unit, configured to: according to the second distance d 2 between the target object and the camera obtained in the second detection period, and the pixel point coordinates of the target object in the scene image (u 2 , v 2 ) acquiring world coordinates (x c2 , y c2 ) of the target object in the second detection period;

a second acquiring unit, configured to: when the focal length adjusted by the camera is set to f 2 , according to the world coordinate (x c2 , y c2 ) of the target object in the second detection period, the second distance d 2 and coordinates of the pixels in the target object in the image scene (u 2, v 2), the focal length f 2 acquires the pixel coordinates (u 2, v 2) of the correspondence relationship;

a first adjusting unit, configured to: by a correspondence relationship between the focal length f 2 and the pixel point coordinates (u 2 , v 2 ), a pixel point coordinate of the target object in the scene image in the first detection period (u 1 , v 1 ), a first distance d 1 between the target object and the camera in the first detection period and a focal length f 1 of the camera in the first detection period, and acquiring the pixel point coordinates (u 1 , v 1 ) The value of the focal length f 2 when the difference between the pixel point coordinates (u 2 , v 2 ) is within a preset range, and adjusts the focal length of the camera to the focal length f 2 .
The apparatus according to claim 9, wherein the adjustment module comprises:

a third acquiring unit, configured to acquire a focal length f 1 and a first distance d 1 of the camera in the first detection period after acquiring the first distance d 1 of the target object and the camera in the first detection period First ratio;

a fourth acquiring unit, configured to set a focal length adjusted by the camera to be f 2 , and obtain a focal length f 2 and the second distance after acquiring the second distance d 2 of the target object and the camera in the second detecting period a second ratio of d 2 ;

a second adjusting unit, configured to acquire a value of a focal length f 2 when a difference between the first ratio and the second ratio is within a preset proportional range, and adjust a focal length of the camera to a focal length f 2 .
The device according to claim 9, wherein the camera control device further comprises:

A determining module, configured to acquire a first distance d 1 and the second distance d 2, d 1 calculates the first distance and the second distance d 2 difference, and when the difference is not within the preset threshold range , determining that the focal length of the camera needs to be adjusted.
The device according to claim 9, wherein the obtaining module comprises:

a gray level acquiring unit, configured to acquire a gray level of a pixel point corresponding to a pixel point coordinate of the target object in the depth image;

The distance obtaining unit is configured to acquire a distance between the target object and the camera by the correspondence between the gray level and the distance.
The device according to any one of claims 9 to 13, wherein the camera control device further comprises:

a setting module, configured to set a focus priority of the target object to a highest priority according to the received setting information;

a focusing module for adjusting a focus position of the camera after adjusting a focal length of the camera to focus the camera at a position with the highest focus priority.
The apparatus according to claim 9, wherein the determining module comprises:

a determining unit, configured to determine, according to a correspondence relationship between pixel coordinates of the target object formed in the scene image and the depth image, and pixel coordinates of the target object in the scene image The pixel point coordinates of the target object in the depth image.
The device according to claim 15, wherein the camera control device further comprises: a correspondence determining module, wherein the correspondence determining module is configured to predetermine pixels formed by the target object in the scene image and the depth image Correspondence of point coordinates;

The correspondence determining module includes:

An imaging relationship expression obtaining unit, configured to acquire an imaging relationship expression of the scene image and the depth image, where the imaging relationship expression is:

Where x is a homogeneous representation of the pixel coordinates of the object in the scene image; x' is a homogeneous representation of the pixel coordinates of the object in the depth image; H is the scene image and the depth image Perspective transformation matrix;

a perspective transformation matrix acquiring unit, configured to acquire pixel point coordinates of the four object points on the scene image and the depth image, respectively, and obtain a value of H in the imaging relationship expression, thereby acquiring the same object in the scene A perspective transformation matrix of the image and the depth image, by which the correspondence relationship of the pixel coordinates formed by the target object in the scene image and the depth image is characterized.
A camera, comprising: a processor, a memory, an image sensor, and a depth sensor,

Wherein the image sensor is configured to generate a scene image including a target object;

The depth image sensor is configured to generate a depth image including a target object;

The memory is configured to store a program for controlling a camera;

The processor is configured to read a program stored in the memory, and perform a camera-controlled operation according to the program, where the camera controlled operation comprises:

Determining, in each detection period, pixel coordinates of the target object in the scene image, and determining pixel coordinates of the target object in the depth image;

Obtaining a distance between the target object and the camera based on pixel coordinates of the target object in the depth image;

The focal length of the camera is adjusted using a first distance obtained by the first detection period and a second distance obtained by the second detection period.
The camera of claim 17 wherein:

The image sensor and the depth sensor are integrated in the same sensor;

or,

The image sensor and the depth sensor are both disposed behind the camera lens, the image sensor and the depth sensor are disposed at different levels, and a semi-reverse half lens is disposed between the camera lens and the image sensor and the depth sensor;

or,

The image sensor is disposed behind the camera lens, and the image sensor is placed side by side with the depth sensor.