RU2809064C1 - Device and method for automatically switching camera operating modes - Google Patents

Abstract

FIELD: smartphone camera.

SUBSTANCE: method for automatically switching operating modes of a smartphone camera and other similar devices. The result is achieved by first capturing an image of the scene, including the target object to be tracked, in one of: close-range, long-range shooting modes, optionally, providing data from at least one of: depth sensors, motion sensors, providing at least one of: stereo data, image data in at least two consecutive frames in one of: close shooting mode, long shooting mode, when receiving stereo data: determine the disparity D_k of the scene including the target object captured by the camera based on the received stereo data at a certain moment of time for each current frame, determine the distance d₁ from the camera to the target object and the measure of confidence R₁ of the resulting disparity D_k based on the obtained stereo data, when receiving image data: calculate the field of motion vectors on the scene including the target object captured by the camera, calculate by at least one of: a shift factor, a rotation factor, a scene zoom factor, based on the calculated field of scene motion vectors, optionally initializing a tracker for the target object for the current frame, determining a distance d₂ from the camera to the target object based on the zoom factor, and determine the measure of confidence R₂, determine the distance d₃ from the camera to the target object based on depth data using depth sensors and determine the measure of confidence R₃, determine the distance d₄ from the camera to the target object based on data from motion sensors and the measure of confidence R₄, when initializing the tracker: determine the distance d₅ and the confidence measure R₅ and perform a hybrid estimate of the distance from the camera to the target object in the current frame based on the obtained distance values d1, d₂, d₃, d₄, d₅ from the camera to the target object and confidence measures R₁, R₂, R₃, R₄, R₅, and the adjusted distance d_h from the camera to the target object in the current frame is determined, then a decision is made to switch the camera to the close shooting mode or to the far shooting mode based on the adjusted distance d_h.

EFFECT: reducing the time it takes for the camera to switch between modes, as well as increasing the accuracy of determining the shooting mode.

43 cl, 21 dwg

Description

Field of technology

The invention relates to methods and devices for automatically switching camera operating modes (hereinafter referred to as camera switching) when fixing the camera on an object depending on the distance to the object, and can be used, in particular, for photo and video shooting with smartphone cameras, as well as in tablet computers running operating systems Android, iOS, HarmonyOS and similar systems, as well as other similar devices that require automatic switching of camera operating modes.

Description of the Prior Art

The prior art knows many ways to switch operating modes of cameras currently used by well-known manufacturers of photo and video equipment; in particular, switching, for example, operating modes of a digital camera can be done manually by the camera user. In some cases, ensuring automatic camera mode switching requires specialized equipment, such as laser/infrared/radio motion sensors, radars, lidars. In this case, it is necessary to take into account the time required to switch camera operating modes and power consumption.

It should be noted that to shoot a target object located at a distance from the camera user, a multi-camera system is used, which includes several cameras that can switch between each other. These cameras may have closer focusing distances (macro camera, close camera), i.e. cameras for working at a short distance from the target object and with more distant ones (long-range camera, telephoto camera), i.e. camera for working at a great distance from the target object. We will consider a multi-camera system with at least two cameras that differ in focusing distances. We will call a camera with a closer focusing distance a near camera. And with the more distant - distant. Switching between cameras (near and far) in a multi-camera system will be called switching modes to near and far shooting modes, respectively.

To understand the problem solved by the present invention, consider typical cases of switching camera modes such as the very long shooting mode (telephoto camera), the long shooting mode and the close shooting mode, illustrated in Figs. 1A, 1B, 2A, 2B and 3A. 3B:

1) switching from long-range shooting mode (wide-angle camera) to very long-range shooting mode (telephoto camera) and back when shooting an object located at a considerable distance (see Fig. 1A, 1B),

2) switching from long-range shooting mode (wide-angle camera) to short-range shooting mode (ultra-wide-angle camera) and back when shooting an object at a close distance, or when it is necessary to capture most of the object or the entire scene (see Fig. 2A, 2B),

3) switching from very long-range shooting mode (telephoto camera) to long-range shooting mode (wide-angle camera) and vice versa, and from long-range shooting mode (wide-angle camera) to short-range shooting mode (ultra-wide-angle camera) and back when shooting an object in focus or out of focus, close or far away from the user depending on the user's requirements (see FIGS. 3A, 3B).

It should be noted that the mode of switching from long-range shooting mode to short-range shooting mode and vice versa lends itself best to automation and is more in demand among the end user.

In this case, the decision to automatically switch the camera in various methods is made by using additional information obtained based on calculations and measurements described below (1), (2) and (3).

(1) One type of such information, based on which the decision to switch the camera is made, is disparity (1), used to calculate the depth map of the scene. In this case, disparity (disparity or a measure of the defocus of a scene), in general, is calculated based on two matching images of the same scene, taken from two different angles, for example, using a dual pixel camera or a stereo camera. FIG. 4A shows the scene to be captured by the camera, and FIG. 4B shows the depth map of the scene from FIG. 4A based on which the camera switching decision is made.

(2) In addition, motion data (2) on the scene and sequential images (frames) from the camera are also used to make the decision to switch the camera, used to determine the directions of the target object and construct a vector field, while the direction and speed of the object can be calculated the necessary moment in time to switch camera lenses to improve the quality of photo and/or video shooting. In addition, to compensate for camera movement during shooting based on the initial position and direction of the camera, data from the current position and direction of the camera, obtained, for example, from an accelerometer, gyroscope, magnetometer and other motion sensors, usually built into smartphones, can be used. FIG. 5A shows the scene to be captured by the camera, and FIG. 5B shows the motion vector field of the scene from FIG. 5A based on which the camera switching decision is made.

(3) To make a decision about camera switching, scene depth data (3) obtained from laser, including infrared sensors, such as Time-of-Flight (ToF) sensors or lidars (LiDAR) can also be used. , based on the technology of measuring distance by emitting light from a laser or LED and measuring the time it takes for the reflected light to return to the receiver. FIG. 6A shows a scene to be captured by the camera, and FIG. 6B shows a range view of the scene from FIG. 6A based on which a camera switching decision is made.

The applicants have studied many similar solutions on the market and identified the main problems that need to be solved in a new method and device for automatically switching camera modes.

Traditional solutions for automatically switching camera modes require significant time to process and/or calculate information, up to a second, which in turn leads to a delay in the process of initializing camera switching and, accordingly, to a decrease in the quality of the image obtained when shooting with the camera.

One of the objectives of the claimed invention is to significantly reduce the time for switching the camera, which should be almost instantaneous and take several milliseconds so that the user does not notice the switching process.

In addition, in known solutions for automatically switching camera modes, errors in switching modes often occur when shooting an object, for example, when shooting an object through a window glass or showcase, when the sensors used mistakenly recognize the glass as an object, which initiates incorrect camera switching and leads to a decrease in the quality of the image obtained when shooting with the camera. The inventors overcome this drawback of the prior art by introducing additional information about the object as input, in particular, a depth map and a field of motion vectors of the scene containing the object, to decide whether to switch the camera mode.

In addition, known camera switching methods and devices often include special equipment, for example, various laser sensors, including infrared sensors, such as time-of-flight (ToF) sensors or LiDARs, which are built into the cameras, which requires significant power consumption and significantly increases the cost of the device. However, such power consumption is completely unacceptable for smartphones. The objective of the claimed method and device is also to reduce energy consumption.

The objective of the claimed method and device is to create a lightweight, compact, accurate device that allows you to automatically switch camera operating modes, preferably switching the camera between long- and short-range shooting modes when the distance between the camera and the target object exceeds a predetermined threshold value, based on information about the depth and the position of the target object obtained by determining the disparity and tracking data of the target object in the scene, as well as data taking into account the movement of the scene of interest including the target object (in particular, motion vector fields). In addition, the claimed solution can use additional motion data obtained from an accelerometer, gyroscope, depth sensors or other sensors to obtain a more accurate estimate of the position of the target object when making a switch decision. If there is a need to save money or the absence of such sensors, or the need to minimize energy consumption, such sensors may not be used, and the quality of the solution will deteriorate slightly. Thus, the solution is flexible and sustainable.

Known from the prior art is the mobile camera system US20160007008A1 (patent holder of Apple Inc.), IPC: G03B 13/34, H04N 13/02, publ. 01/07/2016, which discloses a method for autofocus based on depth data and involves obtaining two images of the same scene from two different cameras with different focal length ranges, in particular, a long-range camera and a telephoto camera. After receiving information from the user about the desired focus on the scene, for example, by touching the screen, a smooth transition is performed by combining frames obtained through the far camera lens and the telephoto lens.

The disadvantages of this solution are the need for direct user participation to control the camera. However, this shooting method can only be used for photo and video shooting in zoom mode.

In WO2020073959A1 (patentee Huawei Technologies Co Ltd.), IPC H04N 5/232, publ. 04/16/2020, a method and an electronic image capturing device are disclosed. The method is based on obtaining distance information from a distance sensor (i.e. laser and/or infrared sensor). Next, the obtained distance information is processed and analyzed to determine the appropriate focal length and then select one of the modes: very long shooting mode (telephoto camera), long shooting mode or close shooting mode.

The disadvantages of this solution are as follows: operation requires an additional distance sensor, the inclusion of which in the device significantly increases the cost of its production; in addition, direct user interaction is required to establish focus on the target object.

US11218641B2 (patent holder Qualcomm Inc.), published 01/04/2022, IPC H04N 5/225, H04N 5/232, discloses a method and apparatus for capturing image data. The method is based on calculating the scaling factor and combining pixels.

When the user-requested image zoom factor and/or the required pixel binning exceeds the camera switching threshold, a switch is made between cameras with different optical zoom factors and/or different focal lengths, i.e. between very long range, long range and close range modes. This method is manual and does not automatically switch the camera mode, since focusing on the target object is done at the user's request and then the process of switching the camera mode is carried out.

The closest technical solution of the claimed invention is a method for switching a camera and a device for a digital camera, disclosed in patent US10,291,842B2 (patent holder Samsung Electronic Co Ltd), IPC G06T7/00; H04N 5/225, publ. 05/14/2014 The method is based on obtaining images using a long-range camera and a very long-range camera (TV camera), calculating a depth map by matching stereo data, and using the depth map to obtain motion information. Based on the received motion information and the calculated speed of the target object, switching between the far camera and the very far camera (TV camera) is initiated.

The disadvantages of this solution are that the method can be used exclusively for photo and video shooting in zoom mode; In addition, two cameras working simultaneously increase the power consumption of the device as a whole.

All of the above solutions do not meet modern requirements dictated by electronic portable devices with built-in cameras for video/photo capture. At the same time, in modern electronic devices it is very difficult to combine the requirements for compactness of the device, allowing it to be used in mobile phones, while simultaneously maintaining the quality of the captured image in real time, which is ensured by automatic switching of camera modes.

To illustrate the advantages of the claimed method and device for automatically switching camera operating modes, below is a comparative analysis of the functionality of the claimed technical solution and the known technical solutions described above.

The claimed invention is characterized by the following:

1) provides switching of the camera from long-range shooting mode to short-range shooting mode and back, and can also be used to automatically switch from telephoto camera mode to long-range shooting mode when the target object is lost. The invention can also be used to switch from far and/or close shooting mode to very close shooting (macro photography) mode and back

2) stereo and image data are required,

3) camera control directly by the user is not required,

4) the method according to the invention uses depth data and motion vector field information of the scene containing the target object.

5) other data, for example, data from various sensors (ToF, LiDAR, UWB, accelerometer, gyroscope and others), can be used to improve accuracy, but are not necessary.

US 10,291,842 B2 (patented by Samsung Electronics Co Ltd) (prototype)

1) provides switching of the camera from long-range shooting mode to telephoto camera mode and back,

2) stereo data is required from the long-range shooting mode and telephoto camera,

3) camera control directly by the user is not required,

4) the method uses information about the movement of the target object, obtained based on depth data.

US20160007008A1(patented by Apple Inc.)

1) provides switching of the camera from long-range shooting mode to telephoto camera mode and back,

2) stereo data is required from the long-range shooting mode and telephoto camera,

3) camera control directly by the user is required when initiating camera mode switching,

4) the method uses data on the depth of the scene.

WO2020073959A1(patented by Huawei Technologies Co Ltd.)

1) provides switching of the camera from a telephoto camera to a long-range shooting mode and to a short-range shooting mode and vice versa,

2) data from the distance sensor to the target object is required,

3) camera control directly by the user is required when initiating camera mode switching,

4) sensor data is used (laser and/or infrared sensors).

US11218641B2 (patented by Qualcomm Inc.)

1) provides switching of the camera from a telephoto camera to a long-range shooting mode (wide-angle camera) and to a short-range shooting mode (ultra-wide-angle camera) and vice versa,

2) stereo data is required from the wide-angle, ultra-wide-angle and telephoto cameras,

3) camera control directly by the user is required when initiating camera mode switching,

4) scaling factor calculations are used.

The essence of the invention

According to a first aspect of the invention, there is provided a method for automatically switching camera operating modes, comprising the following steps:

- preliminarily capture an image of the scene, including the target object to be tracked, in one of: close-range, long-range shooting modes,

- optionally, provide data from at least one of: depth sensors, motion sensors,

- provide at least one of: stereo data, image data in at least two consecutive frames in one of the modes: close shooting mode, long shooting mode,

when receiving stereo data:

- determine the disparity D _k of the scene, including the target object, captured by the camera, based on the obtained stereo data at a certain point in time for each current frame,

- determine the measure of confidence R ₁ of the obtained disparity D _k based on the obtained stereo data,

- the distance d ₁ from the camera to the target object is determined based on a certain disparity D _k for the current frame

when receiving image data:

- comparing image data of at least two consecutive frames and, based on the comparison of said image data, calculating a field of motion vectors in the scene including the target object captured by the camera,

- calculating at least one of: shift coefficient, rotation coefficient, scene scaling coefficient, based on the calculated field of scene motion vectors,

- optionally, initialize the tracker for the target object for the current frame, carry out tracking of the target object on subsequent frames, and calculate the scaling factor of the bounding rectangle of the camera with the target object located in it in the tracking process for each frame,

- determine the exit of the target object from the frame, and if the exit of the target object from the frame is not detected: - determine the distance d ₂ from the camera to the target object based on the scaling factor obtained based on the calculated field of motion vectors,

- determine the measure of confidence R ₂ of the distance d ₂ from the camera to the target object,

- when receiving data from depth sensors: determine the distance d ₃ from the camera to the target object based on depth data using depth sensors, and determine the degree of confidence R ₃ of the distance d ₃ based on depth data using depth sensors,

- when receiving data from motion sensors: determine the distance d ₄ from the camera to the target object based on the data from the motion sensors and determine the degree of confidence R ₄ of the distance d ₄ based on the data from the motion sensors,

- when initializing the tracker: the distance d ₅ from the camera to the target object is determined based on the calculated scaling factor of the bounding rectangle with the target object located in it,

- when initializing the tracker: determine the confidence measure R ₅ of the distance d ₅ from the camera to the target object based on the calculated scaling factor of the bounding rectangle with the target object located in it, and

- perform a hybrid assessment of the distance from the camera to the target object in the current frame based on the obtained distance values d₁, d₂,d₃,d₄,d₅ from camera to target and confidence measures R₁, R₂, R₃, R₄, R₅, and the adjusted distance d_hfrom the camera to the target object in the current frame,

- decide whether to switch the camera to the close shooting mode or to the long shooting mode based on the adjusted distance d _h.

Moreover, in the claimed method, when the camera is in long-range shooting mode, if the resulting adjusted distance d _h is less than a predetermined threshold value, a decision is made to switch the camera to short-range shooting mode, and

when the camera is in close-range shooting mode, if the resulting adjusted distance d _h is greater than a predetermined threshold value, a decision is made to switch the camera to long-range shooting mode.

In this method, the image data represents one of: RGB, YUV and CMYK data, and stereo data is obtained using PDAF sensors of the camera matrix with Dual Pixel technology, while

the disparity value D _k for the selected target object in the current frame is determined by PDAF data obtained by the camera sensor matrix as the local minimum of the SAD curve, and

to determine the disparity D _k for the selected target object in the current frame based on the received PDAF data, which are the right and left images of the target object obtained by the camera sensor matrix, the sum of absolute differences is calculated as the absolute difference between the pixels of the left and right images, on the basis of which SAD curves are constructed.

According to the claimed method, the corrected distance d _h is determined based on the obtained values of the distances d ₁ , d ₂ , d ₃ , d ₄ , d ₅ from the camera to the target object and confidence measures R ₁ , R ₂ , R ₃ , R ₄ , R ₅ for the current frame from the expression according to the expression:

d _h= d ₁ * R ₁ + d ₂ * R ₂ + d ₃ * R ₃ + d ₄ * R ₄ + d ₅ * R ₅

Moreover, the confidence measure R₁for the current frame is calculated based on SAD curve data and is determined from the expression:

R ₁ ,

Where - signal level, - noise level, - global minimum and - subminimum of the SAD curve.

In addition, the confidence measure R₂for the current frame is calculated based on the data of the vector field, and the subsequent affine transformation of the vector field, and the confidence measure R₃for the current frame characterizes the approximation of the signal-to-noise ratio obtained based on depth sensor data and is determined from the expression:

R ₃ =100x ,

where R ₃ is a measure of confidence, Y _S is the signal level, Y _N is the noise level.

According to the method, the confidence measure R₄for the current frame characterizes the approximation of the signal-to-noise ratio obtained based on data from at least one motion sensor and

determined from the expression:

R ₄ =100x ,

where R ₄ is a measure of confidence, Y _S is the signal level, Y _N is the noise level.

In this case, according to the method, the confidence measure R ₅ is calculated based on the results of the tracker.

In addition, in the claimed method:

- the distance d ₁ from the camera to the target object is determined based on a certain disparity D _k for the current frame,

- the distance d ₂ from the camera to the target object is determined based on a certain zoom factor for the current frame,

- the distance d ₃ from the camera to the target object is determined based on depth data using depth sensors, and the distance d ₄ from the camera to the target object is determined based on motion data using motion sensors,

- the distance d ₅ from the camera to the target object is determined based on the scaling factor of the bounding rectangle with the target object located in it.

In this case, according to the method, the scaling factor of the bounding rectangle on the camera screen with the target object located in it is calculated for the current frame k, as the ratio of the sizes of the bounding rectangle of the current frame and the previous frame.

In the claimed method, additionally, optionally, predetermined cases of camera operation are determined when capturing a target object in the current frame, and these cases include at least one of: the target object is a transparent object, the target object is a reflective object, shooting a target object behind glass, low light or white wall effect when the camera looks at a uniform background, local movement of the subject with a stationary background, camera rotation or shake.

When the camera is operating in long-range shooting mode or in short-range shooting mode, the calculation of at least one of: shift coefficient, rotation coefficient, scene scaling factor is performed based on the affine transformation of the calculated field of scene motion vectors, in addition, when the camera is operating in long-range shooting mode shooting, determine a measure of confidence R ₂ of the distance d ₂ from the camera to the target object based on the calculated field of scene motion vectors and the affine transformation of this vector field.

Additionally, when the camera is in close-up mode, when the tracker is initialized, if the bounding box containing the target object extends beyond the boundaries of the current frame, the loss of the target object in the current frame is determined, and when it is determined that the target object is lost in the current frame, a decision is made about switching the camera to long-range shooting mode.

In this case, after establishing the loss of the target object in the current frame: the distance d ₁ from the camera to the target object is additionally determined based on a certain disparity D _k for the current frame, and the distance d ₃ from the camera to the target object is additionally determined based on depth data using sensors depth for the current frame.

According to the first aspect of the invention, the method, when operating the camera in close-up mode, further determines the distance d _h after determining the loss of the target object in the current frame, taking into account the additional determined distances d ₁ and/or d ₃ .

In this case, if the obtained distance d _h after establishing the loss of the target object in the frame is less than a predetermined threshold value, a decision is made to maintain the close shooting mode and move to the next frame, and if the obtained distance d _h after establishing the loss of the target object in the frame is greater or equal to a predetermined threshold value, a decision is made to switch the camera to long-range shooting mode.

In addition, according to the method, at least one of the shift coefficient, the rotation coefficient, and the scene scaling coefficient is also calculated by applying an affine transformation model to the calculated vector field.

According to the second aspect of the invention, a device for automatically switching camera operating modes is provided, comprising:

- an input/output unit, including a graphics processor, a display, a camera, an input/output interface unit,

- a computing unit containing a central processor, memory controller, system memory and storage,

wherein the camera is a multi-camera system, including a photo and/or video camera, and containing an image forming unit containing a short-range lens and a long-range lens, and a sensor matrix, wherein the camera is configured to capture a target object and transmit image data of the target object to system memory of the calculation unit through the input-output interface unit,

the calculation unit is configured to implement the steps of the method according to the first aspect of the invention, and calculate the distance to the target object and make a decision on switching the camera to the close shooting mode or to the far shooting mode, while the resulting distance to the target object and the decision to switch the camera is transmitted to the system memory and through the block I/O interface to the camera to switch the camera mode.

the camera is configured to transmit graphic information through the input-output interface unit and the graphics processor to the device display,

the display is designed to visualize the target object in real time.

Moreover, in the claimed device, the graphics processor is configured to process graphic information received from the camera, and the sensor matrix is a matrix of PDAF sensors, the drive is configured to store photo and video images received by the device’s camera and is a solid-state drive (SSD) .

Moreover, in the claimed device, the input/output block additionally contains at least one motion sensor and a depth sensor, wherein the motion sensor is at least one of: an accelerometer, a gyroscope, a magnetometer, and the depth sensor is at least one of: RaDAR, LiDAR ToF sensor, UWB sensor.

In addition, the memory controller is configured to transfer data from the system memory to the central processing unit for computation.

The claimed device is designed to be built into one of the following: smartphone, tablet computer, photo and video camera.

Brief description of drawings

The above and other features and advantages of the present invention will become more apparent from the following description, taken in conjunction with the drawings, in which:

Fig. 1A shows an image of a scene when switching from a long-range shooting mode to a telephoto camera mode and back when shooting an object located at a distance, according to the prior art.

Fig. 1B is a schematic representation of the viewing angles of a scene when switching from a long-range shooting mode to a telephoto camera mode and back when shooting an object located at a distance, according to the prior art.

Fig. 2A shows an image of a scene when switching from a long-range mode (wide-angle camera) to a close-range mode (ultra-wide-angle camera) and back when shooting a close-up object or capturing an entire scene, according to the prior art.

Fig. 2B is a schematic representation of the viewing angles of a scene when switching from a long-range mode (wide-angle camera) to a close-range mode (ultra-wide-angle camera) and back when shooting a close-up object or capturing an entire scene, according to the prior art.

Fig. 3A shows an image of a scene when switching from a telephoto camera mode to a long-range shooting mode (wide-angle camera) and back, and switching the camera from a long-range shooting mode to a short-range shooting mode (ultra-wide-angle camera) and back when shooting a scene with a change in the required focal length to the object depending on the user's requirements, according to the known level of technology.

Fig. 3B - schematically shows the viewing angles of a scene when switching from a telephoto camera mode to a long-range shooting mode (wide-angle camera) and back, and switching a long-range shooting mode to a short-range shooting mode (ultra-wide-angle camera) and back when shooting a scene, changing the required focal length to object depending on the user's requirements, according to the known level of technology.

FIG. 4A is an image of a scene from which disparity is calculated according to the prior art.

FIG. 4B shows a scene depth map based on the obtained scene disparity from FIG. 4A according to the prior art.

FIG. 5A shows an image of a scene on the basis of which the field of scene motion vectors is calculated according to the prior art.

FIG. 5B illustrates a field of motion vectors based on motion data in the scene of FIG. 5A according to the prior art.

6A is an image of a scene from which a scene depth map is generated based on data obtained from laser sensors in accordance with the prior art.

FIG. 6B shows a range image of the scene from FIG. 6a obtained from data obtained from laser sensors according to the prior art.

FIG. 7A is a flowchart of steps of a method for automatically switching camera operating modes from a long-range shooting mode (wide-angle camera) to a close-range shooting mode (ultra-wide-angle camera) according to an embodiment of the invention.

FIG. 7B is a flowchart of steps of a method for automatically switching camera operating modes from a close-in shooting mode (ultra-wide-angle camera) to a long-range shooting mode (wide-angle camera) according to an embodiment of the invention.

Fig. 8A shows an image of the scene when the camera application is launched in the long-range shooting mode.

FIG. 8B shows automatic motion estimation in a scene as seen by the user on the camera.

FIG. 8C shows the automatic estimation of the distance to the target object as seen by the user on the camera.

8D shows an image of the target object during the measurement evaluation analysis phase of FIGS. 8B, 8C as seen by the user on the camera.

FIG. 8E is a view of the scene when switching the camera, as seen by the user on the camera.

Figure 9 shows an example of a bounding box in the form of 2D grid nodes.

Fig. 10 - shows a block diagram of a device for automatically switching camera operating modes.

Preferred Embodiments of the Invention

Examples of embodiments of the present disclosure will be described in detail below. Examples of embodiments have been illustrated in the accompanying drawings, in which the same or similar reference numerals refer to the same or similar elements, or elements having the same or similar functions. The exemplary embodiments described with reference to the accompanying drawings are illustrative and are used only to explain the present disclosure and should not be construed as limiting the same in any way.

Within the framework of the present invention, the following concepts and terms will be used, the interpretation of which is provided below by the authors of the invention:

Target object - an object selected by the user and to be photographed (captured).

Scene is an area selected by the user in which the target object is located.

A region of interest (ROI) is a specific area (region) of the device screen, selected using the “grid” functionality in the photo and video application inherent in modern smartphones running Android, iOS, etc. operating systems. For example, the device screen may be divided into several equal or unequal rectangles or other shapes that form a grid. One cell or rectangle of this grid, selected by the user, is called a region of interest (ROI). As an example, the inventors considered a screen divided into equal rectangles forming a 9x13 grid, but this example is not limiting. In this case, there are 9x13 ROIs on the screen.

Selected region of interest (selected ROI) is a region of interest defined based on the user-selected area, which may contain the center of the user-selected area, most of it, or the entire user-selected area, depending on the user's requirements when performing the autofocus process on the subject . If the user does not select the region himself, the ROI can be selected automatically. Figure 2A shows the selected ROI. The selected ROI, as well as its coordinates, are considered as input to begin the target tracking process.

Current region of interest (current ROI) - on the first frame, the current ROI corresponds to the selected ROI, and on subsequent frames, the current ROI is the region of interest that the tracker returns for the current frame. The user selects ROI only the first time. It should be noted that each frame is divided conditionally into an ROI grid, for example 13x9, at the first iteration the ROI is selected based on user input (conventionally, the user shows where the object of interest is located), then this information is transferred to the tracking tool (tracker), and for Each subsequent frame is predicted in which ROI from the grid the target object will be located.

Multi-camera system - a system that has at least two cameras that can switch between each other. The cameras can be: a short-range camera, such as an ultra-wide-angle camera; long-range camera, such as a wide-angle camera; a very short-range camera, such as a macro camera, and a very long-range camera, such as a telephoto camera

Close-in mode - short-range camera in a multi-camera system. This camera cannot focus at long distances. For example, it can focus at a distance of less than 30 cm, but the characteristics may be different.

Long-range shooting mode - long-range camera in a multi-camera system. This camera cannot focus at close range. For example, it can focus at a distance of more than 30 cm, but the characteristics may be different.

The present invention can be used for different cameras in a multi-camera system, but within the scope of the present disclosure, the inventors consider switching a camera from a close-in mode to a far-shot mode and vice versa.

A tracking tool (tracker) is a sequence of data processing stages through which, based on the input data, which is the data of the current ROI with the object to be captured, the output provides data (coordinates) of the object on subsequent frames.

PDAF data are images obtained by reflecting light rays from an object in the frame and striking them on opposite (left and right) sides of the camera lens. The object is in focus if the two resulting images match. Images obtained in this way are also called left and right images, respectively.

Dual Pixel is a phase-detection autofocus technology that uses each pixel in the sensor to detect phase and obtain color information, significantly improving both the speed and accuracy of autofocus compared to conventional systems.

SAD curve - a curve on a 2D graph: the X axis corresponds to the displacement in the number of pixels of the left image relative to the right one, the Y axis corresponds to the corresponding sum of pixel-by-pixel differences between the images for the selected fragment; The inventors obtain SAD curves using sensors built into a mobile phone, which are called dual pixels and contain information about two pictures (the left and right image of an object from the phone with a slight shift).

Disparity (mismatch) is a measure of scene defocus, obtained based on analysis of the SAD curve. Formally, disparity is the local minimum point of the SAD curve, and is therefore measured in the number of pixels.

A special (pre-defined) camera use case is any camera use case that differs from standard camera use cases and in which switching the camera may be unreasonable, such as when there is a transparent object, a reflective object, shooting behind glass, turning and camera movement in close-up mode, very small objects, textureless objects, textureless background without an object (the case with a white wall), the scenario of walking through a room, removing an object from the frame, moving the camera away from an object, object movement with a stationary camera.

Typical Camera Use Case - For the purposes of this disclosure, a standard case is understood to be a scenario where the user is shooting a non-transparent and non-reflective medium-sized 2D or 3D target object (the object has some kind of texture), the user approaches with the camera turned on in long-range shooting mode closer to object, the movement is not too fast or slow, uniform in speed, there may be a camera rotation, there may be no rotation. When approaching an object at a certain distance specified by the threshold, it switches to close-range shooting mode. The user takes a photo or video. Then it moves away from the target object; when it moves away a certain distance specified by the threshold, it switches to the long-range shooting mode.

Range image (depth map) - an image formed in such a way that each pixel, instead of color, encodes information about the distance between the corresponding point on the scene and the viewing point (camera).

A measure of confidence is a measure that indicates the extent to which the resulting measurement or calculation (e.g., sensor performance, computed image features) is affected by noise, and therefore the extent to which the accuracy of the measurement can be relied upon. In general, confidence can be assessed as the signal-to-noise ratio by the expression:

where R is a measure of confidence, Y _S is the signal level, Y _N is the noise level.

Examples of calculating a measure of reliability for specific measurements or calculations can be found in the following works, reflecting the prior art:

[1]US8068543B2, IPC H04B 1/66, published November 29, 2011;

[2]Liu, Jun & Ye, Ping & Zhang, Xingchen & Xiao, Gang. (2021). Real-time long-term tracking with reliability assessment and object recovery. IET Image Processing. 10.1049/ipr2.12072.;

[3]J. Zhu, L. Wang, R. Yang, J. E. Davis and Z. pan, "Reliability Fusion of Time-of-Flight Depth and Stereo Geometry for High Quality Depth Maps," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 7, pp. 1400-1414, July 2011, doi: 10.1109/TPAMI.2010.172.

Bounding rectangle is a rectangle that limits the shape of the target object on the scene plane.

Scaling factor is a factor that reflects the change in size of the target object between successive frames. By way of example, and not limitation, the scaling factor can be calculated through the aspect ratio of the corresponding bounding boxes of the target object; The prior art also describes methods for calculating the scaling factor based on the estimation of motion vectors, these include:

[4]WO2013106425A1, IPC H04N 7/26, publ. 07/18/2013;

[5]Wang H, Shen J, Chen Z, Shen J. A Fast Object Tracking Approach Based on the Motion Vector in a Compressed Domain. International Journal of Advanced Robotic Systems. 2013;10(1). doi:10.5772/55210;

[6]Alvar, Saeed & Bajic, Ivan. (2018). MV-YOLO: Motion Vector-Aided Tracking by Semantic Object Detection. 1-5. 10.1109/MMSP.2018.8547125.

Hybrid object distance estimation is a method for estimating the distance from a camera to a target object, using different sources of scene depth data to produce a final distance estimate. By way of example, and not limitation, a hybrid estimate may use a weighted combination of depth data from different sources (e.g., PDAF data, depth sensors, and motion vectors), and may also be based on selecting the most reliable source of depth data through a confidence assessment (reliability).

Long-range to short-range shooting mode switching is the process of switching from using a long-range camera lens to using a short-range camera lens when capturing an image containing a target object in motion and changing the corresponding optimal focal length range of the camera lens.

Shooting mode switching from short-range to long-range is the process of switching from using a long-range camera lens to using a short-range camera lens when capturing an image containing a target object in motion and changing the corresponding optimal focal length range of the camera lens.

Fig. 7A, Fig. 7B present block diagrams of the stages of automatic switching of camera modes, and Fig. 8A-8E show images on the camera that the user observes when shooting a scene and/or target object corresponding to the sequence of stages from Fig. 7A- 7B.

Next, the sequence of stages of automatic camera switching will be described in more detail with reference to Fig. 7A, Fig. 7B, and the corresponding Figs. 8A-8E:

1) The smartphone user launches the camera application, in which the camera's long-range shooting mode can be selected by default or set by the user. Optionally, the user can manually select the target object by touching a corresponding area on the screen with a finger, such as a smartphone. Based on the data received from the smartphone camera, the image itself and stereo data for it are formed. By way of example, and not limitation, stereo data can be obtained via PDAF sensors on a camera sensor with Dual Pixel technology.

It should be noted that it is not necessary to start launching the camera application in far shooting mode; the user can also be in close shooting mode if his settings save the last shooting mode and it was close.

This input step is illustrated in FIG. 8A, where the user starts the camera in long range mode.

2) Using the obtained data, constructing a field of motion vectors and performing disparity assessments and confidence measures.

FIG. 8B shows automatic motion estimation in the scene as seen by the user on the camera.

3) Perform distance estimation through motion vectors and perform distance estimation through stereo data, and also based on the acquired data, special or predefined use cases can be identified for more stable operation and better user experience. At the user's option and with appropriate sensors built into a smartphone or similar device, distance estimation can be performed based on additional scene motion data obtained from a gyroscope, accelerometer, magnetometer, or any other sensors (see FIG. 7A, FIG. 7B)

It should be noted that distance estimation can be performed by various methods known in the art, for example by means of: motion vectors, stereo data, motion sensor data, depth sensor data, data obtained using a tracking algorithm, and different types of data are used. And all these distance estimates that can be obtained in this step 3) are combined into one distance estimate in step 4 using hybrid distance estimation

The automatic estimation of the distance to the target object as observed by the camera user is presented in Fig. 8C. 4) Next, a hybrid distance estimation is performed, which will be described in more detail later in the description, in which the estimates and calculations obtained in steps 2 and 3 are analyzed.

5) Based on the object distance data obtained in the hybrid distance estimation step in step 4, threshold values for lens selection are checked. Based on this check and the analysis of edge cases, a lens is selected for the long-range shooting mode or the near-photographing mode of the camera operation and automatically switches the camera.

FIG. 8E shows the picture that the user sees when switching the camera.

Next, we will dwell on the description of the hybrid algorithm mentioned with reference to Fig. 8C and Fig. 7A, Fig. 7B, on the basis of which an automatic estimate of the distance to the target object observed by the camera user is performed.

Key Point 1 : Hybrid distance estimation for determining the distance to an object (see Fig. 7A, Fig. 7B).

1) If the user is in the camera's long-range shooting mode (wide-angle mode), the user opens (launches) the camera application through which it receives the image data in the current frame, and the images can be encoded in a color space, such as RGB, YUV or CMYK, and stereo data , for example, PDAF data or matched image data from two cameras based on the capture of the target object.

2) The distance to the target object is determined using stereo data. Methods and means of obtaining distance using stereo data are disclosed in the following sources of information: [7] RU2778355C2, publ. 08/17/2022 SAMSUNG ELECTRONICS CO., LTD., KR.

3) Based on the acquired target image data, tracking of the target object by the tracker is initiated, and the acquired tracking data is used to determine the scaling factor of the tracker's bounding box between successive frames of the captured target image. Based on the zoom factor, the distance to the target object is calculated. The resulting tracking data is also used to detect some edge cases, such as losing the target object in the frame. The process of tracking a target object is described in detail in patent [7] RU2778355C2, publ. 08/17/2022, the same patent holder (SAMSUNG ELECTRONICS CO., LTD., KR) as the patent holder of the claimed invention.

4) Based on the acquired two consecutive camera images, a motion estimation process is initiated, which is also used to determine the zoom factor between successive frames of the captured image of the target object. The motion estimation process is described in the following information sources:

[8] Pesquet-Popescu, B., Cagnazzo, M., & Dufaux, F. (2013). Motion Estimation Techniques;

[9] Qian, Xueming. (2012). Global Motion Estimation and Its Applications. 10.5772/34166.

Based on the zoom factor, the distance to the target object is calculated.

The scaling factor in this case is calculated through the estimation of motion vectors, for example, as disclosed in [10] Lai, Lijun & Xu, Zhiyong. (2015). Global Motion Estimation Based on Fourier Mellin and Phase Correlation. 10.2991/cmes-15.2015.173.

5) Other data and methods may be used to determine the distance to the target object.

By way of non-limiting example, PDAF, motion and depth sensor data may also be used.

6) Special use cases are identified, which will be described below.

7) Confidence measures R=100x are calculated (1),

where Y _S is the signal level, Y _N is the noise level for each method used to determine the distance to the target object.

8) The use of hybrid distance estimation in the final estimation of the distance to the target object is characterized by the inclusion of data obtained in stages 2)-7). Moreover, such devices of the prior art use only one source, i.e., for example, only image data, or only stereo data to determine the distance to the target object, which causes some errors in estimating the distance to the object and, accordingly, errors when switching cameras from one mode to another.

The claimed invention according to the first embodiment uses several methods for determining the distance to a target object. For example: using target object tracking, using motion vectors on the scene with the target object, using PDAF data. All these estimates and their confidence measures are accumulated into a single, more accurate estimate of the distance to the target object, allowing accurate camera switching even in the absence of depth sensors. Depth sensors and other sensors (such as motion data sensors) can also be used for hybrid estimation to further increase the accuracy of the solution if such sensors are available.

According to the first embodiment, the hybrid algorithm for estimating the distance to the target object described above with reference to Fig. 7A, Fig. 7B may also include input of data from various types of sensors, for example, an accelerometer, a gyroscope, a magnetometer.

Motion data is determined based on measurements obtained from various types of sensors, such as accelerometers, gyroscopes, magnetometers, as well as various depth sensors, in particular radio wave based (RADAR), based on the technology of measuring distances by emitting laser light and measuring time returning reflected light to the receiver (time-of-flight (ToF) sensors or lidars (LiDAR)), or ultra-wideband (UWB) sensors based on the use of ultra-wideband signals in the range from 6-8 GHz.

Based on the received movement data from various sensors, the distance to the target object is determined, which is entered into a hybrid algorithm for the final assessment of the distance to the target object.

The flexibility of the proposed solution according to the first embodiment allows the inclusion of various motion data from different sensors (for example, accelerometer, gyroscope, ToF, LiDAR or UWB. It should be noted that in the claimed method, a separate estimate of the distance to the object can be calculated for each data source. Data motion sensors can be used collectively or individually and can be used to produce an estimate of the distance to an object, which can be converted along with other estimates, for example based on depth data from the depth sensors, into a common unified estimate via a hybrid distance estimate to the object. A key role in hybrid estimation is played by the definition of confidence measures, which will be described in more detail later in the present, which allows you to combine different distance estimates into one and thereby provide a more accurate assessment of the movement of the target object in the scene, and provide more accurate camera switching .

Key Point 2: Handling a special (predefined) camera use case.

1) The user opens (launches) the camera application (camera far mode (wide angle mode) (see Fig. 7A) or camera close mode (ultra wide angle mode) (see Fig. 7B), through which image data such as RGB is obtained , YUV or CMYK data, and stereo data, such as PDAF data or matched image data from two cameras based on the capture of the target object.

2) The disparity and confidence measure for a frame of selected regions of interest (ROI) are estimated using stereo data.

3) The position of the camera lens is determined based on the obtained disparity and the confidence measure of the selected ROIs.

4) Based on the received image data of the target object, the process of tracking the target object through a tracking tool (tracker) is initiated.

5) Motion vectors in the scene with the target object are determined based on the obtained image data from successive frames.

6) Special use cases are identified based on the analysis of data obtained in Steps 2-6 of Key Point 2. For example, if a target object is lost from the frame, which was determined using the tracking tool, a decision may be made to switch the shooting mode from close to long-range mode. Also, disparity information and confidence measures obtained from stereo data as well as the image itself can be used for scene recognition, target object and background (non-object) identification. For example, if the absence of an object is recognized (the case of a homogeneous background), a decision may be made not to switch the shooting mode from far to near.

It should be noted that traditional known methods for processing target object distance data do not determine cases of special use, for example, when a camera captures a scene with a target object through a window glass or showcase, etc., while the sensors mistakenly recognize glass as an object, which initiates an incorrect switching the camera leads to a decrease in the quality of the image obtained when shooting with the camera.

The ability to identify special (predefined) use cases to automatically switch camera modes improves the accuracy of motion estimation of a target object in a scene, the accuracy of distance determination for a target object, and provides a better user experience. Using only image data to handle special use cases makes the use of expensive hardware (such as motion sensors) unnecessary.

Effects achieved when using a method and device for automatically switching operating modes of a smartphone camera.

General effects:

- provides precise automatic switching between near and far shooting modes;

- provides a lightweight, compact device

suitable for both premium and mass market devices.

Effects achieved through hybrid processing for estimating distance to a target object (Key Point 1)

- ensures the use of various sources of distance information to accurately estimate the distance to the target object;

- appropriate camera switching is ensured;

- the claimed method and device can be used for the mass market.

Effects achieved by incorporating sensor data into a hybrid data processing process for estimating distance to a target object Key Point 1, Option 1):

- provides a flexible solution that allows you to easily enter depth data from other information sources;

-provides the possibility of better functioning for premium devices.

Effects provided by handling a special use case (Key Point 2):

- improved accuracy of the device in special use cases

- lower energy consumption due to rapid detection of special (pre-defined) use cases and their separate processing.

In addition, the authors of the invention conducted studies of the specific operational parameters of the flagship solutions known on the market, as well as the claimed solution, which clearly demonstrate the advantage of the claimed device and method for automatically switching camera operating modes.

In this case, the main operational parameters were selected: operating time of the current solution, power consumption, successful camera switching in %, unwanted switching in %, switching delay in % and unstable camera switching distance in %.

The claimed invention is characterized by the following:

1) operating time ~9 ms,

2) power consumption 60 mA,

3) successful camera switching ~97%,

4) unwanted switching ~0%,

5) switching delay ~2%,

6) unstable camera switching distance ~1%.

US10,291,842B2 (US20160381289А1) (patent holder Samsung Electronics Co Ltd), Galaxy S21ULTRA model:

1) operating time ~15 ms,

2) power consumption 54 mA,

3) successful camera switching ~60%,

4) unwanted switching 31%,

5) switching delay ~8%,

6) unstable camera switching distance ~1%.

US20160007008A1 (patented by Apple Inc.), iPhone 13 Pro Max model:

1) data is unknown from open sources,

2) data is unknown from open sources,

3) camera control directly by the user is required when initiating camera mode switching,

3) successful camera switching 27%,

4) unwanted switching 33%,

5) switching delay ~7%,

6) unstable camera switching distance ~33%.

WO2020073959A1 (patented by Huawei Technologies Co Ltd.), model P50Pro:

1) data is unknown from open sources,

2) data is unknown from open sources,

3) successful camera switching 97%,

4) unwanted switching 15%,

5) switching delay ~66%,

6) unstable camera switching distance ~66%.

The presented analysis of the operational parameters of well-known smartphones on the market clearly demonstrates the advantages of the claimed method and device for automatically switching the operating mode of the smartphone camera.

In addition, the authors of the invention conducted research on the specific functionality of flagship solutions known on the market, as well as the claimed solution, which clearly demonstrate the advantage of the claimed device and method for automatically switching camera operating modes.

In this case, the main functionalities were chosen: the presence of a motion sensor, a switching circuit, automatic switching, solving the problem of switching delay, solving the problem of unwanted switching, solving the problem of unstable behavior in cases of special use.

The claimed invention is characterized by the following:

1) it is not necessary to have a motion sensor,

2) the switching scheme consists of switching the camera from long-range shooting mode to short-range shooting mode and back,

3) the presence of automatic switching,

4) switching delay problems are solved,

5) unwanted switching problems are solved,

6) problems of unstable behavior in cases of special use are solved.

US10,291,842B2 (US20160381289А1) (patent holder Samsung Electronics Co Ltd), Galaxy S21ULTRA model:

1) the presence of a motion sensor,

2) the switching scheme consists of switching the camera from long-range shooting mode to short-range shooting mode and back,

3) the presence of automatic switching,

4) switching delay problems are not solved,

5) problems of unwanted switching are not solved,

6) problems of unstable behavior in cases of special use are not solved.

US20160007008A1 (patented by Apple Inc.), iPhone 13 Pro Max model:

1) the presence of a motion sensor,

2) the switching scheme consists of switching the camera from long-range shooting mode to short-range shooting mode and back,

3) the presence of automatic switching,

4) switching delay problems are not solved,

5) problems of unwanted switching are not solved,

6) problems of unstable behavior in cases of special use are not solved.

WO2020073959A1 (patented by Huawei Technologies Co Ltd.), model P50Pro:

1) the presence of a motion sensor,

2) the switching scheme consists of switching the camera from long-range shooting mode to short-range shooting mode and back,

3) the presence of automatic switching,

4) switching delay problems are not solved,

5) problems of unwanted switching are not solved,

6) problems of unstable behavior in cases of special use are not solved.

The presented analysis of the functionality of well-known smartphones clearly demonstrates the advantages of the claimed method and device for automatically switching operating modes of a camera, for example a smartphone.

Next, the main key provisions of the method and device for automatically switching camera operating modes will be described. As previously stated, it is within the scope of the present invention to automatically switch the camera mode from a far shooting mode to a close shooting mode and vice versa.

When the camera is operating in long-range shooting mode (wide-angle mode), see Fig. 7A, the following steps are performed:

When capturing an image of a scene with a target object in the long-range shooting mode of the camera, data is obtained for the current frame, namely, stereo data of the current image of the kth frame is provided, including, for example, PDAF data obtained using PDAF sensors on the camera matrix with Dual Pixel technology or matched image data from the two cameras based on the capture of the target object, as well as image data if necessary. Each frame of the resulting image is considered as pixels or a group of pixels. Optional: provide data acquisition from depth sensors and/or acquisition of data from motion sensors.

Based on the obtained stereo data using the left and right images from a Dual Pixel camera or another pair of images, for example from a stereo camera, disparity is calculated, which is calculated based on the SAD (sum of absolute differences) curves between the left and right images for the captured scene with target object obtained using the camera.

It should be noted that to determine the disparity (D _k ) for the selected target object for the k-th frame based on the received PDAF data, which is the right and left images of the target object obtained by the camera sensor matrix, the sum of absolute differences is calculated as the absolute difference between pixels left and right images, on the basis of which SAD curves are constructed. Formally, disparity is the local minimum point of the SAD curve, and is therefore measured in the number of pixels. In this case, the SAD curve is a curve on a 2D graph: the X axis corresponds to the shift in the number of pixels of the left image relative to the right one, the Y axis corresponds to the corresponding sum of pixel-by-pixel differences between the images for the selected fragment; The inventors obtain SAD curves using sensors built into a mobile phone, which are called dual pixels and contain information about two pictures (the left and right image of an object from the phone with a slight shift).

Publications that describe disparity estimation through stereo reconstruction are presented below:

[11] Raajan N, Ramkumar M, Monisha B, Jaiseeli C, Venkatesan SP (2012) Disparity estimation from stereo images. In: International conference on modeling optimization and computing, Springer, vol 38, pp 462-472;

[12] S. Mukherjee and R. M. R. Guddeti, “A hybrid algorithm for disparity calculation from sparse disparity estimates based on stereo vision,” in Proceedings of IEEE International Conference on Signal Processing and Communications (SPCOM), (IEEE, 2014), pp. 1-6.

Based on the obtained SAD curves, the measure of confidence R ₁ and the obtained disparity D _k for the captured target object for the k-th frame are estimated.

In relation to the claimed solution, the signal-to-noise ratio can be approximated using two minimum values of the SAD curve - the global minimum and the subminimum, based on the specified we determine the confidence measure R ₁ :

Where - signal level, - noise level, - global minimum and - subminimum. Formula (2) characterizes the approximation of the signal-to-noise ratio using the global minimum and subminimum of the SAD curve.

The data processing process continues and the distance d ₁ to the target object captured by the camera is determined based on the obtained stereo data.

Optionally:

- Image data of at least two consecutive image frames is compared and based on the comparison of said image data, a field of motion vectors mv _{i, j} in the scene including the target object captured by the camera is calculated. The central region of the calculated field of vectors mv _{i, j} is selected, to which the affine transformation model is applied and the shift, rotation, and scaling coefficients are calculated. All coefficients are calculated simultaneously. The basic principles of affine transformation are disclosed in the following source of information:

[13] I.M. Vinogradov. Affine transformation // Mathematical encyclopedia. - M.: Soviet Encyclopedia. - 1977-1985.

Each vector mv _ij shows the movement of one point from frame to frame, i.e. ( , ) are the coordinates of the beginning of the vector on the frame under consideration k , and ( , ) - coordinates of the end of the vector on frame k+1. In this case (A i, j) are the indices of the traversal over the field of vectors, i.e. vectors from many points on one frame. The vector mv _ij describes the transition from the point of origin of the vector with coordinates ( , ) in the current frame k to the end point of the vector with coordinates ( , ) in the next frame k+1 (i.e. the vector itself ). In matrix form for one pair of coordinates

( , )->( , ) affine transformation looks like this:

Those. an affine transformation is an equation of the form (4) for each vector from the region, where Q _k , P _k and A are components in matrix form characterizing the vectors in the current and subsequent frames k and k+1, respectively. The shift, rotation and scaling coefficients are obtained from combinations of the desired roots of the equation , Where -values of matrix A.

They are calculated using the least squares method (LSM) - searching for roots by minimizing the sum of squared differences between the left and right sides of the equations for each vector, i.e. function minimization , in matrix form, where matrix :

Revealing

To minimize, it is necessary to equate the gradient of the function to 0.

We get a new matrix equation: (7). Finally, the roots of the equation (matrix coefficients ) can be found by the Gaussian method. In this case, k denotes the current frame, T-transposed matrix.

Example for a matrix the required coordinate transformation:

Values - are responsible for shifting the current frame along the X and Y axes, respectively, relative to the previous frame. Values are responsible for changing the scale, and the scaling factor is calculated as:

Where

ScaleChange _{k +1} - scale change factor for frame k+1,

ScaleChange _{k -} scale change factor for the current frame k,

max(α ₁ ,α ₅ ) - maximum value from α ₁ and α _5.

Meaning - is responsible for the horizontal stretching of the target object on the frame (shear mapping) (for example, transforming a square into a parallelogram), and is considered as obviously absent.

The rotation coefficient is also calculated using the affine transformation model. The affine model considers the rotation of points on the frame around the center of the frame, for example, the rotation of a point (x, y) by the rotation angle θ in radians. So the rotation angle θ from our coordinate transformation can be calculated as follows:

Where

α ₁ ,α ₄ - the corresponding values of matrix A.

- Determine the distance d ₂ from the camera to the target object based on the calculated zoom factor; on the current frame k it is calculated as follows:

Where - initial distance from the camera to the target object when switching to close shooting mode

It should be noted that the transition from the long-range shooting mode to the short-range shooting mode is carried out upon reaching a certain threshold K, which is pre-determined depending on the type of camera used; as an example, the threshold value was chosen by the authors of the invention to be 30 cm, this distance is the initial distance for close shooting mode, but the indicated value is given by the authors only as an example and may vary.

-Determine the measure of confidence R ₂ of the distance d ₂ from the camera to the target object.

For example, the confidence measure _R2 can be calculated as follows:

where N is the total number of motion vectors (mv _{i, j} ), length(mv _{i, j} ) is the length of the vector mv _{i, j} , avg_magnitude_bound - manually set as a parameter for the average value of the length of vectors from the vector field, R ₂ _threshold - manually is specified as a parameter for the minimum acceptable measure of confidence. Thus, abnormally long vectors lead to a low confidence measure, which is consistent with our subtask of frame-by-frame estimation of object scale changes.

- Determine the distance d ₃ from the camera to the target object based on depth data using depth sensors,

- Determine a measure of confidence R ₃ distance d ₃ based on depth data using depth sensors,

Since depth sensors and motion sensors are used to calculate the confidence measures R ₃ and R ₄ for distances d ₃ and d ₄ , respectively, their readings can be assessed directly as signal/noise using expression (1) presented earlier.

In particular, the assessment of the confidence measure R ₃ can be performed in the manner described in the following sources:

[3] J. Zhu, L. Wang, R. Yang, J. E. Davis and Z. pan, "Reliability Fusion of Time-of-Flight Depth and Stereo Geometry for High Quality Depth Maps," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 7, pp. 1400-1414, July 2011, doi: 10.1109/TPAMI.2010.172.

- Determine the distance d ₄ from the camera to the target object based on data from motion sensors, according to expression (1). Determine the measure of confidence R ₄ of the distance d ₄ based on data from motion sensors, as disclosed in the publications:

[14] Terra, R., Figueiredo, L., Barbosa, R. S., & Anacleto, R. (2014). Traveled Distance Estimation Algorithm for Indoor Localization. Procedia Technology, 17, 248-255.

[15] Shi, Yonglei & Fang, Liqing & Guo, Deqing & Qi, Ziyuan & Wang, Jinye & Che, Jinli. (2021). Research on distance measurement method based on micro-accelerometer. AIP Advances. 11.055126.10.1063/5.0054463.

The assessment of the confidence measure _R4 can be performed in the manner described in the following sources (the sensors are combined under the name IMU - Inertial Measurement Unit - can be calculated through RMSE), as disclosed in the publication:

[16] Cho YS, Jang SH, Cho JS, Kim MJ, Lee HD, Lee SY, Moon SB. Evaluation of Validity and Reliability of Inertial Measurement Unit-Based Gait Analysis Systems. Ann Rehabil Med. 2018 Dec;42(6):872-883. doi: 10.5535/arm.2018.42.6.872. publ. 2018 Dec 28. PMID: 30613081; PMCID: PMC6325313.

- Initialize the tracker for the target object only on the first frame (initial) by means of a bounding rectangle on the camera screen and track the target object on subsequent frames, and calculate the scaling factor of the bounding rectangle with the target object located in it,

-The distance d ₅ from the camera to the target object is determined based on the calculated scaling factor of the bounding rectangle with the target object located in it when the tracker is initialized,

-Determine the measure of confidence R ₅ of the distance d ₅ from the camera to the target object based on the calculated scaling factor of the bounding rectangle with the target object located in it when initializing the tracker

For example, the confidence measure R ₅ is defined as:

On each frame, the tracker first defines a set of key points to track, for optimization purposes - in our case - it is evenly distributed across the bounding box in the form of 2D mesh nodes.

An example of a bounding box in the form of 2D grid nodes is presented in Fig. 9.

The tracker then uses optical flow to track each point. Thus, a set of predicted vectors is obtained. The final result (scale change, displacement) is obtained through a “voting” procedure taking into account each vector.

It should be noted that in more detail, methods and techniques for obtaining a measure of confidence when initializing a tracker are disclosed in the publication:

[17] Z. Kalal, K. Mikolajczyk and J. Matas, “Forward-Backward Error: Automatic Detection of Tracking Failures,” 2010 20th International Conference on Pattern Recognition, Istanbul, Turkey, 2010, pp. 2756-2759, doi: 10.1109/ICPR.2010.675.

Further, this description provides only a brief description of the parameters for obtaining expression (13), which, as already indicated, are disclosed in detail in the publication [17]:

Where: - normalized cross-correlation coefficients. This is a measure of the similarity of image fragments. For each vector, it is calculated for a pair of fragments (corresponding to the bounding box on the frame, see Fig.9) from image i and image i+1 with a shifted center according to the vector . The similarity measure value is calculated for each pair of fragments obtained from one of the motion vectors . The median of all values of the similarity measure of image fragments is taken as the final estimate.

- forward-backward tracking error: a tracker is applied for each key point when moving from frame k to frame k+1 (forward tracking). To calculate the error, you must also apply the tracker to the image frames in reverse order, using the points obtained from forward tracking as key starting points. As a result, the distance between the forward tracking starting point and the backward tracking ending point determines the error. The final value is calculated based on the median of all error values for each key point

“good_point_fraction” - the fraction of the original key points after filtering. Filtering occurs as follows: the above values And are counted for each key point, those points are filtered (not taken into account in the share) at which the count gave values below the threshold (threshold values are known in advance and are set manually as a result of preliminary experiments). The filtering process is described in detail in the publication [17]).

The symbol “ ̂ ”(hat) above each expression means a normalized value in the range [0,1]. It is obtained each time by comparison with a threshold value (manually configured). Thus, it is also guaranteed that R ₅ is in the range [0,1]. And if the tracker loses the object, the value of the confidence measure R ₅ is set to 0 until the tracker is reinitialized (see publication [17]).

Next, a hybrid estimation of the distance from the camera to the target object in the current frame is performedk based on the obtained distance values d₁, d₂,d₃,d₄,d₅ from camera to target and confidence measures R₁, R₂, R₃, R₄, R₅, and the adjusted distance d_hfrom the camera to the target object in the current frame. In this case, the corrected distance dh is calculated from the expression:

d _h= d ₁ * R ₁ +d ₂ * R ₂ +d ₃ * R ₃ +d ₄ * R ₄ +d ₅ * R ₅ (14);

- decide to switch the camera to close-up mode if the resulting corrected distance d _h is less than a predetermined threshold value K (for example, 30 cm, but the value may be different). This threshold value depends on the specific parameters of the camera used when capturing the image, i.e. at what distances each mode (near, far) can focus.

Operating steps of the camera in the close-up mode (ultra-wide-angle mode) will be described with reference to FIG. 7B.

When you switch the camera to close-up mode, the following steps occur.

When capturing an image of a scene with a target object in the camera's close-in mode, data is obtained for the current frame, i.e. The data of the current image (picture) of the k-th frame is provided. Optional: provide data acquisition from depth sensors and/or acquisition of data from motion sensors, for example, from an accelerometer, gyroscope, magnetometer.

The tracking function is initialized and first finds and then tracks the target object, which, when initializing tracking, is highlighted by a bounding rectangle through the autofocus tracker or through the central region of the image, the basic principles of operation of the tracker are disclosed in detail in the patent: [7] RU2778355C2, publ. 08/17/2022, the same patent holder (SAMSUNG ELECTRONICS CO., LTD., KR) as the patent holder of the claimed invention.

Finding the target object can be done automatically or directly by the user by pressing a finger on the screen in the area of the target object. Methods for finding and tracking an object when shooting with a camera are widely known in the art:

[18] Zhang, X. Q., Jiang, R. H., Fan, C. X. et al. Advances in Deep Learning Methods for Visual Tracking: Literature Review and Fundamentals. Int. J. Autom. Comput. 18, 311-333 (2021). https://doi.org/10.1007/s11633-020-1274-8);

[19] Soleimanitaleb, Zahra & Keyvanrad, Mohammad. (2022). Single Object Tracking: A Survey of Methods, Datasets, and Evaluation Metrics.;

[20] Luo, Wenhan & Xing, Junliang & Milan, Anton & Zhang, Xiaoqing & Liu, Wei & Zhao, Xiaowei & Kim, Tae-Kyun. (2017). Multiple Object Tracking: A Literature Review. Artificial Intelligence. 293. 10.1016/j.artint.2020.103448.

Close-in mode recognizes a special (predefined) use case: when the target object is lost in the scene captured by the camera, i.e. the target object moves in and out of the camera's field of view

At the same time, if the target object is lost in the frame, the camera switches from close shooting mode to far shooting mode.

The image data of at least two consecutive frames is compared and based on the comparison of said image data, motion vectors mv _{i, j} are obtained for each pixel or group of pixels of the scene including the target object. The process of obtaining motion vectors when shooting with a camera is described in detail in the publications:

[21] Heuer, . (1999). Global motion estimation in image sequences using robust motion vector field segmentation. 261-264. 10.1145/319463.319620;

[22] Manikandan, M. & Kumar, Vijay & Narayanadass, Ramadass. (2006). Motion estimation method for video compression - An overview. 5 pp. - 5. 10.1109/WOCN.2006.1666680.

An update is performed (every k frame, where k>1, so for each subsequent frame new motion vectors are obtained.

The central region of the calculated field of vectors mv _{i, j} is selected, to which the affine transformation model is applied and the shift, rotation, and scaling coefficients are calculated. All coefficients are calculated simultaneously. Example (x, y - coordinates, k+1 - current frame, k - previous):

The distance d ₂ from the camera to the target object is determined based on the calculated zoom factor. The process of determining the distance from the camera to the target object based on motion vectors when shooting with a camera is described in detail in the publication:

[23]Ho, Hann Woei & Croon, Guido & Chu, Q.P. (2017). Distance and velocity estimation using optical flow from a monocular camera. International Journal of Micro Air Vehicles. 9. 198-208. 10.1177/1756829317695566.

Determine the measure of confidence R ₂ of the distance d ₂ from the camera to the target object

For example, the confidence measure _R2 can be calculated as follows:

where N is the total number of motion vectors mv _{i, j,} length(mvi, j) is the length of the vector mvi, j, avg_magnitude_bound is manually set as a parameter for the average value of the length of vectors from the vector field, R2_threshold is manually set as a parameter for the minimum acceptable measure of confidence.

Or like this:

R ₂ = Error(ROI-of-image-frame _i+1 , f (ROI-of-Image-frame _i )) ,

g de f is a function that applies resizing and shifting calculated using an affine transformation.

Error is a function for calculating the error between the image f(I _k-1 ) modified by the function f on the (k-1) frame and the original image I _i on the k-th frame.

Optionally:

The distance to the target object is estimated by the tracker by measuring changes in the size of the tracker's bounding box on the camera screen. Calculate the scaling factor of the bounding rectangle containing the target object,

The distance d ₅ from the camera to the target object is determined based on the calculated scaling factor of the bounding rectangle with the target object located in it when the tracker is initialized. Estimating distance using a bounding box tracker is disclosed in the following references:

[24] Vajgl, M.; Hurtik, P.; Nejezchleba, T. Dist-YOLO: Fast Object Detection with Distance Estimation. Appl. Sci. 2022, 12, 1354. https://doi.org/10.3390/app12031354).

A measure of confidence R ₅ of the distance d ₅ from the camera to the target object is determined based on the calculated scaling factor of the bounding rectangle with the target object located in it when initializing the tracker

For example, the confidence measure R ₅ is defined as:

(13)

On each frame, the tracker first defines a set of key points to track, for optimization purposes - in our case - it is evenly distributed across the bounding box in the form of 2D mesh nodes.

An example of a bounding box in the form of 2D grid nodes is presented in Fig. 9.

The tracker then uses optical flow to track each point. Thus, a set of predicted vectors is obtained. The final result (scale change, displacement) is obtained through a “voting” procedure taking into account each vector.

It should be noted that in more detail, methods and techniques for obtaining a measure of confidence when initializing a tracker are disclosed in the publication already mentioned above:

[17] Z. Kalal, K. Mikolajczyk and J. Matas, “Forward-Backward Error: Automatic Detection of Tracking Failures,” 2010 20th International Conference on Pattern Recognition, Istanbul, Turkey, 2010, pp. 2756-2759, doi: 10.1109/ICPR.2010.675.

Further, this description provides only a brief description of the parameters for obtaining expression (13), which, as already indicated, are disclosed in detail in the publication [17]:

Where: - normalized cross-correlation coefficients. This is a measure of the similarity of image fragments. For each vector, it is calculated for a pair of fragments (corresponding to the bounding box on the frame, see Fig.9) from image i and image i+1 with a shifted center according to the vector . The similarity measure value is calculated for each pair of fragments obtained from one of the motion vectors . The median of all values of the similarity measure of image fragments is taken as the final estimate.

- forward-backward tracking error: a tracker is applied for each key point when moving from frame k to frame k+1 (forward tracking). To calculate the error, you must also apply the tracker to the image frames in reverse order, using the points obtained from forward tracking as key starting points. As a result, the distance between the forward tracking starting point and the backward tracking ending point determines the error. The final value is calculated based on the median of all error values for each key point

“good_point_fraction” - the fraction of the original key points after filtering. Filtering occurs as follows: the above values And are counted for each key point, those points are filtered (not taken into account in the share) at which the count gave values below the threshold (threshold values are known in advance and are set manually as a result of preliminary experiments). The filtering process is described in detail in the publication [17]).

The symbol “^” (hat) above each expression means a normalized value in the range [0,1]. It is obtained each time by comparison with a threshold value (manually configured). Thus, it is also guaranteed that R ₅ is in the range [0,1]. And if the tracker loses the object, the value of the confidence measure R ₅ is set to 0 until the tracker is reinitialized (see publication [17]).

Optionally:

- provide stereo data from the smartphone camera

determine the disparity D _k of the scene, including the target object, captured by the camera, based on the obtained stereo data at a certain point in time for each current frame,

- determine the measure of confidence R ₁ of the obtained disparity D _k based on the obtained stereo data,

- determine the measure of confidence R ₃ distance d ₃ based on depth data using depth sensors,

- determine the distance d ₄ from the camera to the target object based on motion sensor data,

- determine the measure of confidence R ₄ of the distance d ₄ based on data from motion sensors.

Next, a hybrid estimation of the distance from the camera to the target object in the current frame is performedk based on the obtained distance values d₁,d₂,d₃,d₄,d₅ from camera to target and confidence measures R₁,R₂,R₃,R₄,R₅, and the adjusted distance d_hfrom the camera to the target object in the current frame. In this case, the corrected distance dh is calculated from the expression:

d _h= d ₁ * R ₁ +d ₂ * R ₂ +d ₃ * R ₃ +d ₄ * R ₄ +d ₅ * R ₅ (14).

And they decide to switch the camera to long-range shooting mode if the resulting corrected distance d _h is greater than or equal to a predetermined threshold value K (for example, 35 cm, but the value may be different). This threshold value depends on the specific parameters of the camera used when capturing the image, i.e. at what distances each mode (near, far) can focus.

The value may depend on the distance values at which the Long Shot mode focuses. In this case, this value must differ from the value for switching to the close-up shooting mode, since if these values coincide, there will be constant switching from one mode to another.

Next, a complete diagram of the operation of the method for automatically switching the operating mode of the camera from receiving input data to outputting the result will be described according to the claimed invention with reference to Fig. 7A, Fig. 7B, a block diagram of the sequence of stages of the claimed method is presented.

Key Point 1: Hybrid Target Distance Estimation (with reference to Figure 7A, Figure 7B).

Input data for initializing the claimed method:

- image data (picture), for example a scene containing a target object and obtained from a camera built into a smartphone or similar device, image data includes:

- image data based on YUV or RGB model, etc., obtained for the current k-th frame from the current camera: long-range shooting mode or short-range shooting mode,

- image data obtained from previous frames in close-up mode

- stereo image data, for example, obtained from a camera using dual pixel technology or PDAF image data obtained using PDAF sensors on the device’s camera matrix, for the current frame in long-range shooting mode,

- optionally, motion data obtained, for example, using a motion sensor, accelerometer or gyroscope and depth data, obtained, for example, using a depth sensor,

- K, M - threshold values for entering and exiting the current mode, i.e. in this case, to switch the camera to short-range shooting mode and long-range shooting mode, respectively. Threshold values are calculated empirically based on the range of optimal focal lengths for the far camera lens and the near camera lens for a particular camera technology implementation. As a result of experimental studies and comfortable observation conditions for the user and based on the selection of optimal values, the following threshold values were obtained, for example, as a non-limiting example: K=30cm and M=35cm or K=29cm and M=32cm.

The presented threshold values are not limited to the values presented and are given in this disclosure only as an illustrative example.

Output data obtained as a result of executing the claimed method:

T ∈ { far shooting mode, close shooting mode } - making a decision about the need to initialize switching the camera mode.

Sequence of camera switching steps (hybrid algorithm used for camera switching)

A. Long Range Mode

1. Calculation of SAD (Sum of Absolute Differences) curves for the entire scene captured by the camera based on stereo data obtained in long-range shooting mode (SAD curve represents curves on a 2D graph: the X axis corresponds to the displacement in the number of pixels of the left image relative to the right, Y axis - the corresponding sum of pixel-by-pixel differences between images for the selected fragment; SAD curves are obtained by the authors using sensors built into a mobile phone, which are called dual pixels and contain information about two pictures (left and right image of an object from the phone with a slight shift ).

2. Calculation of disparity D _k based on the constructed SAD curves (disparity is the point of local minimum of the SAD curve).

3. From each SAD curve, the minimum values of the SAD curve are extracted: the global minimum and subminimum, as well as other points of the curve, which are used to assess the confidence measure R ₁ of the calculated disparity:

, (2)

Where - signal level, -noise level, -global minimum and -subminimum Formula (2) characterizes the approximation of the signal-to-noise ratio using the global minimum and subminimum of the SAD curve.

4. Definition of a special use case, which is covered in detail in Key Statement 2. If no special use case is identified, execution of the method continues.

5. The distance (d ₁ ) from the camera to the target object is determined based on the calculated disparity D _k for the current frame (calculation of the distance based on disparity is described in detail in [7] patent RU2778355C2).

6. (Option 1)

The image data of at least two consecutive frames is compared and based on the comparison of said image data, motion vectors mv _{i, j} are obtained for each pixel or group of pixels of the scene including the target object captured by the camera. The process of obtaining motion vectors when shooting with a camera is described in detail in the publications:

[21] Heuer, . (1999). Global motion estimation in image sequences using robust motion vector field segmentation. 261-264. 10.1145/319463.319620;

[22] Manikandan, M. & Kumar, Vijay & Narayanadass, Ramadass. (2006). Motion estimation method for video compression - An overview. 5 pp. - 5. 10.1109/WOCN.2006.1666680.

An update is performed (every k frame, where k>1, so for each subsequent frame new motion vectors are obtained.

The central region of the calculated field of vectors mv _{i, j} is selected, to which the affine transformation model, which was described in detail earlier in this description, is applied, and the shift, rotation, and scaling coefficients are calculated. All coefficients are calculated simultaneously. Example (x, y - coordinates, k+1 - current frame, k - previous) for a matrix the required coordinate transformation:

(8)

The distance d ₂ from the camera to the target object is determined based on the calculated zoom factor. The process of determining the distance from the camera to the target object based on motion vectors when shooting with the camera is described in detail in the publication

[23] Ho, Hann Woei & Croon, Guido & Chu, Q. P.. (2017). Distance and velocity estimation using optical flow from a monocular camera. International Journal of Micro Air Vehicles. 9. 198-208. 10.1177/1756829317695566.

The distance d ₂ and the confidence measure R ₂ of the distance d ₂ from the camera to the target object are determined based on the calculated zoom factor on the current frame k according to expression (11) described above.

7. (Option 2) Initialize the tracker for the target object using the bounding box on the camera screen and track the target object in subsequent frames, and calculate the scaling factor of the bounding box with the target object located in it.

The distance d ₅ from the camera to the target object is determined based on the calculated scaling factor of the bounding rectangle with the target object located in it when the tracker is initialized.

The confidence measure R ₅ of the distance d ₅ from the camera to the target object is determined based on the calculated scaling factor of the bounding rectangle with the target object located in it when initializing the tracker according to expression (13) described earlier in this description.

8. (Option 3) Determine the distance d ₃ from the camera to the target object based on depth data using depth sensors, determine the confidence level R ₃ of the distance d ₃ based on depth data using depth sensors.

9. (Option 4) The distance d ₄ from the camera to the target object is determined based on the motion sensor data, and the confidence level R ₄ of the distance d ₄ is determined based on the motion sensor data.

10. Perform a hybrid estimation of the distance from the camera to the target object in the current framek based on the obtained distance values d₁,d₂,d₃,d₄,d₅ from camera to target and confidence measures R₁,R₂,R₃,R₄,R₅, and the adjusted distance d_hfrom the camera to the target object in the current frame. In this case, the corrected distance dh is calculated from the expression:

d _h= d ₁ * R ₁ +d ₂ * R ₂ +d ₃ * R ₃ +d ₄ * R ₄ +d ₅ * R ₅ (14).

11. A decision is made to switch the camera to close-up mode if the resulting corrected distance d _h is less than a predetermined threshold value K (for example, 30 cm, but the value may be different). This threshold value depends on the specific parameters of the camera used when capturing the image, i.e. at what distances each mode (near, far) can focus.

Key Point 1 : Hybrid algorithm for estimating distance to target object (with reference to Fig. 7A, Fig. 7Bb).

Input data for initializing the claimed method:

- image data, for example scenes containing a target object and obtained from a camera built into a smartphone or similar device, image data includes:

- image data based on YUV or RGB model, etc., obtained for the current k-th frame from the current camera: long-range shooting mode or short-range shooting mode;

- image data obtained from previous frames in close shooting mode;

- stereo image data, for example, obtained from a camera using dual pixel technology or PDAF image data obtained using PDAF sensors on the device’s camera matrix, for the current frame;

- optionally, motion data obtained, for example, using a motion sensor, accelerometer or gyroscope and depth data, obtained, for example, using a depth sensor,

- K, M - threshold values for entering and exiting the current mode, i.e. in this case, to switch the camera to short-range shooting mode and long-range shooting mode, respectively. Threshold values are calculated empirically based on the range of optimal focal lengths for the far camera lens and the near camera lens for a particular camera technology implementation. As a result of experimental studies and comfortable observation conditions for the user and based on the selection of optimal values, the following threshold values were obtained, for example, as a non-limiting example: K=30cm and M=35cm or K=29cm and M=32cm. The presented threshold values are not limited to the values presented and are given in this disclosure only as an illustrative example.

Output data obtained as a result of executing the claimed method:

T ∈ { far shooting mode, close shooting mode } - deciding whether to initiate camera switching.

Sequence of camera switching stages (hybrid switching algorithm)

B. Close-in Mode

1. The tracking function is initialized and first finds and then tracks the target object, which is highlighted with a bounding box when tracking is initialized.

Finding the target object can be done automatically or directly by the user by pressing a finger on the screen in the area of the target object. Methods for finding and tracking an object when shooting with a camera are widely known in the art:

[18] Zhang, X. Q., Jiang, R. H., Fan, C. X. et al. Advances in Deep Learning Methods for Visual Tracking: Literature Review and Fundamentals. Int. J. Autom. Comput. 18, 311-333 (2021). https://doi.org/10.1007/s11633-020-1274-8;

[19] Soleimanitaleb, Zahra & Keyvanrad, Mohammad. (2022). Single Object Tracking: A Survey of Methods, Datasets, and Evaluation Metrics;

[20] Luo, Wenhan & Xing, Junliang & Milan, Anton & Zhang, Xiaoqing & Liu, Wei & Zhao, Xiaowei & Kim, Tae-Kyun. (2017). Multiple Object Tracking: A Literature Review. Artificial Intelligence. 293. 10.1016/j.artint.2020.103448.

2. Definition of a special use case, which is discussed in detail in Key Statement 2. If the special use case in which the switch to the long-range shooting mode occurs is not identified, the method continues in this case

3. The image data of at least two consecutive frames is compared and based on the comparison of the specified image data, motion vectors mv _{i, j} are obtained for each pixel or group of pixels of the scene including the target object. The process of obtaining motion vectors when shooting with a camera is described in detail in the publications:

[21] Heuer, (1999). Global motion estimation in image sequences using robust motion vector field segmentation. 261-264. 10.1145/319463.319620;

[22] Manikandan, M. & Kumar, Vijay & Narayanadass, Ramadass. (2006). Motion estimation method for video compression - An overview. 5 pp. - 5. 10.1109/WOCN.2006.1666680.

4. Perform an update every frame k, where k>1, thus obtaining new motion vectors for each subsequent frame.

5. The central region of the calculated field of vectors mv _{i, j} is selected, to which the affine transformation model, which was described in detail earlier in this description, is applied, and the shift, rotation, and scaling coefficients are calculated. All coefficients are calculated simultaneously. Example (x, y - coordinates, k+1 - current frame, k - previous frame) for a matrix the required coordinate transformation:

(8)

6. Determine the distance d ₂ from the camera to the target object based on the calculated zoom factor. The process of determining the distance from the camera to the target object based on motion vectors when shooting with a camera is described in detail in the publication:

[23] Ho, Hann Woei & Croon, Guido & Chu, Q. P.. (2017). Distance and velocity estimation using optical flow from a monocular camera. International Journal of Micro Air Vehicles. 9. 198-208. 10.1177/1756829317695566.

7. Determine the distance d ₂ and the measure of confidence R ₂ of the distance d ₂ from the camera to the target object based on the calculated zoom factor on the current frame k according to expression (11) described above.

8. (Option 1) The distance d ₅ from the camera to the target object is determined based on the calculated scaling factor of the bounding rectangle with the target object located in it when initializing the tracker according to expression (13) described earlier in this description.

9. (Option 2) Calculate SAD (Sum of Absolute Differences) curves for the entire scene captured by the camera based on the stereo data obtained (SAD curve represents curves on a 2D graph: the X axis corresponds to the offset in the number of pixels of the left image relative to the right, Y axis - the corresponding sum of pixel-by-pixel differences between images for the selected fragment; SAD curves are obtained by the authors using sensors built into a mobile phone, which are called dual pixels and contain information about two pictures (left and right image of an object from a phone, smartphone or similar device with a slight shift).

Calculation of disparity D _k based on constructed SAD curves (disparity is the point of local minimum of the SAD curve).

From each SAD curve, the minimum values of the SAD curve are extracted: the global minimum and subminimum, as well as other points of the curve, which are used to assess the confidence measure R ₁ of the calculated disparity:

(2)

Where - signal level, -noise level, -global minimum and -subminimum Formula (1) characterizes the approximation of the signal-to-noise ratio using the global minimum and subminimum of the SAD curve

The distance (d ₁ ) from the camera to the target object is determined based on the calculated disparity D _k for the current frame (calculation of the distance based on disparity is described in detail in patent [7] RU2778355C2).

10. (Option 3) Determine the distance d ₃ from the camera to the target object based on depth data using depth sensors, determine the degree of confidence R ₃ of the distance d ₃ based on depth data using depth sensors.

11. (Option 4) The distance d ₄ from the camera to the target object is determined based on the motion sensor data, and the confidence level R ₄ of the distance d ₄ is determined based on the motion sensor data.

12. Perform a hybrid estimation of the distance from the camera to the target object in the current framek based on the obtained values of distances d₁,d₂,d₃,d₄,d₅ from camera to target and confidence measures R₁,R₂,R₃,R₄,R₅, and the adjusted distance d_hfrom the camera to the target object in the current frame. In this case, the corrected distance d_h calculated from the expression:

d _h= d ₁ * R ₁ +d ₂ * R ₂ +d ₃ * R ₃ +d ₄ * R ₄ +d ₅ * R ₅ (14).

13. A decision is made to switch the camera from close shooting mode to long shooting mode if the resulting corrected distance d _h is greater than a predetermined threshold value M (for example, 35 cm, but the value may be different). This threshold value depends on the specific parameters of the camera used when capturing the image, i.e. at what distances each mode (near, far) can focus.

Key position 2 (with reference to Fig. 7A, Fig. 7B)

Handling a special case (predefined case) of use. In the step of determining the special use case of Key Position 1 for the long-range shooting mode of the camera operation, the special use case is processed.

Input data:

Image I _k on frame k

A measure of confidence R ₁ and disparity D _k calculated at the stages of Key Point 1.

Optionally, the output of the tracker (the bounding box and the presence or absence of the target object at frame k).

Optionally, motion vectors (GMV data) obtained during Key Point 1 steps.

Optionally, motion data obtained, for example, using a motion sensor, accelerometer or gyroscope, and depth data, obtained, for example, using a depth sensor.

Output:

Ui-i case defined from predefined U use cases.

Sequence of stages for processing a special use case (processing algorithm)

A. Long-range shooting mode (wide-angle mode) (see Fig. 7A)

1. Calculate the function f _R (R ₁ , D _k , I _k ), which depends on the data: confidence measure R ₁ , disparity D _k , image I _k c of the k-th frame obtained at the stages of Key Position 1 . Optionally, it can also depend on data: the output of the tracker (bounding box and the presence or absence of the target object in frame k), motion vectors (GMV data) obtained during the stages of Key Statement 1, motion data obtained from e.g. using a motion sensor, accelerometer or gyroscope and depth data obtained, for example, using a depth sensor.

In a simple case, the function f _R can be equal to R ₁ .

f _R (R ₁ , D _k , I _k )=R ₁

The result of the function f _R (R ₁ , D _k , I _k ) is compared with the threshold Rt.

A) if the result of the function f _R (R ₁ , D _k , I _k ) exceeds the threshold, i.e. f _R (R ₁ , D _k , I _k )) > R _t , a standard use case is concluded,

B) if the result of the function f _R (R ₁ , D _k , I _k ) does not exceed the threshold, i.e. f _R (R ₁ , D _k , I _k ) ≤ R _t , a conclusion is made about a special use case, namely the case of a textureless object on a plain background or a plain background. For example, shooting a plain wall without a pattern or a table without objects. The Rt threshold is selected based on experimental data. For example, if the result of the function f _R (R ₁ , D _k , I _k ) measures from 0 to 1, the value of Rt could be 0.8.

2. Optionally, in addition to the case of a plain background, the case of low illumination is also recognized by means of another function f _R1 (R ₁ , D _k , I _k ) from the same initial data.

It should be noted that using hybrid distance estimation solves the problem of distance inaccuracy in cases such as: the target object is a transparent object, the target object is a reflective object, shooting a target object behind glass, low light conditions or a white wall effect when the camera is looking at homogeneous background, local movement of the object with a stationary background, camera rotation or shake.

3. The transition continues to the stages of Key Statement 1 according to Figs. 7A, 7B.

Key position 2 (with reference to Fig.7a, Fig.7b)

Handling a special (predefined) use case. In the step of determining the special use case of Key Position 1 (see FIG. 7B) for the close shooting mode of the camera operation, the special use case is processed.

Input data:

Image I _k on frame k

Reliability R ₅ obtained at the stages of Key Statement 1

Tracker output (bounding box and presence or absence of target object at frame N)

Optionally, motion vectors (GMV data) obtained during Key Point 1 steps.

Optionally, motion data obtained, for example, using a motion sensor, accelerometer or gyroscope, and depth data, obtained, for example, using a depth sensor.

Output:

Ui-i case defined from predefined U use cases.

Sequence of stages for processing a special use case (processing algorithm)

B. Close-in mode (Ultra Wide Angle mode) (see Fig. 7B).

1. Initiate the target object tracking process using the bounding box function for the target object's outline.

Determining the initialization bounding box in object tracking procedures is widely known in the art, and can be accomplished by: image segmentation, manual selection of a given target object by the user, initiation at the center of the bounding box, and is disclosed in the following references:

[25] He, Mengqi & Zhang, Jing & Yu, Wenxin. (2022). Salient Object Detection via Bounding-box Supervision.

2. Optionally, the use of the automatic tracking function can be performed after converting the image (image of the object): from the image in the far shooting mode to the image in the near shooting mode, namely the bounding box in the long shooting mode (obtained by the tracker available in the long shooting mode) , is converted to a bounding box in close mode by a function that converts a far image into a close image. An example of automatic object tracking when taking an image is disclosed in detail in [7] patent RU2778355C2, publ. 08/17/2022, the same patent holder (SAMSUNG ELECTRONICS CO., LTD., KR) as the patent holder of the claimed invention.

3. Optionally, constructing a field of motion vectors and updating the field of motion vectors (GMV data) and tracking data every frame k, where k>1, thus obtaining new motion vectors for each subsequent frame.

4. Check if the target object is lost during tracking:

The tracker tracks the object. If the bounding box extends completely outside the frame, the target object is set to be lost in the current frame. In case of its loss:

Optionally, scene changes are checked using a field of motion vectors.

Optionally, the reliability R ₅ obtained at the stages of Key Point 1 can be used to confirm or refute the loss of an object.

Optionally, after the loss of an object, an additional check of the hybrid estimate d _h obtained at the stages of Key Point 1 can be carried out, if at least one of the estimates d ₁ and d _{3 was used in its calculation.}

When the case of loss of an object is confirmed, it switches to the long-range shooting mode.

5. Optionally, motion data can be used to determine when the camera is rotated relative to the object.

6. The transition continues to the stages of Key Statement 1 according to Figs. 7A, 7B.

The claimed invention provides the use of a device and method for switching camera operating modes, taking into account the following functionality:

1) The camera's switching process is smooth and precise, and is consistent throughout the switching process because different information from many different sensors is used to make the switching decision.

2) The camera switching process is protected from unwanted camera switches and image jitter or flickering because the invention takes into account appropriate thresholds for switching the camera to the far shooting mode and the close shooting mode.

3) The method and device for automatically switching camera operating modes ensure stable camera switching in predetermined use cases, such as the target object leaving the frame, low light/white wall effect when the camera looks at a uniform background, local object movements with a stationary background , rotation or camera shake.

4) The decision to switch the camera mode in the claimed invention is made in real time thanks to the expanded use of various sources of information and highly optimized processes for processing the received data.

5) The solutions implemented in the claimed method and device for switching camera operating modes do not require large computing resources to operate on a mobile phone.

Next, with reference to Fig. 10, a device for automatically switching the camera operating mode will be described, configured to implement the steps of the method, and contains:

- an input/output unit 1 including: a graphics processor 2, a display 3, a camera 4, an input/output interface unit 5 including an input interface and an output interface (not shown), and

- computing unit 6 containing the central processor 7, which consists of parts of a chipset (system on a chip, SoC), for example Qualcomm Snapdragon, Nvidia Orin, Intel Quark, Samsung Exynos, Apple Silicon, as well as a memory controller 8, system memory 9 and storage 10.

In this case, camera 4 represents a multi-camera system, including a photo and/or video camera, containing an image forming unit, including lenses for close shooting and long-range shooting of the camera, a sensor matrix, which is a matrix of PDAF sensors, providing stereo data when the camera is operating, while the camera is made with the ability to capture a target object and/or target scene and transfer image data of the target object and/or target scene to the system memory of the computing unit through the input-output interface unit 5. An example of a camera used in this solution is described in detail in the following publications:

[26] A. S. Olagoke, H. Ibrahim and S. S. Teoh, “Literature Survey on Multi-Camera System and Its Application,” in IEEE Access, vol. 8, pp. 172892-172922, 2020, doi: 10.1109/ACCESS.2020.3024568.)

[27] -US20170082823A1, IPC G02B27/64, publ. 03/23/2017.

The calculation block 6 is configured to calculate the distance to the target object and/or target scene and select between the near and far lens of the camera 4, while the resulting distance and the corresponding lens type (wide-angle or ultra-wide-angle lens) are transferred to the system memory 9 and through the block 5 input/output interface to camera 4 for automatically switching operating modes of camera 4.

The camera 4 is configured to transmit graphic information through the input-output interface unit 5 and the graphics processor 2 to the display 3 of the device.

The display 3 of the device is configured to visualize the target object and/or target scene in real time.

The input/output block 1 additionally contains motion sensors 11 (accelerometer and/or gyroscope and/or magnetometer) and depth sensors (RaDAR and/or LiDAR and/or ToF sensor and/or UWB sensor).

The graphics processor 2 is configured to process graphic information received from the camera 4 of the device.

The drive 10 is configured to store photo and video images obtained by the camera 4 of the device and may be a solid-state drive (SSD).

The device for automatically switching camera operating modes according to the invention is designed to be built into one of: a smartphone, a tablet computer, a photo and video camera.

Industrial applicability

The device and method for automatically switching camera operating modes can be used for photo and video shooting with smartphone cameras, as well as in tablet computers running Android, iOS, HarmonyOS and similar operating systems, as well as in other similar devices that require automatic switching camera operating modes.

Claims (76)

1. A method for automatically switching camera operating modes, containing the following steps: - preliminarily capture an image of the scene, including the target object to be tracked, in one of the modes: near mode, long-range shooting mode, - optionally, provide data from at least one of: depth sensors, motion sensors, - provide receipt of at least one of: stereo data, image data in at least two consecutive frames in one of the modes: close shooting mode, long shooting mode, when receiving stereo data: - determine the disparity D _k of the scene, including the target object, captured by the camera, based on the obtained stereo data at a certain point in time for each current frame, - determine the measure of confidence R ₁ of the obtained disparity D _k based on the obtained stereo data, - determine the distance d ₁ from the camera to the target object based on a certain disparity D _k for the current frame, when receiving image data: - comparing image data of at least two consecutive frames and, based on the comparison of said image data, calculating a field of motion vectors in the scene including the target object captured by the camera, - calculating at least one of: a shift coefficient, a rotation coefficient, a scene scaling factor based on the calculated field of scene motion vectors, - optionally, initialize the tracker for the target object for the current frame, track the target object on subsequent frames and calculate the scaling factor of the bounding rectangle of the camera with the target object located in it in the tracking process for each frame, - determine the exit of the target object from the frame and in the event that the exit of the target object from the frame is not detected: - determine the distance d ₂ from the camera to the target object based on the scaling factor obtained from the calculated field of motion vectors, - determine the measure of confidence R ₂ of the distance d ₂ from the camera to the target object, - when receiving data from depth sensors, the distance d ₃ from the camera to the target object is determined based on the depth data using depth sensors and the confidence level R ₃ of the distance d ₃ is determined based on the depth data using depth sensors, - when receiving data from motion sensors, the distance d ₄ from the camera to the target object is determined based on the motion sensor data and the confidence level R ₄ of the distance d ₄ is determined based on the motion sensor data, - upon initialization, the tracker determines the distance d ₅ from the camera to the target object based on the calculated scaling factor of the bounding rectangle with the target object located in it, - upon initialization, the tracking means determine the confidence measure R ₅ of the distance d ₅ from the camera to the target object based on the calculated scaling factor of the bounding rectangle with the target object located in it and - perform a hybrid assessment of the distance from the camera to the target object in the current frame based on the obtained distance values d₁,d₂,d₃,d₄, d₅ from camera to target and confidence measures R₁, R₂, R₃, R₄, R₅ and the adjusted distance d_hfrom the camera to the target object in the current frame, - decide whether to switch the camera to the close shooting mode or to the long shooting mode based on the adjusted distance d _h . 2. The method according to claim 1, in which when the camera is in the long-range shooting mode, if the resulting adjusted distance d _h is less than a predetermined threshold value, a decision is made to switch the camera to the close-range shooting mode. 3. The method according to claim 1, in which when the camera is in the short-range shooting mode, if the resulting adjusted distance d _h is greater than a predetermined threshold value, a decision is made to switch the camera to the long-range shooting mode. 4. The method of claim 1, wherein the image data is one of RGB, YUV and CMYK data. 5. The method according to claim 1, in which stereo data is obtained using PDAF sensors of a camera matrix with Dual Pixel technology. 6. The method according to claim 1, in which the disparity value D _k for the selected target object in the current frame is determined by PDAF data obtained by the camera sensor matrix as the local minimum of the SAD curve. 7. The method according to claim 6, in which to determine the disparity D _k for the selected target object in the current frame based on the received PDAF data, representing the right and left images of the target object obtained by the camera sensor matrix, the sum of absolute differences is calculated as the absolute difference between the pixels of the left and right images, on the basis of which SAD curves are constructed. 8. The method according to claim 1, in which the corrected distance d _h is determined based on the obtained values of distances d ₁ , d ₂ , d ₃ , d ₄ , d ₅ from the camera to the target object and confidence measures R ₁ , R ₂ , R ₃ , R ₄ , R ₅ for the current frame from the expression according to the expression d _h= d ₁ * R ₁ +d ₂ * R ₂ +d ₃ * R ₃ +d ₄ * R ₄ +d ₅ * R ₅ . 9. Method according to one of paragraphs. 1, 5-8, in which the measure of confidence is R₁for the current frame calculated based on SAD curve data. 10. The method according to claim 9, in which the confidence measure R ₁ for the current frame is determined from the expression R ₁ , Where - signal level, - noise level, - global minimum and - subminimum of the SAD curve. 11. Method according to one of paragraphs. 1, 5-8, in which the measure of confidence is R₂for the current frame is calculated based on the vector field data and the subsequent affine transformation of the vector field. 12. Method according to one of paragraphs. 1, 5-8, in which the measure of confidence is R₃for the current frame characterizes the approximation of the signal-to-noise ratio obtained based on depth sensor data. 13. The method according to claim 12, in which the confidence measure R₃ for the current frame determined from the expression R ₃ =100x , where R ₃ is a measure of confidence, Y _S is the signal level, Y _N is the noise level. 14. Method according to one of paragraphs. 1, 5-8, in which the measure of confidence is R₄for the current frame characterizes the approximation of the signal-to-noise ratio obtained based on data from at least one motion sensor. 15. The method according to claim 14, in which the confidence measure R₄ for the current frame determined from the expression R ₄ =100x , where R is a measure of confidence, Y _S is the signal level, Y _N is the noise level. 16. Method according to one of paragraphs. 1, 5-8, in which the confidence measure R ₅ is calculated based on the results of the tracker. 17. The method according to claim 1, in which the distance d ₁ from the camera to the target object is determined based on a certain disparity D _k for the current frame. 18. The method according to claim 1, in which the distance d ₂ from the camera to the target object is determined based on a certain zoom factor for the current frame. 19. The method according to claim 1, in which the distance d ₃ from the camera to the target object is determined based on depth data using depth sensors. 20. The method according to claim 1, in which the distance d ₄ from the camera to the target object is determined based on motion data using motion sensors. 21. The method according to claim 1, in which the distance d ₅ from the camera to the target object is determined based on the scaling factor of the bounding rectangle with the target object located in it. 22. The method according to claim 21, in which the scaling factor of the bounding rectangle on the camera screen with the target object located in it is calculated for the current frame k as the ratio of the sizes of the bounding rectangle of the current frame and the previous frame. 23. The method of claim 1, further optionally defining predetermined cases of camera operation when capturing a target object in the current frame, wherein said cases include at least one of: the target object is a transparent object, the target object represents a reflective object, shooting a target object behind glass, low light or white wall effect when the camera is looking at a uniform background, local subject movement with a stationary background, camera rotation or shake. 24. The method according to one of claims 1 to 23, in which when the camera is operating in the long-range shooting mode or in the close-range shooting mode, the calculation of at least one of: the shift coefficient, the rotation coefficient, the scene scaling coefficient is performed based on an affine transformation of the calculated field of vectors stage movements. 25. The method according to one of claims 1 to 23, in which when the camera is operating in long-range shooting mode, the confidence measure R ₂ of the distance d ₂ from the camera to the target object is determined based on the calculated field of scene motion vectors and the affine transformation of this vector field. 26. The method according to one of claims 1 to 25, in which when the camera is operating in close shooting mode, when the tracker is initialized, if the bounding rectangle including the target object extends beyond the boundaries of the current frame, the loss of the target object in the current frame is established. 27. The method according to claim 26, in which when it is determined that the target object is lost in the current frame, a decision is made to switch the camera to the long-range shooting mode. 28. The method according to claim 27, wherein the distance d ₁ from the camera to the target object is further determined based on the determined disparity D _k for the current frame after determining the loss of the target object in the current frame. 29. The method according to claim 26 or 27, in which the distance d ₃ from the camera to the target object is additionally determined based on depth data using depth sensors for the current frame after determining the loss of the target object in the current frame. 30. The method according to one of claims 1-29, in which when the camera is operating in close-up mode, the distance d _h is additionally determined after determining the loss of the target object in the current frame, taking into account the additionally determined distances d ₁ and/or d ₃ . 31. The method according to claim 30, in which, if the obtained distance d _h after determining the loss of the target object in the frame is less than a predetermined threshold value, a decision is made to maintain the close shooting mode and the transition to the next frame is performed. 32. The method according to one of claims 1 and 30, in which, if the obtained distance d _h after determining the loss of the target object in the frame is greater than or equal to a predetermined threshold value, a decision is made to switch the camera to the long-range shooting mode. 33. The method of claim 1, wherein at least one of a shift factor, a rotation factor, and a scene scaling factor is calculated by applying an affine transformation model to the calculated field of vectors. 34. A device for automatically switching camera operating modes, containing: - an input/output unit, including a graphics processor, a display, a camera, an input/output interface unit, - a computing unit containing a central processor, memory controller, system memory and storage, wherein the camera is a multi-camera system, including a photo and/or video camera and containing an image forming unit containing a short-range lens and a long-range lens, and a sensor matrix, wherein the camera is configured to capture a target object and transmit image data of the target object to system memory of the calculation unit through the input-output interface unit, the calculation block is configured to implement the stages of the method according to one of claims. 1-33, calculating the distance to the target object and making a decision to switch the camera to the close shooting mode or to the far shooting mode, while the resulting distance to the target object and the decision to switch the camera are transmitted to the system memory and through the input-output interface block to the camera to switch camera mode, the camera is configured to transmit graphic information through the input-output interface unit and the graphics processor to the device display, the display is designed to visualize the target object in real time. 35. The device according to claim 34, wherein the graphics processor is configured to process graphic information received from the camera of the device. 36. The device of claim 34, wherein the sensor array is a PDAF sensor array. 37. The device according to claim 34, in which the drive is configured to store photo and video images received by the device’s camera. 38. The device of claim 37, wherein the storage device is a solid state drive (SSD). 39. The device of claim 34, wherein the input/output unit further comprises at least one motion sensor and a depth sensor. 40. The device according to claim 39, wherein the at least one motion sensor is at least one of: an accelerometer, a gyroscope, a magnetometer. 41. The device according to claim 34, wherein the at least one depth sensor is at least one of: RaDAR, LiDAR, ToF sensor, UWB sensor. 42. The apparatus of claim 34, wherein the memory controller is configured to transfer data from the system memory to the central processing unit for computation. 43. The device according to one of paragraphs 34-42, designed to be built into one of: a smartphone, a tablet computer, a photo and video camera.