WO2020238569A1

WO2020238569A1 - Control method and control device for terminal, terminal, and computer readable storage medium

Info

Publication number: WO2020238569A1
Application number: PCT/CN2020/088888
Authority: WO
Inventors: 王路; 吕向楠
Original assignee: Oppo广东移动通信有限公司
Priority date: 2019-05-30
Filing date: 2020-05-07
Publication date: 2020-12-03
Also published as: CN110198409A; CN110198409B

Abstract

A control method for a terminal (10). The terminal (10) comprises a depth camera (11) comprising a light emitter (111) and a light receiver (112). The control method comprises: controlling a light emitter (111) to emit a preset number of frames of testing laser light to a current scene; controlling a light receiver (112) to receive the testing laser light reflected by the current scene; acquiring depth information of the current scene according to the received testing laser light; determining whether the depth information contains a depth value less than a preset safe distance; and if so, controlling the terminal (10) to enter a safe mode. Also disclosed are a control device (20) for a terminal (10), a terminal (10), and a computer readable storage medium (200).

Description

Terminal control method and control device, terminal and computer readable storage medium

Priority information

This application requests the priority and rights of the patent application with patent application number 201910465376.2 filed with the State Intellectual Property Office of China on May 30, 2019, and the full text is incorporated herein by reference.

Technical field

This application relates to the field of three-dimensional imaging technology, and more specifically, to a terminal control method, a terminal control device, a terminal, and a computer-readable storage medium.

Background technique

The depth camera can further obtain the depth information of the objects in the scene by projecting laser light into the scene and receiving the laser light reflected by the objects in the scene. Normally, the power and specifications of the laser are designed in accordance with safety standards. And the safety margin of the skin is relatively abundant.

Summary of the invention

The embodiments of the present application provide a terminal control method, a terminal control device, a terminal, and a computer-readable storage medium.

The control method of the terminal of the embodiment of the present application is used for a terminal, the terminal includes a depth camera, the depth camera includes a light transmitter and a light receiver, and the control method includes: controlling the light transmitter to emit a predetermined light to the current scene Frame number of test laser; control the light receiver to receive the test laser reflected by the current scene; obtain the depth information of the current scene according to the received test laser; determine whether there is a depth less than the preset safety distance in the depth information ; And if yes, control the terminal to enter a safe mode.

The control device of the terminal of the embodiment of this application is used in a terminal, the terminal includes a depth camera, the depth camera includes a light transmitter and a light receiver, and the control device includes a first control module, a second control module, and an acquisition module , A first judgment module and a third control module, the first control module is used to control the light transmitter to emit a predetermined number of test lasers to the current scene; the second control module is used to control the light receiver Receive the test laser reflected by the current scene; the acquisition module is used to acquire the depth information of the current scene according to the received test laser; the first determination module is used to determine whether the depth information is less than a preset safety distance The depth; the third control module is used to control the terminal to enter a safe mode if there is a depth less than a preset safe distance in the depth information.

The terminal of the embodiment of the present application includes a depth camera and a processor, the depth camera includes a light transmitter and a light receiver, and the processor is configured to: control the light transmitter to emit a predetermined number of test lasers to the current scene; control The light receiver receives the test laser reflected by the current scene; obtains the depth information of the current scene according to the received test laser; determines whether there is a depth less than a preset safety distance in the depth information; and if so, controls the The terminal enters the safe mode.

One or more non-volatile computer-readable storage media containing computer-readable instructions in the embodiment of the present application, which when executed by a processor, cause the processor to execute the control method of the embodiment of the present application .

In the terminal control method, the terminal control device, the terminal, and the computer-readable storage medium of the embodiments of the present application, the optical transmitter is controlled to emit the test laser, and the optical receiver receives the reflected test laser, and the reflected test laser First obtain the depth information, and judge whether there is a depth less than the preset safe distance based on the depth information. If it does, it is judged that if the current laser is irradiated on the user's body, such as the eyes, it will easily cause harm to the user, and further control the terminal entry Safe mode, so that when users are closer, the security of using the terminal is also higher.

The additional aspects and advantages of the embodiments of the present application will be partly given in the following description, and part of them will become obvious from the following description, or be understood through the practice of the embodiments of the present application.

Description of the drawings

The above and/or additional aspects and advantages of the present application will become obvious and easy to understand from the description of the embodiments in conjunction with the following drawings, in which:

FIG. 1 is a schematic structural diagram of a terminal according to an embodiment of the present application;

FIG. 2 is a schematic diagram of a system architecture of a terminal according to an embodiment of the present application;

FIG. 3 is a schematic flowchart of a terminal control method according to an embodiment of the present application;

4 is a schematic diagram of modules of a terminal control device according to an embodiment of the present application;

FIG. 5 is a schematic diagram of laser pulses emitted by a terminal according to an embodiment of the present application;

6 is a schematic flowchart of a terminal control method according to an embodiment of the present application;

FIG. 7 is a schematic flowchart of a terminal control method according to an embodiment of the present application;

FIG. 8 is a schematic diagram of modules of a terminal control device according to an embodiment of the present application;

FIG. 9 is a schematic diagram of a scene of depth information acquired by a terminal according to an embodiment of the present application;

FIG. 10 is a schematic flowchart of a terminal control method according to an embodiment of the present application;

11 is a schematic diagram of modules of a terminal control device according to an embodiment of the present application;

FIG. 12 is a schematic diagram of a scene of a terminal control method according to an embodiment of the present application;

FIG. 13 is a schematic flowchart of a terminal control method according to an embodiment of the present application;

FIG. 14 is a schematic diagram of modules of a terminal control device according to an embodiment of the present application;

15 is a schematic diagram of the principle of acquiring depth information in a set mode by a terminal according to an embodiment of the present application;

FIG. 16 is a schematic diagram of interaction between a non-volatile computer-readable storage medium and a processor in an embodiment of the present application.

Detailed ways

The implementation of the present application will be further described below in conjunction with the drawings. The same or similar reference numerals in the drawings indicate the same or similar elements or elements with the same or similar functions throughout.

In addition, the implementation manners of the application described below in conjunction with the drawings are exemplary, and are only used to explain the implementation manners of the application, and cannot be understood as a limitation of the application.

In this application, unless expressly stipulated and defined otherwise, the “on” or “under” of the first feature on the second feature may be in direct contact with the first and second features, or indirectly through an intermediary. contact. Moreover, the "above", "above" and "above" of the first feature on the second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the level of the first feature is higher than the second feature. The “below”, “below” and “below” of the second feature of the first feature may mean that the first feature is directly below or obliquely below the second feature, or it simply means that the level of the first feature is smaller than the second feature.

Please refer to FIG. 1, the terminal 10 of the embodiment of the present application includes a housing 15, a depth camera 11 and a processor 12. The terminal 10 may be a terminal such as a mobile phone, a tablet computer, a notebook computer, a smart watch, etc. The description of this application takes the terminal 10 as a mobile phone as an example for description. It is understood that the specific form of the terminal 10 is not limited to a mobile phone.

Both the depth camera 11 and the processor 12 can be installed on the housing 15. The housing 15 includes a front 151 and a back 152, and the front 151 and the back 152 are opposite to each other. The front 151 can also be used to install a display screen 14, which can be used to display images, text and other information. The depth camera 11 can be installed on the front 151 to facilitate selfies or video calls, etc.; the depth camera 11 can also be installed on the back 152 to facilitate shooting scenes and others; in addition, it can also be installed on both the front 151 and the back 152. Independent working depth camera 11.

The depth camera 11 includes a light transmitter 111 and a light receiver 112. The light transmitter 111 of the depth camera 11 can emit laser light, such as infrared laser, which is reflected after reaching the object in the scene. The reflected laser light can be received by the light receiver 112, and the processor 12 can emit according to the light transmitter 111 The laser light and the laser light received by the light receiver 112 calculate the depth information of the object. In one example, the depth camera 11 may obtain depth information through a time of flight (TOF) ranging method. In another example, the depth camera 11 may obtain depth information through a structured light ranging principle. The description of this application takes the depth camera 11 to obtain depth information through the principle of structured light ranging as an example for description.

In the example shown in FIG. 1, the depth camera 11 is installed on the back 152 of the housing 15. It can be understood that the depth camera 11 (that is, the rear depth camera 11) installed on the back 152 needs to meet the normal use of photographing distant objects. Therefore, usually the optical power of the laser light emitted by the light emitter 111 needs to be set to be larger. Satisfy the accuracy of obtaining depth information. However, the rear depth camera 11 is also required to be able to photograph close objects or people at the same time. When the distance is close, the laser with higher optical power is likely to cause harm to people. Therefore, for the rear depth camera 11, ensuring the safety of the depth camera 11 is particularly important and difficult.

The terminal 10 may further include a visible light camera 13. Specifically, the visible light camera 13 may include a telephoto camera and a wide-angle camera, or the visible light camera 13 may include a telephoto camera, a wide-angle camera, and a periscope camera. The visible light camera 13 can be arranged close to the depth camera 11. For example, the visible light camera 13 can be arranged between the light emitter 111 and the light receiver 112, so that the light emitter 111 and the light receiver 112 have a longer distance, which improves The length of the baseline (baseline) of the depth camera 11 improves the accuracy of acquiring depth information.

Please refer to FIG. 2, both the optical transmitter 111 and the optical receiver 112 are connected to the processor 12. The processor 12 may provide an enable signal for the optical transmitter 111. Specifically, the processor 12 may provide an enable signal for the driver 16, wherein the driver 16 is used to drive the optical transmitter 111 to emit laser light. The optical receiver 112 is connected to the processor 12 through an I2C bus. When the optical receiver 112 is used in conjunction with the optical transmitter 111, in an example, the optical receiver 112 can control the projection timing of the optical transmitter 111 through a strobe signal (strobe signal), where the strobe signal is based on the optical receiver 112 The strobe signal can be regarded as an electrical signal with alternating high and low levels, and the light transmitter 111 projects laser light according to the laser projection timing indicated by the strobe signal. Specifically, the processor 12 can send an image acquisition instruction through the I2C bus to enable the depth camera 11 to work. After the optical receiver 112 receives the image acquisition instruction, it controls the switching device 17 through the strobe signal. If the strobe signal is high, Then the switch device 17 sends a pulse signal (pwn) to the driver 16, and the driver 16 drives the light emitter 111 to project laser light into the scene according to the pulse signal. If the strobe signal is low, the switch device 17 stops sending the pulse signal to the driver 16. The light emitter 111 does not project laser light; or, when the strobe signal is low, the switching device 17 sends a pulse signal to the driver 16, and the driver 16 drives the light emitter 111 to project laser light into the scene according to the pulse signal. When the level is high, the switching device 17 stops sending pulse signals to the driver 16, and the light emitter 111 does not project laser light.

In another example, the strobe signal may not be used when the optical receiver 112 and the optical transmitter 111 cooperate. At this time, the processor 12 sends an image acquisition command to the optical receiver 112 and simultaneously sends a laser projection command to the driver 16. The receiver 112 starts to acquire the acquired image after receiving the image acquisition instruction, and when the driver 16 receives the laser projection instruction, it drives the light transmitter 111 to project laser light. When the light emitter 111 projects laser light, the laser light forms a laser pattern with spots and is projected on an object in the scene. The light receiver 112 collects the laser pattern reflected by the object to obtain the speckle image, and sends the speckle image to the processor 12 through the Mobile Industry Processor Interface (MIPI). Each time the optical receiver 112 sends a speckle image to the processor 12, the processor 12 receives a data stream. The processor 12 may calculate the depth information according to the speckle image and the reference image pre-stored in the processor 12.

1 to 3, the control method of the embodiment of the present application can be used to control the aforementioned terminal 10. The control method includes the steps:

031: Control the light emitter 111 to emit a predetermined number of test lasers to the current scene;

032: Control the optical receiver 112 to receive the test laser reflected by the current scene;

033: Obtain the depth information of the current scene according to the received test laser;

034: Determine whether there is a depth less than the preset safety distance in the depth information; and

035: If yes, the control terminal 10 enters the safe mode.

1 to 4, the control device 20 of the embodiment of the present application can be used to control the aforementioned terminal 10. The control device 20 includes a first control module 21, a second control module 22, an acquisition module 23, a first judgment module 24, and The third control module 25. The first control module 21 can be used to implement step 031; the second control module 22 can be used to implement step 032; the acquisition module 23 can be used to implement step 033; the first judgment module 24 can be used to implement step 034; the third control module 25 can be used to implement Step 035. In other words, the first control module 21 can be used to control the light transmitter 111 to emit a predetermined number of test lasers to the current scene; the second control module 22 can be used to control the light receiver 112 to receive the test laser reflected by the current scene; The module 23 can be used to obtain the depth information of the current scene according to the received test laser; the first judging module 24 can be used to judge whether there is a depth less than the preset safety distance in the depth information; the third control module 25 can be used to determine the depth information. There is a depth less than the preset safety distance, and the control terminal 10 enters the safety mode.

1 to 3, the processor 12 in the embodiment of the present application can be used to implement

steps

031, 032, 033, and 034, that is, the processor 12 can be used to: control the light emitter 111 to transmit a preset to the current scene The number of frames of test laser; control the light receiver 112 to receive the test laser reflected by the current scene; obtain the depth information of the current scene according to the received test laser; determine whether there is a depth less than the preset safety distance in the depth information; and if so , The control terminal 10 enters the safe mode.

Specifically, the processor 12 first controls the light emitter 111 to emit a predetermined number of test lasers to the current scene. The predetermined number of frames may be one frame, and correspondingly, the processor 12 may send a pulse control signal to the optical transmitter 111; the predetermined number of frames may be multiple frames, and correspondingly, the processor 12 may send multiple pulses to the optical transmitter 111 control signal. The optical power of the test laser can be set to be less than the optical power of the laser emitted by the optical transmitter 111 in normal use. Specifically, the test can be achieved by controlling the amplitude of the test laser to be small, and the duty ratio of the test laser to be small, etc. The optical power of the laser is small.

The processor 12 controls the light receiver 112 to receive the test laser light reflected by the current scene. The processor 12 can control the optical transmitter 111 and the optical receiver 112 to be turned on at the same time, that is, the processor 12 can implement

steps

031 and 032 at the same time. In the embodiment of the present application, the laser light emitted by the light transmitter 111 has a specific pattern (such as a speckle pattern). The laser light is reflected by the object and then received by the light receiver 112, and the light receiver 112 collects the laser light reflected by the object. After that, a speckle image is formed.

Then, the processor 12 obtains the depth information of the current scene according to the received test laser. Specifically, a pre-calibrated reference image may be stored in the memory of the terminal 10, and the processor 12 processes the aforementioned speckle image and reference image to obtain a depth image of the current scene, where the depth image contains depth information. In an example, the depth image includes multiple pixels, and the pixel value of each pixel is the depth of the current scene corresponding to the pixel. For example, the pixel value of a certain pixel is 20, and the certain pixel corresponds to point A in the scene. Then the pixel value 20 means that the distance from the depth camera 11 to the point A is 20. It can be understood that the smaller the pixel value, the smaller the distance between the corresponding position of the current scene and the depth camera 11.

Then, the processor 12 determines whether there is a depth less than a preset safety distance in the depth information. The safety distance can be set according to relevant safety standards and user attributes, for example, according to the maximum laser energy that the user's eyes can withstand per unit time, according to the target user population of the terminal 10, and according to the target usage scenario of the terminal 10, etc. set. The safety distance can be set to any distance such as 100 mm, 200 mm, 250 mm, 1000 mm, etc., and there is no restriction here. As described above, the depth information may include the depths of multiple locations in the current scene, and the processor 12 may compare the depth of each location with the safety distance, and when the depth of at least one location is less than the safety distance, the position is determined Objects (such as people) are more susceptible to laser damage.

Then, the processor 12 controls the terminal 10 to enter the safety mode when there is a depth less than the preset safety distance in the depth information. As described above, when it is determined that an object is more likely to be injured by the laser, the control terminal 10 enters the safe mode to ensure that the object in the current scene will not be injured. Specifically, the processor 12 controls the terminal 10 to enter the safe mode, which may be to control the terminal 10 to send a prompt signal (for example, control the display screen 14 to display a prompt window prompting the user to stay away, control the speaker of the terminal 10 to send out a prompt voice prompting the user to stay away, and control The vibration motor of the terminal 10 sends out a vibration that prompts the user to stay away, etc.), which may be controlling the light emitter 111 to emit laser at a preset safe frequency, or controlling the light emitter 111 to emit one of the lasers at a preset safe amplitude. Kind or more.

5, the processor 12 controls the waveform of the laser light emitted by the optical transmitter 111 by default as shown in L1, a high level indicates that the optical transmitter 111 is emitting laser light, and a low level indicates that the optical transmitter 111 is not emitting laser light. As shown in Figure 5, L2 is the waveform of the laser light emitted by the optical transmitter 111 controlled by the processor 12 at a preset safe frequency. The safe frequency may be smaller than the default frequency of the laser emitted by the optical transmitter 111, for example, the safe frequency is the default frequency. 1/2, 1/3, etc., so that the user's energy per unit time by laser irradiation is low and avoid harm to the user. L3 in Fig. 5 is the waveform of the laser light emitted by the processor 12 controlling the optical transmitter 111 at a preset safe amplitude, where the safe amplitude may be smaller than the default amplitude of the laser emitted by the optical transmitter 111, for example, the safe amplitude It is 2/3, 1/2, 1/3, etc. of the default amplitude. L4 in FIG. 5 is the waveform of the laser light emitted by the processor 12 controlling the optical transmitter 111 at a safe frequency and a safe amplitude. It can be understood that after changing the waveform of the laser, the depth camera 11 can still be used to obtain a depth image in the scene, which has little impact on the user experience.

In the related prior art, when the user is close to the depth camera, the laser light emitted by the depth camera irradiates the user with too much energy, which is likely to cause harm to the user, and the use safety of the depth camera is low. In summary, in the terminal 10, the control method, and the control device 20 of the embodiment of the present application, the optical transmitter 111 is controlled to emit the test laser, and the optical receiver 112 receives the reflected test laser, and first obtains the depth according to the reflected test laser. According to the depth information, it is judged whether there is a depth less than the preset safety distance. If it exists, it is judged that if the current laser is irradiated on the user's body, such as the eyes, it is likely to cause harm to the user, and the terminal 10 is further controlled to enter the safe mode. So that when the user's use distance is relatively short, the security of using the terminal 10 is also high. At the same time, since the use distance of the user is pre-detected by the depth camera 11, there is no need to add a distance detection device other than the depth camera 11 for pre-detection, which reduces the size and manufacturing cost of the terminal 10.

Referring to FIG. 6, in some embodiments, the control method further includes steps:

066: If there is no depth less than the preset safety distance in the depth information, the control terminal 10 obtains the depth information of the current scene in the set mode; and

067: Determine whether there is a depth less than the preset safety distance in the depth information acquired in the set mode.

When it is determined in step 067 that there is a depth less than the preset safety distance in the depth information acquired in the set mode, step 065 can also be implemented: controlling the terminal 10 to enter the safe mode.

4 and 6, in some embodiments, the third control module 25 can also be used to implement step 066, and the first judgment module 24 can also be used to implement step 067. In other words, the third control module 25 can also be used to control the terminal 10 to obtain the depth information of the current scene in a set mode if there is no depth less than the preset safety distance in the depth information; the first judgment module 24 can also be used To determine whether there is a depth less than the preset safety distance in the depth information acquired in the set mode. The third control module 25 can also be used to implement step 065 when determining that the depth information acquired in the set mode has a depth less than the preset safety distance, that is, control the terminal 10 to enter the safety mode.

1 and 6, in some embodiments, the processor 12 can also be used to implement

steps

066 and 067, that is to say, the processor 12 can also be used if the depth information does not have a safety distance smaller than the preset safety distance. The control terminal 10 obtains the depth information of the current scene in the set mode; and determines whether there is a depth less than the preset safety distance in the depth information obtained in the set mode. The processor 12 is further configured to implement step 065 when determining that there is a depth less than the preset safety distance in the depth information acquired in the set mode, that is, control the terminal 10 to enter the safety mode.

For the content and specific implementation details of

steps

061, 062, 063, 064, and 065 in FIG. 6, please refer to the description of

steps

031, 032, 033, 034, and 035 in the specification of this application, which will not be repeated here.

Specifically, when there is no depth less than the preset safety distance in the depth information, it can be determined that the object in the current scene is not too close to the depth camera 11, and acquiring the depth information in the set mode will not cause harm to the user. The processor 12 may control the depth camera 11 to acquire the depth information of the current scene in a set mode. Specifically, the set mode may be the default working mode of the depth camera 11 of the terminal 10, and the set mode includes information such as the set waveform of the laser emitted by the light transmitter 111, such as the L1 waveform shown in FIG. 5.

The processor 12 determines whether there is a depth less than a preset safety distance in the depth information acquired in the set mode. In combination with the above specific description of the processor 12 implementing step 034, the depth information acquired in the set mode includes the depth of each position in the scene, and the depth can be reflected by the pixel value of the pixel in the depth image. By comparing each depth with the safety distance, when there is a depth less than the safety distance, it is judged that although there is no object too close to the depth camera 11 at the initial moment in the scene, there are objects too close to the depth camera 11 in the scene during use. It is necessary to ensure the safety of the user at this time. Therefore, the terminal 10 can be controlled to be in the aforementioned security mode.

Further, if there is no depth less than the preset safety distance in the depth information obtained in the set mode, the processor 12 may continue to control the terminal 10 to obtain the depth information of the current scene in the set mode.

Referring to FIG. 7, in some embodiments, the control method further includes step 076: judging whether there are human eyes in the current scene according to the depth information. When it is judged that there are human eyes in the current scene, step 074 is implemented.

Referring to FIGS. 7 and 8, in some embodiments, the control device 20 further includes a second judgment module 26, which can be used to implement step 076, that is, the second judgment module 26 can be used to The depth information determines whether there are human eyes in the current scene. When judging that there are human eyes in the current scene, the first judging module 24 implements step 074.

1 and FIG. 7, in some embodiments, the processor 12 may also be used to implement step 076, that is, the processor 12 may also be used to determine whether there are human eyes in the current scene according to the depth information. When judging that there are human eyes in the current scene, the processor 12 implements step 074.

For the content and specific implementation details of

steps

071, 072, 073, 074, and 075 in FIG. 7, please refer to the description of

steps

Specifically, since the human eye’s ability to tolerate laser light is significantly lower than that of the skin on the rest of the human body, the human eye is often hurt first when the human eye is injured. Therefore, it can be judged whether there is a human eye in the current scene. When, judge whether the current use distance is less than the safety distance. In an example, if it is determined that there are no human eyes, the processor 12 may directly implement step 076 to improve the timeliness of obtaining depth information.

As mentioned above, the depth information can be characterized by the pixel values of multiple pixels in the depth image, and the processor 12 can match the preset human eye model according to the distribution of the pixel values of the multiple pixels, such as the presence in the depth image. If the matching degree exceeds the predetermined threshold, it is determined that there are human eyes in the current scene. If there is no area in the depth image with the matching degree exceeding the predetermined threshold, then it is determined that there are no human eyes in the current scene.

Please refer to FIG. 9, the depth image I includes multiple pixels P, and the pixel value of each pixel P (such as 21, 22, 23, 24) represents the depth of the corresponding position of the pixel P. For example, in the area D of the depth image I, according to the distribution of pixel values in the area D, it is judged that the depth distribution of the object corresponding to the area D is roughly that the depth of the middle strip area is smaller, and the depth around the strip area gradually increases. Large, the depth distribution has a high degree of matching with the human eye model of the orthoscopic depth camera 11, so it is judged that there are human eyes in the current scene, and this area D corresponds to the position of the human eye in the current scene.

Of course, in other embodiments, the processor 12 can also use the visible light image of the current scene acquired by the visible light camera 13 to jointly confirm whether there are human eyes in the current scene. Specifically, at the same time, it can judge whether there are people in the current scene by identifying the characteristic information in the visible light image. Eyes, when the presence of human eyes is recognized through both visible light images and depth information, it is determined that there are living human eyes in the current scene, and situations where there are only human eye photos or only human eye molds are excluded.

Referring to FIG. 10, in some embodiments, the predetermined number of frames includes at least two frames, and the control method further includes the steps:

01031: Obtain the first depth information of the current scene according to the received test laser of the previous frame;

01032: Obtain the second depth information of the current scene according to the received test laser of the next frame; and

01033: Calculate the depth information of the current scene when the light emitter 111 emits the next frame of laser light according to the first depth information, the second depth information, the emission time of the previous frame of test laser light and the emission time of the next frame of test laser light.

Referring to FIG. 10 and FIG. 11, in some embodiments, the predetermined number of frames includes at least two frames, and the acquisition module 23 includes a first acquisition unit 231, a second acquisition unit 232, and a first calculation unit 233. The first obtaining unit 231 can be used to implement step 01033, the second obtaining unit 232 can be used to implement step 01032, and the first calculating unit 233 can be used to implement step 01033. In other words, the first acquisition unit 231 can be used to acquire the first depth information of the current scene according to the received test laser of the previous frame; the second acquisition unit 232 can be used to acquire the current scene information according to the received test laser of the next frame Second depth information; the first calculation unit 233 can be used to calculate the light emitter 111 to emit the next frame of laser light according to the first depth information, the second depth information, the emission time of the previous frame of test laser and the emission time of the next frame of test laser Time depth information of the current scene.

1 and 10, in some embodiments, the predetermined number of frames includes at least two frames, and the processor 12 may also be used to implement

steps

01033, 01033, and 01033. In other words, the processor 12 may be used to obtain the first depth information of the current scene according to the received test laser of the previous frame; obtain the second depth information of the current scene according to the received test laser of the next frame; and The depth information, the second depth information, the emission time of the previous frame of test laser light, and the emission time of the next frame of test laser light are used to calculate the depth information of the current scene when the light transmitter 111 emits the next frame of laser light.

Among them, the content and specific implementation details of

steps

0101, 0102, 0104, and 0105 in FIG. 10 can refer to the description of

steps

031, 032, 034, and 035 in the specification of this application, which will not be repeated here, steps 01031, 01032 and 01033 may be a substep of step 033.

It can be understood that when the optical transmitter 111 emits the test laser, the distance between the user and the depth camera 11 may be greater than the safe distance. Based on this judgment, the depth camera 11 may acquire depth information in a set mode, that is, may use the default optical power The laser is emitted into the current scene, and there is a time difference between the optical transmitter 111 emitting the test laser and the default optical power, which may cause the user to be less than safe from the depth camera 11 when the laser is emitted with the default optical power. Distance, causing the user to be injured by the laser.

12, in this embodiment, the emission time of the test laser in the previous frame is t1, the first depth information of the object T in the current scene at time t1 is d1, and the emission time of the test laser in the next frame is t2, t2 The second depth information of the object T is d2. According to the received test laser of the previous frame and the next frame, the method of obtaining the first depth information d1 and the second depth information d2 of the current scene respectively can refer to the above description of the processor 12 implementing step 033, which will not be repeated here. . Among them, the previous frame and the next frame only indicate that the two frames have sequential test lasers, which does not mean that the previous frame and the next frame can only be two adjacent frames.

It can be seen from FIG. 12 that when the test laser is emitted, the object T and the terminal 10 are in a relative motion state, for example, the terminal 10 is not moving, and the object T (such as a person or an object) is approaching the terminal 10, or the object T( For example, the person or object being photographed does not move, the user holding the terminal 10 in his hand is approaching the object T (for example, the person or object being photographed), and the relative distance between the object T and the terminal 10 is constantly changing. The first depth information is d1, the second depth information is d2, the emission time t1 of the test laser in the previous frame and the emission time t2 of the test laser in the next frame can calculate the relative motion state of the object T and the terminal 10, for example, d1-d2=k(t2-t1), the motion coefficient k is obtained.

The processor 12 further calculates the depth of the object T at time t3 according to the time t3 when the light emitter 111 emits (not actually emitted) the next frame of laser (the waveform of the laser may be different from the waveform of the test laser) and the above-mentioned relative motion state Information d3, where d3-d2=k(t3-t2), or d3-d1=k(t3-t1), and the depth information d3 is used as the depth information in step 0104 to determine whether there is less than The depth of the preset safety distance. When the judgment result in step 0104 is yes, it means that the next frame of laser cannot be emitted at t3, and the terminal 10 needs to enter the safe mode.

Referring to Figure 13, in some embodiments, step 066 includes the steps:

0131: Control the light transmitter 111 to emit laser light to the current scene at the first operating frequency;

0132: Control the optical receiver 112 to acquire the collected image at the second operating frequency, which is greater than the first operating frequency;

0133: distinguish the first image collected when the light emitter 111 is not emitting laser light and the second image collected when the light emitter 111 is emitting laser light in the collected images; and

0134: Calculate depth information based on the first image, the second image, and the reference image.

Referring to FIGS. 13 and 14, in some embodiments, the third control module 25 includes a first control unit 251, a second control unit 252, a distinguishing unit 253, and a second calculation unit 254. The first control unit 251 can be used to implement step 0131, the second control unit 252 can be used to implement step 0132, the distinguishing unit 253 can be used to implement step 0133, and the second calculation unit 254 can be used to implement step 0134. In other words, the first control unit 251 can be used to control the light transmitter 111 to emit laser light to the current scene at the first operating frequency; the second control unit 252 can be used to control the light receiver 112 to acquire the collected images at the second operating frequency; The unit 253 can be used to distinguish the first image collected when the light emitter 111 is not emitting laser light and the second image collected when the light emitter 111 emits laser light; the second calculation unit 254 can be used to distinguish the first image collected according to the first image , The second image and the reference image calculate the depth information.

Referring to FIG. 1 and FIG. 13, in some embodiments, the processor 12 may also be used to implement

steps

0131, 0132, 0133, and 0134. That is to say, the processor 12 can be used to control the optical transmitter 111 to emit laser light to the current scene at the first operating frequency; to control the optical receiver 112 to acquire the collected images at the second operating frequency, which is greater than the first operating frequency; In the collected images, distinguish the first image collected when the light emitter 111 is not emitting laser light and the second image collected when the light emitter 111 emits laser light; and calculate the depth information based on the first image, the second image, and the reference image .

Specifically, the operating frequencies of the optical receiver 112 and the optical transmitter 111 are different (that is, the second operating frequency is greater than the first operating frequency). For example, as shown in FIG. 15, the solid line represents the timing of the optical transmitter 111 emitting laser light, and the dashed line represents the light receiving The timing of acquiring the captured image and the number of frames of the captured image by the device 112. The dotted line represents the frame number of the speckle image formed by only the infrared laser emitted by the light transmitter 111 obtained from the first image and the second image, as shown in FIG. From top to bottom, it is a solid line, a dashed line, and a dot-dash line in sequence, wherein the second operating frequency is twice the first operating frequency. Please refer to the solid and dashed parts in FIG. 15, the processor 12 controls the optical receiver 112 to first receive the infrared light in the environment (hereinafter referred to as ambient infrared light) when the optical transmitter 111 is not projecting laser light to obtain the Nth frame of acquisition image ( This is the first image, which can also be called the background image); subsequently, the processor 12 controls the light receiver 112 to receive the ambient infrared light and the infrared laser emitted by the light transmitter 111 when the light transmitter 111 projects laser light to obtain the first image N+1 frames of collected images (the second image at this time, which can also be called interference speckle images); subsequently, the processor 12 controls the light receiver 112 to receive ambient infrared light when the light transmitter 111 does not project laser light to obtain The N+2th frame acquires an image (the first image at this time), and so on, the light receiver 112 acquires the first image and the second image alternately.

It should be noted that the processor 12 may control the light receiver 112 to first obtain the second image, and then obtain the first image, and alternately execute the acquisition of the collected images according to this sequence. In addition, the above-mentioned multiple relationship between the second operating frequency and the first operating frequency is only an example. In other embodiments, the multiple relationship between the second operating frequency and the first operating frequency may also be three times or four times. , Five times, six times and so on.

The processor 12 distinguishes each captured image and determines whether the captured image is the first image or the second image. After the processor 12 obtains at least one frame of the first image and at least one frame of the second image, it can calculate the depth information according to the first image, the second image, and the reference image. Specifically, since the first image is collected when the light emitter 111 is not projecting laser light, the light that forms the first image includes only ambient infrared light, and the second image is collected when the light emitter 111 is projecting laser light, forming the first image. The light of the two images includes both the ambient infrared light and the infrared laser emitted by the light emitter 111. Therefore, the processor 12 can remove the part of the collected image formed by the ambient infrared light in the second image according to the first image, so as to obtain The collected image formed by the infrared laser emitted by the light transmitter 111 (ie, the speckle image formed by the infrared laser).

It can be understood that the ambient light includes infrared light with the same wavelength as the laser light emitted by the light transmitter 111 (for example, includes ambient infrared light at 940 nm), and when the light receiver 112 acquires the captured image, this part of the infrared light will also be absorbed by the light receiver. 112 received. When the brightness of the scene is high, the proportion of ambient infrared light in the light received by the light receiver 112 will increase, resulting in inconspicuous laser speckles in the collected image, thereby affecting the calculation of the depth image. In this embodiment, the light transmitter 111 and the light receiver 112 work at different operating frequencies, and the light receiver 112 can collect the first image formed by only ambient infrared light and the first image emitted by the ambient infrared light and the light transmitter 111 at the same time. The second image formed by the infrared laser, and based on the first image, the part of the image formed by the ambient infrared light in the second image is removed, so that the laser speckle can be distinguished, and the infrared laser emitted by only the light emitter 111 can be used The acquired images are formed to calculate the depth information, and the laser speckle matching is not affected, which can avoid partial or complete loss of the depth information, thereby improving the accuracy of the depth information.

In some embodiments, step 0133 includes:

01331: Determine the working state of the light emitter 111 at the acquisition time according to the acquisition time of each frame of the acquired image;

01332: Add an image type to each frame of collected images according to the working status; and

01333: distinguish the first image from the second image according to the image type.

Please refer to FIG. 14 again. In some embodiments, step 01331, step 01332, and step 01333 can all be implemented by the distinguishing unit 253. In other words, the distinguishing unit 253 can also be used to determine the working status of the light emitter 111 at the time of collection according to the collection time of each frame of the collected image; add the image type to each frame of the collected image according to the working status and distinguish according to the image type The first image and the second image.

Referring to FIG. 1 and FIG. 2, in some implementation manners, step 01331, step 01332, and step 01333 may all be implemented by the processor 12. In other words, the processor 12 can also be used to determine the working status of the light emitter 111 at the time of collection according to the collection time of each frame of the collected image; add the image type to each frame of the collected image according to the working status and distinguish according to the image type The first image and the second image.

Specifically, each time the processor 12 receives a frame of the captured image from the light receiver 112, it adds an image type (stream_type) to the captured image, so that the first image and the second image can be distinguished according to the image type in subsequent processing. Specifically, during the period when the optical receiver 112 acquires the captured image, the processor 12 will monitor the working status of the optical transmitter 111 in real time via the I2C bus. Each time the processor 12 receives a frame of captured image from the light receiver 112, it will first acquire the acquisition time of the acquired image, and then determine according to the acquisition time of the acquired image that the working state of the light transmitter 111 is projection The laser is still not projected, and the image type is added to the captured image based on the judgment result. The collection time of the collected image may be the start time, end time, any time between the start time and the end time when the light receiver 112 obtains each frame of the collected image, and so on. In this way, it is possible to realize the correspondence between each frame of the collected image and the working state (laser projected or not) of the light emitter 111 during the acquisition of the frame of the collected image, and accurately distinguish the type of the collected image. In an example, the structure of the image type stream_type is shown in Table 1:

Table 1

When stream is 0 in Table 1, it means that the data stream at this time is an image formed by infrared light and/or infrared laser. When light is 00, it means that the data stream at this time is acquired without any equipment projecting infrared light and/or infrared laser (only ambient infrared light), then the processor 12 can add an image type of 000 to the collected image , To identify this captured image as the first image. When light is 01, it means that the data stream at this time is acquired when the light transmitter 111 projects infrared lasers (both ambient infrared light and infrared lasers). The processor 12 may add an image type of 001 to the captured image to identify this captured image as the second image. The processor 12 can then distinguish the image types of the collected images according to stream_type.

In some embodiments, the processor 12 includes a first storage area, a second storage area, and a logical subtraction circuit, and the logical subtraction circuit is connected to both the first storage area and the second storage area. The first storage area is used to store the first image, the second storage area is used to store the second image, and the logical subtraction circuit is used to process the first image and the second image to obtain a speckle image formed by infrared lasers. Specifically, the logical subtraction circuit reads the first image from the first storage area, reads the second image from the second storage area, and performs subtraction on the first image and the second image after acquiring the first image and the second image The speckle image formed by infrared laser is processed. The logic subtraction circuit is also connected to the depth calculation module in the processor 12 (for example, it may be an integrated circuit ASIC dedicated to calculating depth, etc.). The logic subtraction circuit sends the speckle image formed by the infrared laser to the depth calculation module. The depth calculation module calculates depth information based on the speckle image formed by the infrared laser and the reference image.

Please refer to FIG. 16, this application also provides one or more non-volatile computer-readable storage media 200 containing computer-readable instructions. When the computer-readable instructions are executed by the processor 300, the processor 300 executes the control method described in any one of the foregoing embodiments. The processor 300 may be the processor 12 in FIGS. 1 and 2.

For example, referring to FIG. 3, when the computer-readable instructions are executed by the processor 300, the processor 300 is caused to perform the following steps:

035: If yes, the control terminal 10 enters the safe mode.

In the description of this specification, reference is made to the terms “certain embodiments”, “one embodiment”, “some embodiments”, “exemplary embodiments”, “examples”, “specific examples”, or “some examples”. The description means that a specific feature, structure, material, or characteristic described in conjunction with the embodiment or example is included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above-mentioned terms do not necessarily refer to the same embodiment or example. Moreover, the described specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in an appropriate manner.

In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present application, "a plurality of" means at least two, for example, two, three, unless otherwise specifically defined.

Although the embodiments of the present application have been shown and described above, it can be understood that the above-mentioned embodiments are exemplary and should not be construed as limiting the present application. A person of ordinary skill in the art can comment on the foregoing within the scope of the present application. The embodiments undergo changes, modifications, substitutions and modifications, and the scope of this application is defined by the claims and their equivalents.

Claims

A control method of a terminal, the terminal includes a depth camera, and the depth camera includes a light transmitter and a light receiver, wherein the control method includes:

Controlling the light emitter to emit a predetermined number of test lasers to the current scene;

Controlling the optical receiver to receive the test laser reflected by the current scene;

Obtain the depth information of the current scene according to the received test laser;

Determine whether there is a depth less than a preset safety distance in the depth information; and

If yes, control the terminal to enter the safe mode.
The control method according to claim 1, wherein the control method further comprises:

If there is no depth less than the preset safety distance in the depth information, control the terminal to acquire the depth information of the current scene in a set mode;

Determine whether there is a depth less than the preset safety distance in the depth information obtained in the set mode; and

If yes, control the terminal to enter the safe mode.
The control method according to claim 1, wherein said controlling said terminal to enter a safe mode comprises:

Control the terminal to send out a reminder signal; and/or

Controlling the light emitter to emit laser light at a preset safe frequency; and/or

The light emitter is controlled to emit laser light at a preset safe amplitude.
The control method according to claim 1, wherein the control method further comprises:

Judging whether there are human eyes in the current scene according to the depth information; and

If yes, perform the step of judging whether there is a depth less than a preset safety distance in the depth information.
The control method according to claim 1, wherein the predetermined number of frames includes at least two frames, and the obtaining the depth information of the current scene according to the received test laser includes:

Acquire the first depth information of the current scene according to the received test laser of the previous frame;

Obtain the second depth information of the current scene according to the received test laser of the next frame; and

Calculate the current scene when the light emitter emits the next frame of laser light based on the first depth information, the second depth information, the emission time of the previous frame of test laser light, and the emission time of the next frame of test laser light In-depth information.
The control method according to claim 2, wherein the controlling the terminal to obtain the depth information of the current scene in a set mode comprises:

Controlling the light emitter to emit laser light to the current scene at the first operating frequency;

Controlling the optical receiver to acquire a captured image at a second operating frequency, where the second operating frequency is greater than the first operating frequency;

Distinguish between the first image collected when the light emitter is not emitting laser light and the second image collected when the light emitter emits laser light; and

Calculate depth information according to the first image, the second image, and the reference image.
The control method according to claim 6, wherein the first image collected when the light emitter is not emitting laser light and the first image collected when the light emitter emits laser light are distinguished from the collected images. The second image includes:

Determining the working state of the light emitter at the acquisition time according to the acquisition time of the acquired image in each frame;

Adding an image type to each frame of the collected image according to the working state; and

The first image and the second image are distinguished according to the image type.
A control device of a terminal, the terminal includes a depth camera, the depth camera includes a light transmitter and a light receiver, and is characterized in that the control device includes:

The first control module is configured to control the light emitter to emit a predetermined number of test lasers to the current scene;

The second control module is configured to control the optical receiver to receive the test laser reflected by the current scene;

The acquisition module is used to acquire the depth information of the current scene according to the received test laser;

The first judgment module is used to judge whether there is a depth less than a preset safety distance in the depth information; and

The third control module is configured to control the terminal to enter a safe mode if there is a depth less than a preset safe distance in the depth information.
The control device according to claim 8, wherein:

The third control module is further configured to control the terminal to acquire the depth information of the current scene in a set mode if there is no depth less than a preset safety distance in the depth information;

The first judgment module is also used to judge whether there is a depth less than a preset safety distance in the depth information acquired in the set mode;

The third control module is further configured to control the terminal to enter a safe mode if there is a depth less than a preset safe distance in the depth information obtained in the set mode.
The control device according to claim 8, wherein said controlling said terminal to enter a safe mode comprises:

Control the terminal to send out a reminder signal; and/or

Controlling the light emitter to emit laser light at a preset safe frequency; and/or

The light emitter is controlled to emit laser light at a preset safe amplitude.
The control device according to claim 8, wherein the control device further comprises a second judgment module, and the second judgment module is configured to judge whether there are human eyes in the current scene according to the depth information;

When the second judgment module judges that there are human eyes in the current scene according to the depth information, the first judgment module is used to judge whether there is a depth less than a preset safe distance in the depth information.
The control device according to claim 8, wherein the predetermined number of frames includes at least two frames, and the acquisition module includes:

The first acquiring unit is configured to acquire the first depth information of the current scene according to the received test laser of the previous frame;

The first acquiring unit is configured to acquire the second depth information of the current scene according to the received test laser of the next frame; and

The first calculation unit is configured to calculate the emission of the light emitter according to the first depth information, the second depth information, the emission time of the previous frame of test laser light, and the emission time of the next frame of test laser light The depth information of the current scene in the next frame of laser.
The control device according to claim 9, wherein the third control module comprises:

The first control unit is configured to control the light transmitter to emit laser light to the current scene at the first operating frequency;

A second control unit, configured to control the optical receiver to acquire a captured image at a second operating frequency, where the second operating frequency is greater than the first operating frequency;

A distinguishing unit for distinguishing, in the collected images, a first image collected when the light emitter is not emitting laser light and a second image collected when the light emitter is emitting laser light; and

The second calculation unit is configured to calculate depth information according to the first image, the second image, and the reference image.
The control device according to claim 13, wherein the distinguishing unit is configured to:

Determining the working state of the light emitter at the acquisition time according to the acquisition time of the acquired image in each frame;

Adding an image type to each frame of the collected image according to the working state; and

The first image and the second image are distinguished according to the image type.
A terminal, characterized in that it includes a depth camera and a processor, the depth camera includes a light transmitter and a light receiver, and the processor is configured to:

Controlling the light emitter to emit a predetermined number of test lasers to the current scene;

Controlling the optical receiver to receive the test laser reflected by the current scene;

Obtain the depth information of the current scene according to the received test laser;

Determine whether there is a depth less than a preset safety distance in the depth information; and

If yes, control the terminal to enter the safe mode.
The terminal according to claim 15, wherein the processor is further configured to:

If there is no depth less than the preset safety distance in the depth information, control the terminal to acquire the depth information of the current scene in a set mode;

Determine whether there is a depth less than the preset safety distance in the depth information obtained in the set mode; and

If yes, control the terminal to enter the safe mode.
The terminal according to claim 15, wherein the processor is further configured to:

Control the terminal to send out a reminder signal; and/or

Controlling the light emitter to emit laser light at a preset safe frequency; and/or

The light emitter is controlled to emit laser light at a preset safe amplitude.
The terminal according to claim 15, wherein the processor is further configured to:

Judging whether there are human eyes in the current scene according to the depth information; and

If yes, perform the step of judging whether there is a depth less than a preset safety distance in the depth information.
The terminal according to claim 15, wherein the predetermined number of frames includes at least two frames, and the processor is further configured to:

Acquire the first depth information of the current scene according to the received test laser of the previous frame;

Obtain the second depth information of the current scene according to the received test laser of the next frame; and

Calculate the current scene when the light emitter emits the next frame of laser light based on the first depth information, the second depth information, the emission time of the previous frame of test laser light, and the emission time of the next frame of test laser light In-depth information.
The terminal according to claim 16, wherein the processor is further configured to:

Controlling the light emitter to emit laser light to the current scene at the first operating frequency;

Controlling the optical receiver to acquire a captured image at a second operating frequency, where the second operating frequency is greater than the first operating frequency;

Distinguish between the first image collected when the light emitter is not emitting laser light and the second image collected when the light emitter emits laser light; and

Calculate depth information according to the first image, the second image, and the reference image.
The terminal according to claim 20, wherein the processor is further configured to:

Determining the working state of the light emitter at the acquisition time according to the acquisition time of the acquired image in each frame;

Adding an image type to each frame of the collected image according to the working state; and

The first image and the second image are distinguished according to the image type.
One or more non-volatile computer-readable storage media containing computer-readable instructions, which when executed by a processor, cause the processor to execute the control described in any one of claims 1-7 method.