WO2022206605A1

WO2022206605A1 - Method for determining target object, and photographing method and device

Info

Publication number: WO2022206605A1
Application number: PCT/CN2022/083080
Authority: WO
Inventors: 徐其超; 陈帅; 刘蒙; 吴虹
Original assignee: 华为技术有限公司
Priority date: 2021-03-29
Filing date: 2022-03-25
Publication date: 2022-10-06
Also published as: CN115225756A

Abstract

Disclosed in embodiments of the present application are a method for determining a target object, and a photographing method and device, for use in conveniently and accurately determining a target object and improving the convenience of photographing. The method for determining the target object comprises: obtaining an image to be processed; determining object information of each object in said image, the object information comprising area information and attribute information; extracting second index data of each piece of attribute information from a pre-established data set, the data set comprising attribute key words and attribute indexes; for each object, associating the second index data with the area information to obtain an association between the index data and the area information; obtaining first voice information; extracting first index data of the first voice information from the data set on the basis of an attribute key word corresponding to the second index data in the data set; and determining, on the basis of the association and the second index data, target area information corresponding to the first index data, an object corresponding to the target area information being a target object.

Description

Method, photographing method and device for determining target object

This application claims the priority of the Chinese patent application filed with the State Intellectual Property Office on March 29, 2021, the application number is 202110336707.X, and the application name is "Method for Determining Target Object, Shooting Method and Device", the entire content of which is approved by Reference is incorporated in this application.

technical field

The present application relates to the technical field of image processing, and in particular, to a method, a photographing method, and an apparatus for determining a target object.

Background technique

With the continuous development of terminal technology and image processing technology, the shooting function of terminal equipment is also becoming more and more powerful.

At present, when a terminal device is shooting, a target object is manually selected, and then the target object is shot to obtain a video or picture of the tracking object.

When the user manually selects the target object, the actual selected target object may be inconsistent with the target object that the user wants to specify due to reasons such as clicking errors, that is, the target object cannot be accurately selected. In addition, when a user holds an item in his hand, etc., he needs to put down the item in his hand first, and then manually select the target object, which is less convenient.

SUMMARY OF THE INVENTION

The embodiments of the present application provide a method for determining a target object, a shooting method and an apparatus, which can allow a user to conveniently and accurately specify a target object.

In a first aspect, an embodiment of the present application provides a method for determining a target object. The method is applied to a terminal device, by first acquiring an image to be processed, and the image to be processed includes at least one object; and then determining object information of each object in the image to be processed , the object information includes area information and attribute information, the area information is used to describe the position of the object in the image to be processed; then, the second index data of each attribute information is extracted from a pre-established data set, the data set includes attribute keywords Word and attribute index; for each object, associate the second index data with the region information to obtain the association relationship between the index data and the region information; when acquiring the first voice information, based on the second index data in the dataset corresponding to attribute key words, extract the first index data of the first voice information from the data set; finally, based on the association relationship and the second index data, determine the target area information corresponding to the first index data, wherein, the object corresponding to the target area information for the target object.

In the embodiment of the present application, the object information of each object in the image is identified in advance, and based on the pre-established data set, the association relationship between the index data of each object and the area information is established; then, when the voice command is collected, The object in the image corresponding to the voice command is determined based on the association relationship, so that the user can conveniently and quickly specify the target object.

In some possible implementations of the first aspect, the above-mentioned process of determining target area information corresponding to the first index data based on the association relationship and the second index data may include: for each second index data, The indexes in the first index data are respectively matched with the indexes in the first index data, and the number of items of the target index is determined, and the target index is the index in the second index data that matches the index in the first index data; based on the association relationship , and the area information corresponding to the second index data with the largest number of items is determined as the target area information.

In some possible implementations of the first aspect, the above process of extracting the second index data of each attribute information from the pre-established data set may include: for each attribute information, searching for the first index data matching the attribute information from the data set Key word; the index data of the first key word in the data set is used as the second index data.

In some possible implementations of the first aspect, the process of extracting the first index data of the first voice information from the data set based on the attribute keywords corresponding to the second index data in the data set may include: Carry out voice recognition to obtain a first voice recognition result; according to the first voice recognition result, determine that the first voice information is a voice command for specifying a target object; search for the first voice from the attribute keyword corresponding to the second index data Identify the second key word matching the result; and use the index data of the second key word in the data set as the first index data.

In some possible implementations of the first aspect, the above-mentioned process of performing speech recognition on the first voice information to obtain the first voice recognition result may include: extracting voiceprint features of the first voice information; determining the voiceprint features and pre-storing Similarity between voiceprint features; when the similarity is greater than or equal to a preset threshold, input the voiceprint feature and the first speech information into the semantic understanding model to obtain the first speech recognition result output by the semantic understanding model.

In some possible implementations of the first aspect, the above process of determining object information of each object in the image to be processed may include: inputting the image to be processed into the target recognition model; and obtaining object information output by the target recognition model.

In some possible implementations of the first aspect, the first voice information is a voice command for specifying the target object for the first time, or a voice command for increasing or decreasing the target object, or a voice command for replacing the target object. In this implementation, the user can not only specify the target object for the first time by voice, but also can increase or decrease the target object and replace the target object by voice on the basis of specifying the target object, which further increases the convenience of shooting and improves the user experience.

In some possible implementations of the first aspect, the image to be processed includes N original images with different viewing angles, where N is a positive integer greater than or equal to 1. After determining the target area information corresponding to the first index data, the method further includes: obtaining a first output image of the target object according to the target area information based on the original image, and displaying the first output image in the viewfinder frame.

In some possible implementations of the first aspect, the above process of obtaining the first output image of the target object based on the original image and the target area information may include: determining the first target image from N original images, and the first target image It is the image with the smallest field of view in the image set, and the image set includes at least one original image that can completely contain the bounding box selection area of the target object; according to the target area information, the first sub-image of the target object is cropped from the first target image. ; generating a first output image based on the first sub-image.

In some possible implementations of the first aspect, when there are at least two target objects, and each target object corresponds to a first sub-image, the above-mentioned process of obtaining the first output image based on the first sub-image may include: : Splicing the first sub-images corresponding to each target object to obtain the first output image.

In some possible implementations of the first aspect, after displaying the first output image in the viewfinder frame, the method further includes: acquiring second voice information; performing voice recognition on the second voice information to obtain a second voice recognition result ; According to the second voice recognition result, determine that the second voice information is a voice command for adjusting the display mode of the target object; Determine the second target image from the original image; Cut out the second child of the target object from the second target image image; generating a second output image based on the second sub-image, and displaying the second output image in the viewfinder.

In this implementation manner, the user can adjust the display mode of the target object through voice, which further improves the convenience of shooting.

In some possible implementations of the first aspect, the method may further include: in the third target image, determining the distance between each edge of the bounding box selection area of the target object and the corresponding edge of the image, and the third The target image is the image with the largest field of view in the image set; if the minimum value among the multiple distances is less than or equal to the preset distance threshold, prompt information is displayed in the viewfinder, and the prompt information is used to prompt adjustment of the camera pose.

In this implementation manner, when the target object is about to exceed the maximum capture range of the camera, the user experience can be improved by prompting the user to adjust the camera pose.

In some possible implementations of the first aspect, the above attribute information includes at least one of the following: type, gender, age group, activity information, hair length, hair color, clothing type, clothing color, and posture.

In a second aspect, an embodiment of the present application provides a method for determining a target object, which is applied to a terminal device. In the method, a picture captured by a camera is displayed in a viewfinder, and the picture includes at least one object; after obtaining the first voice information When the first voice information is used to designate the target object, the object corresponding to the first voice information in the screen is determined as the target object.

In some possible implementations of the second aspect, the above process of determining the object corresponding to the first voice information in the picture as the target object may include: extracting the first index data of the first voice information from a pre-established data set, The data set includes attribute key words and attribute indexes; based on the association relationship of each object, the target area information corresponding to the first index data is determined, and the object corresponding to the target area information is the target object;

The association relationship is a mapping relationship between the second index data of the attribute information of the object and the area information, and the area information is used to describe the position of the object in the original image collected by the camera.

In some possible implementations of the second aspect, before determining the target area information corresponding to the first index data based on the association relationship of each object, the method further includes: determining object information of each object in the original image, where the object information includes area information and attribute information; extract second index data of attribute information of each object from the data set; for each object, associate the second index data with the area information to obtain the association relationship between the index data and the area information.

In some possible implementations of the second aspect, the above process of extracting the second index data of the attribute information of each object from the data set may include: for each attribute information, searching the data set for a first key matching the attribute information word; the index data of the first key word in the data set is used as the second index data.

In some possible implementations of the second aspect, the above process of determining object information of each object in the original image may include: inputting the original image into the target recognition model; and obtaining object information output by the target recognition model.

In some possible implementations of the second aspect, the above-mentioned process of determining the target area information corresponding to the first index data based on the association relationship of each object may include: for each second index data, Each index is respectively matched with each index in the second index data, and the number of items of the target index is determined, and the target index is an index in the second index data that matches the index in the first index data; The area information corresponding to the second index data with the largest number of items is determined as the target area information.

In some possible implementations of the second aspect, the above process of extracting the first index data of the first voice information from the pre-established data set may include: performing voice recognition on the first voice information to obtain a first voice recognition result; According to the first speech recognition result, it is determined that the first speech information is a speech command for specifying the target object; the second key word that matches the first speech recognition result is searched from the attribute key words corresponding to the second index data; The index data of the second key word in the data set is used as the first index data.

In some possible implementations of the second aspect, the above-mentioned process of performing speech recognition on the first voice information to obtain the first voice recognition result may include: extracting voiceprint features of the first voice information; determining the voiceprint features and pre-storing Similarity between voiceprint features; when the similarity is greater than or equal to a preset threshold, input the voiceprint feature and the first speech information into the semantic understanding model to obtain the first speech recognition result output by the semantic understanding model.

In some possible implementations of the second aspect, the cameras include N cameras with different viewing angles, where N is a positive integer greater than or equal to 1.

After determining the target area information corresponding to the first index data based on the association relationship of each object, the method further includes: obtaining a first output image of the target object according to the target area information based on N original images with different viewing angles, and Display the first output image in the viewfinder.

In some possible implementations of the second aspect, the above-mentioned process of obtaining the first output image of the target object according to the target area information based on the original images of N different viewing angles may include: determining the first output image from the N original images. target image, the first target image is the image with the smallest field of view in the image set, and the image set includes at least one original image that can completely contain the bounding box selection area of the target object; according to the target area information, crop out from the first target image a first sub-image of the target object; generating a first output image based on the first sub-image.

In some possible implementations of the second aspect, when there are at least two target objects, and each target object corresponds to a first sub-image, the above-mentioned process of obtaining the first output image based on the first sub-image may include: : Splicing the first sub-images corresponding to each target object to obtain the first output image.

In some possible implementations of the second aspect, after displaying the first output image in the viewfinder, the method further includes: acquiring second voice information; and performing voice recognition on the second voice information to obtain a second voice recognition result ; According to the second voice recognition result, determine that the second voice information is a voice command for adjusting the display mode of the target object; Determine the second target image from the original image; Cut out the second child of the target object from the second target image image; generating a second output image based on the second sub-image, and displaying the second output image in the viewfinder.

In some possible implementations of the second aspect, the method further includes: in the third target image, determining the distance between each edge of the bounding box selection area of the target object and the corresponding edge of the image, the third target image The image is the image with the largest field of view in the image set; if the minimum value among the multiple distances is less than the preset distance threshold, prompt information is displayed in the viewfinder, and the prompt information is used to prompt adjustment of the camera pose.

In a third aspect, an embodiment of the present application provides a shooting method, which is applied to a terminal device. In the method, a picture captured by a camera is displayed in a viewfinder, and the picture includes at least one object; when a first instruction is received, the first instruction is used for Designate at least two target objects, and determine at least two objects in the picture corresponding to the first instruction as at least two target objects; then, display the picture of each target object in sub-areas in the viewfinder, wherein one area displays one The picture of the target object.

In some possible implementations of the third aspect, the first instruction is a voice instruction.

In some possible implementations of the third aspect, the above process of determining the at least two objects in the screen corresponding to the first instruction as the at least two target objects may include: extracting the first instruction of the first instruction from a pre-established data set 1. index data, the data set includes attribute keywords and attribute indexes; based on the association relationship of each object, the target area information corresponding to the first index data is determined, and the object corresponding to the target area information is the target object;

Wherein, the association relationship is the mapping relationship between the second index data of the attribute information of the object and the region information. The region information is used to describe the position of the object in the original image collected by the camera.

In some possible implementations of the third aspect, before determining the target area information corresponding to the first index data based on the association relationship of each object, the method further includes: determining object information of each object in the original image, where the object information includes area information and attribute information; extract second index data of attribute information of each object from the data set; for each object, associate the second index data with the area information to obtain the association relationship between the index data and the area information.

In some possible implementations of the third aspect, the above process of extracting the second index data of the attribute information of each object from the data set may include: for each attribute information, searching for the first key matching the attribute information from the data set word; the index data of the first key word in the data set is used as the second index data.

In some possible implementations of the third aspect, the above process of determining object information of each object in the original image may include: inputting the original image into the target recognition model; and obtaining object information output by the target recognition model.

In some possible implementations of the third aspect, the above-mentioned process of determining the target area information corresponding to the first index data based on the association relationship of each object may include: for each second index data, Each index is respectively matched with each index in the second index data, and the number of items of the target index is determined, and the target index is an index in the second index data that matches the index in the first index data; The area information corresponding to the second index data with the largest number of items is determined as the target area information.

In some possible implementations of the third aspect, the above process of extracting the first index data of the first instruction from the pre-established data set may include: searching attribute keywords corresponding to the second index data for the first instruction The matching second key word; the index data of the second key word in the data set is used as the first index data.

In some possible implementations of the third aspect, the cameras include N cameras with different field of view, where N is a positive integer greater than or equal to 1; the process of displaying the picture of each target object in sub-regions in the viewfinder frame may include: : For each target object, determine the first target image from N original images, the first target image is the image with the smallest field of view in the image set, and the image set includes at least one image that can completely contain the bounding box selection area of the target object. The original image; for each target object, according to the target area information, the first sub-image of the target object is cropped from the first target image; the first sub-image corresponding to each target object is stitched to obtain a first output image; Display the first output image in the viewfinder.

In some possible implementations of the third aspect, after displaying the picture of each target object in sub-regions in the viewfinder frame, the method further includes: acquiring second voice information; performing voice recognition on the second voice information to obtain the second voice information. Voice recognition result; according to the second voice recognition result, determine that the second voice information is a voice command for adjusting the display mode of the target object; determine the second target image from the original image; second sub-image; generating a second output image based on the second sub-image, and displaying the second output image in the viewfinder.

In some possible implementations of the third aspect, the method further includes: in the third target image, determining the distance between each edge of the bounding box selection area of the target object and the corresponding edge of the image, the third target image The image is the image with the largest field of view in the image set; if the minimum value among the multiple distances is less than the preset distance threshold, prompt information is displayed in the viewfinder, and the prompt information is used to prompt adjustment of the camera pose.

In a fourth aspect, an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, the above-mentioned first aspect or the second The method of any of the aspect or the third aspect.

In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned first aspect or second aspect or third aspect is implemented item method.

In a sixth aspect, an embodiment of the present application provides a chip system, the chip system includes a processor, the processor is coupled to a memory, and the processor executes a computer program stored in the memory to implement the first aspect or the second aspect or the first aspect. The method of any one of the three aspects. The chip system may be a single chip, or a chip module composed of multiple chips.

In a seventh aspect, an embodiment of the present application provides a computer program product that, when the computer program product runs on a terminal device, enables the terminal device to execute the method described in any one of the first aspect or the second aspect or the third aspect.

It can be understood that, for the beneficial effects of the foregoing second aspect to the seventh aspect, reference may be made to the relevant descriptions in the foregoing first aspect, which will not be repeated here.

Description of drawings

FIG. 1 is a schematic diagram of a hardware structure of a terminal device 100 according to an embodiment of the present application;

FIG. 2 is a block diagram of a software structure of a terminal device 100 according to an embodiment of the application;

3 is a schematic block diagram of a flow of a photographing method provided by an embodiment of the present application;

4 is a schematic diagram of a tracking shooting in a video recording scene provided by an embodiment of the present application;

5 is a schematic diagram of stable tracking provided by an embodiment of the present application;

FIG. 6 is a schematic diagram of a tracking shooting of multi-target tracking provided by an embodiment of the present application;

7A is a schematic diagram of an image processing process in a tracking shooting scene provided by an embodiment of the present application;

7B is a schematic diagram of an interface in a tracking shooting scene provided by an embodiment of the present application;

8 is a schematic diagram of replacing a target object provided by an embodiment of the present application;

9A is a schematic diagram of another image processing process in a tracking shooting scene provided by an embodiment of the present application;

9B is a schematic diagram of another interface in a tracking shooting scene provided by an embodiment of the present application;

FIG. 9C is a schematic diagram of an original image set provided by an embodiment of the present application;

FIG. 9D is a schematic diagram of a sub-image provided by an embodiment of the present application;

10 is a schematic diagram of tracking shooting in a shooting scene provided by an embodiment of the present application;

11 is a schematic block diagram of a shooting process provided by an embodiment of the present application;

FIG. 12 is a schematic flowchart of a voice shooting process provided by an embodiment of the present application;

FIG. 13 is a schematic flowchart of a method for determining a target object provided by an embodiment of the present application;

FIG. 14 is another schematic flowchart of the method for determining a target object provided by an embodiment of the present application;

FIG. 15 is another schematic flow diagram of the photographing method provided by the embodiment of the present application;

16 is a schematic block diagram of an apparatus for determining a target object provided by an embodiment of the present application;

17 is another schematic block diagram of an apparatus for determining a target object provided by an embodiment of the present application;

FIG. 18 is a schematic block diagram of a photographing apparatus provided by an embodiment of the present application.

Detailed ways

In the following description, for the purpose of illustration rather than limitation, specific details such as a specific system structure and technology are set forth in order to provide a thorough understanding of the embodiments of the present application.

The following first exemplarily introduces terminal devices that may be involved in the embodiments of the present application.

Referring to FIG. 1 , it is a schematic block diagram of a hardware structure of a terminal device 100 according to an embodiment of the present application. As shown in FIG. 1 , the terminal device 100 may include a processor 110, a memory 120, an audio module 130, a speaker 130A, a receiver 130B, a microphone 130C, a camera 140, a display screen 150, and a sensor module 160. The sensor module 160 may include, but is not limited to, a touch sensor 160A and the like.

It can be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the terminal device 100 . In other embodiments of the present application, the terminal device 100 may include more or less components than those shown in the drawings, or combine some components, or separate some components, or arrange different components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

For example, in some embodiments, the terminal device 100 may further include at least one of the following: an earphone interface, a button, a motor, an indicator, and a subscriber identification module (SIM) card interface 1, an external memory interface, a universal string Line bus (universal serial bus, USB) interface, charging management module, power management module, battery, antenna, mobile communication module, and wireless communication module, etc.

The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (application processor, AP), a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP) ), controller, memory, video codec, digital signal processor (DSP), and/or neural-network processing unit (NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

The controller may be the nerve center and command center of the terminal device 100 . The controller can generate an operation control signal according to the instruction operation code and timing signal, and complete the control of fetching and executing instructions. A memory may also be provided in the processor 110 for storing instructions and data.

In some embodiments, the processor 110 may include one or more interfaces. For example, the interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a mobile industry processor interface (mobile industry processor interface, MIPI), general-purpose input and output ( general-purpose input/output, GPIO) interface.

Among them, the I2C interface is a bidirectional synchronous serial bus, including a serial data line (serial data line, SDA) and a serial clock line (derail clock line, SCL). In some embodiments, the processor 110 may contain multiple sets of I2C buses. The processor 110 may be respectively coupled to the touch sensor 160A, the camera 140 and the like through different I2C bus interfaces. For example, the processor 110 may couple the touch sensor 160A through the I2C interface, so that the processor 110 communicates with the touch sensor 160A through the I2C bus interface, so as to realize the touch function of the terminal device 100 .

The I2S interface can be used for audio communication. In some embodiments, the processor 110 may contain multiple sets of I2S buses. The processor 110 may be coupled with the audio module 130 through an I2S bus to implement communication between the processor 010 and the audio module 130 .

The MIPI interface can be used to connect the processor 110 with the display screen 150, the camera 140 and other peripheral devices. MIPI interfaces include camera serial interface (CSI), display serial interface (DSI), etc. In some embodiments, the processor 110 communicates with the camera 140 through a CSI interface to implement the shooting function of the terminal device 100 . The processor 110 communicates with the display screen 150 through the DSI interface to implement the display function of the terminal device 100 .

The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface may be used to connect the processor 110 with the camera 140, the display screen 150, the audio module 130, the sensor module 160, and the like. The GPIO interface can also be configured as I2C interface, I2S interface, MIPI interface, etc.

It can be understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is only a schematic illustration, and does not constitute a structural limitation of the terminal device 100 . In other embodiments of the present application, the terminal device 100 may also adopt different interface connection manners in the foregoing embodiments, or a combination of multiple interface connection manners.

The terminal device 100 implements a display function through a GPU, a display screen 150, an application processor, and the like. The GPU is a microprocessor for image processing, and is connected to the display screen 150 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or alter display information.

The display screen 150 is used to display images, videos, and the like. The display screen 150 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (active-matrix organic light). emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diode (quantum dot light emitting diodes, QLED) and so on. In some embodiments, the terminal device 100 may include 1 or N display screens 150 , where N is a positive integer greater than 1.

The terminal device 1000 can realize the shooting function through the ISP, the camera 140, the video codec, the GPU, the display screen 150, and the application processor.

The ISP is used to process the data fed back by the camera 140 . For example, when taking a photo, the shutter is opened, the light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing, converting it into an image visible to the naked eye. ISP can also perform algorithm optimization on image noise, brightness, and skin tone. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP may be provided in the camera 140 .

Camera 140 is used to capture still images or video. The object is projected through the lens to generate an optical image onto the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. DSP converts digital image signals into standard RGB, YUV and other formats of image signals. In some embodiments, the terminal device 100 may include 1 or N cameras 140 , where N is a positive integer greater than 1.

When the terminal device 100 includes at least two cameras 140, the at least two cameras 140 may include cameras with different viewing angles. For example, the terminal device 100 includes a wide-angle camera, a main camera, and a telephoto camera. The three cameras have different viewing angles, and images with different viewing angles can be collected through cameras with different viewing angles.

A digital signal processor is used to process digital signals, in addition to processing digital image signals, it can also process other digital signals.

Video codecs are used to compress or decompress digital video. The terminal device 100 may support one or more video codecs. In this way, the terminal device 100 can play or record videos in various encoding formats, for example, moving picture experts group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4 and so on.

The NPU is a neural-network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transfer mode between neurons in the human brain, it can quickly process the input information, and can continuously learn by itself. Applications such as intelligent cognition of the terminal device 100 can be implemented through the NPU, such as image recognition, face recognition, speech recognition, text understanding, and the like.

The memory 120 may include internal memory and/or external memory, and the internal memory may be used to store computer-executable program code, which includes instructions. The processor 110 executes various functional applications and data processing of the terminal device 100 by executing the instructions stored in the internal memory. The internal memory may include a program storage area and a data storage area. The storage program area can store an operating system, an application program required for at least one function (such as a sound playback function, an image playback function, etc.), and the like. The storage data area may store data and the like created during the use of the terminal device 100 . In addition, the internal memory may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (UFS), and the like.

The terminal device 100 may implement audio functions through an audio module 130, a speaker 130A, a receiver 130B, a microphone 130C, an application processor, and the like. Such as music playback, recording, etc.

The audio module 130 is used for converting digital audio information into analog audio signal output, and also for converting analog audio input into digital audio signal. Audio module 130 may also be used to encode and decode audio signals. In some embodiments, the audio module 130 may be provided in the processor 110 , or some functional modules of the audio module 130 may be provided in the processor 110 .

Speaker 130A, also referred to as a "speaker", is used to convert audio electrical signals into sound signals. The terminal device 100 can listen to music through the speaker 130A, or listen to a hands-free call, or send out voice prompt information and the like.

The receiver 130B, also referred to as "earpiece", is used to convert audio electrical signals into sound signals. When the terminal device 100 answers a call or a voice message, the voice can be received by placing the receiver 130B close to the human ear.

The microphone 130C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound by approaching the microphone 130C through a human mouth, and input the sound signal into the microphone 130C. The terminal device 100 may be provided with at least one microphone 130C.

The touch sensor 160A is also called "touch panel". The touch sensor 160A may be disposed on the display screen 150 , and the touch sensor 160A and the display screen 150 form a touch screen, also referred to as a “touch screen”. The touch sensor 160A is used to detect touch operations on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to touch operations may be provided through display screen 150 . In other embodiments, the touch sensor 160A may also be disposed on the surface of the terminal device 100 , which is different from the position where the display screen 150 is located.

After the hardware architecture of the terminal device 100 is introduced, the software architecture of the terminal device 100 will be exemplarily introduced below with reference to FIG. 2 .

The software system of the terminal device 100 may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. The embodiments of the present application take an Android system with a layered architecture as an example to exemplarily describe the software structure of the terminal device 100 .

FIG. 2 is a block diagram of a software structure of a terminal device 100 according to an embodiment of the present application.

The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, which are, from top to bottom, an application layer, an application framework layer, an Android runtime (Android runtime) and system libraries, and a kernel layer.

The application layer can include a series of application packages.

As shown in Figure 2, the application package can include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and so on.

The application framework layer provides an application programming interface (application programming interface, API) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

As shown in Figure 2, the application framework layer may include window managers, content providers, view systems, telephony managers, resource managers, notification managers, and the like.

A window manager is used to manage window programs. The window manager can get the size of the display screen, determine whether there is a status bar, lock the screen, take screenshots, etc. Content providers are used to store and retrieve data and make this data accessible to applications. The data may include video, images, audio, calls made and received, browsing history and bookmarks, phone book, etc.

The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, and so on. View systems can be used to build applications. A display interface can consist of one or more views. For example, the display interface including the short message notification icon may include a view for displaying text and a view for displaying pictures.

The telephony manager is used to provide the communication function of the terminal device 100 . For example, the management of call status (including connecting, hanging up, etc.). The resource manager is used to provide various resources for the application, such as localization strings, icons, pictures, layout files, video files and so on. The notification manager enables applications to display notification information in the status bar, which can be used to convey notification-type messages, and can automatically disappear after a brief pause without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also display notifications in the status bar at the top of the system in the form of graphs or scroll bar text, such as notifications of applications running in the background, and notifications that appear on the screen in the form of dialog windows. For example, text information is prompted in the status bar, a prompt sound is issued, the electronic device vibrates, and the indicator light flashes.

Android Runtime includes core libraries and a virtual machine. Android runtime is responsible for scheduling and management of the Android system.

The core library consists of two parts: one is the function functions that the java language needs to call, and the other is the core library of Android.

The application layer and the application framework layer run in virtual machines. The virtual machine executes the java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, safety and exception management, and garbage collection.

A system library can include multiple functional modules. For example: surface manager (surface manager), media library (Media Libraries), 3D graphics processing library (eg: OpenGL ES), 2D graphics engine (eg: SGL), etc. The Surface Manager is used to manage the display subsystem and provides a fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of a variety of commonly used audio and video formats, as well as still image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing. 2D graphics engine is a drawing engine for 2D drawing. The kernel layer is the layer between hardware and software. The kernel layer contains at least display drivers, camera drivers, audio drivers, and sensor drivers.

The software and hardware workflows of the terminal device 100 are exemplarily described below with reference to the capturing and photographing scene.

When the touch sensor 160A receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes touch operations into raw input events (including touch coordinates, timestamps of touch operations, etc.). Raw input events are stored at the kernel layer. The application framework layer obtains the original input event from the kernel layer, and identifies the control corresponding to the input event. Taking the touch operation as a touch click operation, and the control corresponding to the click operation is the control of the camera application icon, as an example, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer, and then starts the camera driver by calling the kernel layer. Camera 140 captures still images or video.

In a specific application, the terminal device 100 may be a mobile phone, a tablet computer, or other types of terminal devices, and the specific type of the terminal device 100 is not limited in this embodiment of the present application.

After the hardware architecture and software architecture of the terminal device 100 are exemplarily introduced, the technical solutions provided by the embodiments of the present application are described in detail below by taking the terminal device 100 as an example.

Please refer to FIG. 3. FIG. 3 is a schematic flow diagram of a photographing method provided by an embodiment of the present application. The method is applied to the terminal device 100, and the method may include the following steps:

Step S301 , start the shooting application, enter the video recording mode or the photographing mode, and display the captured image in the viewfinder.

It can be understood that the above shooting applications refer to shooting applications, which are usually camera applications.

After detecting the operation for starting the shooting application, the terminal device 100 starts the shooting application in response to the operation, and invokes the camera driver through the kernel layer to capture the image through the camera and display the image in the viewfinder.

For example, referring to a schematic diagram of a tracking shooting in a video recording scene shown in FIG. 4, as shown in FIG. 4, the main interface 42 of the mobile phone 41 displays applications such as the camera 43, smart life, settings, calendar, gallery, and clock, etc. At this time, the terminal device 100 is embodied as a mobile phone 41 , and the photographing application is embodied as a camera 43 .

After receiving the click operation on the camera 43, the mobile phone 41 starts the camera 43 in response to the click operation. After the mobile phone 41 enters the video recording mode, the video recording preview interface 44 is displayed, and the object 46 , the object 47 , the object 48 , and the object 49 are displayed in the viewfinder frame of the video recording preview interface 44 . In addition, the video preview interface 44 also includes an icon 45, the icon 45 is used to turn on and off the tracking shooting mode, that is, the user can click the icon 45 to enter or exit the tracking shooting mode. At this time, the icon 45 indicates that the mobile phone 41 has not entered the tracking shooting mode.

In this embodiment of the present application, tracking shooting may also be referred to as tracking shooting or target tracking. When tracking the movement of the target, the tracking shooting can keep the tracking target in the center of the screen as much as possible without using a mobile terminal device or manually adjusting the focal length, avoiding the screen shake caused by the user's mobile terminal device.

Step S302, a trigger operation is detected, and in response to the trigger operation, the voice shooting mode is entered.

It should be noted that the above trigger operation is used to trigger the voice shooting mode, and the trigger operation may include operations such as single-click, double-click, or long-press. For example, the trigger operation is a click operation. After the mobile phone 41 in FIG. 4 receives the user's click operation on the icon 45 , it enters the tracking shooting mode. In the tracking shooting mode, the mobile phone 41 can call the microphone to collect voice information in real time, so as to realize functions such as allowing the user to specify the target object through voice, that is, after the mobile phone 41 enters the tracking shooting mode, it can be regarded as entering the voice shooting mode. Of course, the mobile phone 41 can also enter and exit the voice shooting mode through other icons or buttons.

The trigger operation can also be a voice command, that is, the user can turn on and off the tracking shooting mode through a voice command. For example, after the terminal device 100 receives the voice "turn on voice photography" or "turn on voice video", it performs voice recognition on the voice to obtain a voice recognition result. When the voice recognition result includes a preset keyword, it can be determined that the voice is A voice command for triggering the voice shooting mode; and then in response to the voice command, the voice shooting mode is entered. The preset keywords may be preset, for example, the preset keywords may include "open", "enter", "photograph", "shoot", and "video".

After the terminal device 100 enters the voice shooting mode, it performs target recognition on original images with different viewing angles, and collects voice information in real time through a microphone to identify the voice information.

In some other embodiments, after the terminal device 100 starts the shooting application, it can also automatically enter the voice shooting mode, and it is no longer necessary to perform the voice shooting mode through a trigger operation. In other words, the above step S302 is an optional step.

In step S303, target recognition is performed on the N original images respectively, and object information of each object in each original image is obtained.

Wherein, the terminal device 100 includes N cameras with different field of view, and N pieces of original images with different field of view are obtained through N cameras with different field of view, where N is a positive integer greater than or equal to 1. For example, when the terminal device 100 is a mobile phone, the mobile phone may include a wide-angle camera, a telephoto camera, and a main camera, and the three cameras have different field angles. Through the three cameras with different field of view, the mobile phone can collect three original images with different field of view.

When the terminal device 100 includes at least two cameras with different field of view, the terminal device 100 can call N cameras with different field of view to collect N original images with different field of view when starting the shooting application; When starting the shooting application, only one camera is called for image acquisition, and the collected image is displayed in the viewfinder of the preview interface. After entering the voice shooting mode, N cameras are called to collect N different views. Original image of field angle.

After collecting the N original images, the terminal device 100 may perform target recognition on each original image to obtain object information of each object in each original image. Each original image may contain one or more objects, which can include people, animals, and text.

Object information includes attribute information and area information. The area information is used to represent the area where the object is located in the original image, that is, the area information can be used to represent the location and size of the object in the original image. In general, the area information may be pixel coordinate information of the bounding box selection area of the object, and the bounding box selection area is usually a rectangular area, and of course it may be an area of other shapes. Of course, the area information may also be coordinate information of the area where the object is actually located in the image.

Attribute information can be used to describe the object, and the attribute information includes, but is not limited to, type, gender, age group, activity information, hair length, hair color, clothing type, clothing color, and posture. Different objects may contain different types of properties. Exemplarily, for a character, the types of attribute information include gender, age group, and clothing color. For animals, the types of attribute information include categories and colors.

The type refers to the category of the object, and the category may exemplarily include characters, animals, and characters. The activity in the attribute information is used to describe the current action of the object, which may exemplarily include skateboarding, running, and hula hooping. That is, through the activity information in the attribute information, it can be known whether the current action of the object is running or playing hula hoop. Gestures can be used to describe the current pose of an object, for example, a pose can tell whether a person is standing or sitting.

In some embodiments, the terminal device 100 may respectively input N original images into the target recognition model to obtain the object information output by the target recognition model. The object recognition model may be pre-built and pre-trained. The target recognition model can include a target detection network model and a target classification network model, etc., which are used to identify the image to identify the attribute information of the objects in the image, as well as the location and size of each object in the image. Wait. Of course, the terminal device 100 may also perform target recognition in other ways, for example, by combining image semantic segmentation and instance segmentation, etc., to recognize the object information of each object in the image.

Taking FIG. 4 as an example, after the mobile phone 41 collects the original image corresponding to the video preview interface 44, the original image can be input into the target recognition model to obtain object information of each object output by the target recognition model. Exemplarily, the mobile phone 41 can identify the information of the object 47 included in the original image, and through the information of the object 47, it can be determined that the object is a woman and is playing hula hoops, as well as its position and size in the original image.

The terminal device 100 may perform target recognition on each frame of the collected original image in real time, or may perform target recognition every preset number of frames.

Step S304 , extract the second index data of the attribute information of each object from the pre-established data set.

It should be noted that the above data set is a pre-established database of common user information, which may be embodied as a database, and the data set is preset with different attribute indexes for different object types. For example, for character types, it may include, but is not limited to, the following attribute categories: gender, age group, hair length, hair color, clothing type, clothing color, posture, and activity. A corresponding attribute index is set under each attribute type. In addition to characters, object types can also include but are not limited to animals, characters, and the like. For the convenience of description, the following description will take a database as an example.

As an example and not a limitation, the attribute indexes of the character types in the database may be as shown in Table 1 below.

Table 1

The terminal device 100 extracts the attribute information of each object in each original image, and matches the attribute information of each object with the attribute keywords in the corresponding attribute types in the database, so as to find the attribute information in the database that matches the attribute information. The first keyword that matches. The first key term may include words and/or words. If the first keyword matching the attribute information is found in the database, the index data of the first keyword is used as the second index data. Taking the scene shown in FIG. 4 as an example, the object 46 is a little boy playing a skateboard, and the object is a character type. In Table 1 above, the attribute category index corresponding to the character type includes A to N.

First, the mobile phone 41 obtains the attribute information and area information of the object 46 through target recognition. The attribute information may include: male, child, skateboard, short hair, black hair, white T-shirt, . . . and standing.

The attribute information of the object 46 is matched with the attribute key words in the corresponding attribute type in the database. Specifically, for the gender attribute category, the "male" in the attribute information is matched with the attribute keyword corresponding to 001A and the keyword corresponding to 002A in Table 1 above, and it is determined that "male" in the attribute information is ” matches property index 001A.

For the age group attribute types, the "children" in the attribute information are respectively corresponding to the attribute key words corresponding to 001B, the attribute key words corresponding to 002B, the attribute key words corresponding to 003B, and the attribute key words corresponding to 004B in Table 1 above. The attribute key words are matched, and the attribute index matching "little boy" is determined to be 003B.

By analogy, the attribute types such as skateboard, short hair, black hair, white T-shirt, ..., and standing in the attribute information are matched with the attribute keywords of the corresponding attribute types in Table 1 above to obtain the object. The attribute index corresponding to the attribute information of 46 is obtained, that is, the second index data of the object 46 is obtained.

By way of example and not limitation, the property indexes of object 46 are shown in Table 2 below.

Table 2

Similarly, the property index of object 47 and the property index of object 48 can be obtained.

Among them, the attribute index of the object 47 is shown in Table 3 below.

table 3

The property indexes of object 48 are shown in Table 4 below.

Table 4

Step S305 , for each object, associate the second index data with the area information to obtain an association relationship of each object, where the association relationship is an association relationship between the index data and the area information.

For each object, after extracting the second index data of the attribute information of the object, the terminal device 100 associates the second index data of the object with the region information of the object in each original image to establish each object The relationship between the index data and the region information of each original image. Based on the association relationship, the region information of the object can be found through the index data. Taking the scene of FIG. 4 as an example, the second index data of the object 46 in FIG. 4 may be as shown in Table 2 above. The mobile phone 41 includes a wide-angle camera, a main camera and a telephoto camera, and the wide-angle camera has the largest field of view, the main camera has the second largest field of view, and the telephoto camera has the smallest field of view. It is assumed that the original image captured by the wide-angle camera and the main camera includes the object 46 , while the original image captured by the telephoto camera does not include the object 46 . In this case, the area information of the object refers to the pixel coordinate information of the bounding box selection area of the object.

The second index data of the object 46 in the above table 2 are respectively associated with the coordinates of the bounding box selection area of the object 46 in the original image collected by the wide-angle camera, and the bounding box selection area of the object 46 in the original image collected by the main camera. The coordinate association establishes the association relationship between the index data of the object 46 and the area information.

As an example and not a limitation, the association relationship of the objects 46 is shown in Table 5 below.

table 5

It can be seen from Table 5 above that the index data of the object 46 includes 001A, 003B, 001C and 002N, etc. Through the index data, the area information of the object 46 in the original image collected by the wide-angle camera, and the original image collected by the main camera of the object 46 can be found. Region information in the image.

Exemplarily, the processor 110 of the terminal device 100 performs target recognition on the original image captured by the camera 140 to obtain object information of each object in the original image; the processor 110 reads the database in the memory 120 and extracts each object from the database. The second index data of the attribute information of the object; for each object, the processor 110 associates the second index data with the area information, obtains the association between the index data and the area information, and stores the association in the memory 120 .

At a certain moment, after the processor 110 collects the first voice through the microphone 130C and the audio module 130, it performs voice recognition on the first voice to obtain the first voice recognition result; the processor 110 then reads the database from the memory 120, The first index data of the first speech recognition result is extracted from the database, the pre-established association relationship is read, and the target area information corresponding to the first index data is determined according to the first index data and the association relationship.

Step S306, detecting the first voice information.

Specifically, after entering the voice shooting mode, the terminal device 100 can call the microphone to collect the first voice information in real time.

Step S307: Perform speech recognition on the first speech information to obtain a first speech recognition result.

After collecting the first voice information, the terminal device 100 can extract the first voiceprint feature of the first voice information, and then determine the similarity between the first voiceprint feature and the pre-stored voiceprint feature. If the similarity between the feature and the pre-stored voiceprint feature is greater than the preset threshold, it is considered that the first voice information is the voice of a specific user, then voice recognition is performed according to the first voiceprint feature and the first voice information to obtain the first voice recognition result.

When the similarity between the first voiceprint feature and the pre-stored voiceprint feature is less than or equal to the preset threshold, the terminal device 100 may prompt the user to have no permission, or may store the first voiceprint feature for the next voice command identification.

In a specific application, the terminal device 100 may input the first voiceprint feature and the first voice information into the semantic understanding model, and obtain the first voice recognition result output by the semantic understanding model. The semantic understanding model may include a voice feature analysis module and a semantic analysis module, the voice feature analysis module may be used to filter background interference sounds, and the semantic analysis module may be used to extract user voice commands.

In other embodiments, the terminal device 100 may directly recognize the first voice information without extracting the voiceprint feature of the first voice information, that is, without distinguishing whether the first voice information is the voice of a specific user, to obtain The first speech recognition result.

Step S308, when it is determined according to the first speech recognition result that the first speech information is the speech used to specify the target object, extract the first index of the first speech recognition result from the data set based on the attribute keywords corresponding to the second index data data.

After obtaining the first voice recognition result, the terminal device 100 may further confirm whether the first voice information is a voice command. When the first voice recognition result includes a specific keyword, the first voice information may be considered as the voice for specifying the target object, and the specific keyword is preset. For example, the specific keyword may include "shooting" , Track, and Take Photo.

The first voice information can be a voice command for specifying the target object for the first time, or a voice command for increasing or decreasing the target object, or a voice command for replacing the target object, all of which can be considered to be used for specifying the target object. voice commands. Adding or subtracting target objects and replacing target objects are relative to the specified target objects.

For example, the voice "follow the little boy playing skateboard" collected in Fig. 4 is the voice command for specifying the target object for the first time. After the interface 412 is displayed, the target object can be replaced from the object 46 to other objects by collecting the first voice information, and other objects can also be added or subtracted as the target object based on the object 46 .

When the first voice information is not the voice for specifying the target object, and the first voice information is not a specific voice command, the terminal device 100 may prompt the user to re-input the voice, or may not perform a prompting operation. The specific voice command may exemplarily include commands to adjust the display size and display area of the tracked object, and the like.

When the first voice information is not a voice for specifying the target object, but a specific voice command, the terminal device 100 may perform an operation corresponding to the specific voice command.

When the first voice information is the voice used to specify the target object, the terminal device 100 may match the first voice recognition result with the attribute keywords corresponding to the second index data, and find out the voice that matches the first voice recognition result. The second key word, and then the attribute index corresponding to the second key word that matches the first speech recognition result is used as the index data of the first speech recognition result.

It should be noted that, in some other embodiments, the first speech recognition result may also be matched with each key word in the database. However, compared with matching with all the attribute keywords in the database, the matching with the attribute keywords corresponding to the second index data reduces the matching range and improves the matching accuracy and matching speed.

For example, referring to FIG. 4 , the mobile phone 41 displays the video recording preview interface 44 in response to the user operation, and after entering the voice shooting mode, the mobile phone 41 respectively performs target recognition on the N original images collected, and obtains the information of each object in each original image. object information, and extract the second index data of the attribute information of each object from the database, and establish an association relationship between the index data and the area information. Then, at a certain moment, the mobile phone 41 collects the first voice information through the microphone, and the first voice information is "tracking the little boy playing skateboard". After determining that the first voice information is the voice of a specific user according to the pre-stored voiceprint feature, the mobile phone 41 performs voice recognition on the first voice information, and obtains the first voice recognition result as "tracking the boy playing skateboard". The recognition result contains the keyword "tracking", so it is determined that the voice is a voice command used to specify the target object, and then the index data of "skateboard" and "boy" are searched from the attribute keywords corresponding to the second index data to obtain The first index data of the first speech recognition result. As shown in Table 1 above, it can be obtained that the index data of "skateboard" is 001C, and the index data of "boy" are 001A and 003B, that is, the first index data includes 001C, 001A and 003B.

Step S309: Determine the target area information corresponding to the first index data according to the association relationship between the index data and the area information, and the object corresponding to the target area information is the target object.

Based on the association relationship, the terminal device 100 searches the index data set for an index matching the first index data, determines the object with the largest number of matching items as the target object, and determines the area information corresponding to the object as the target area information.

If no index matching the first index data can be found in the index data set, the terminal device 100 may prompt the user that no object matching the voice can be found. The prompt mode may include voice prompt and/or text prompt. For example, the terminal device can send out a voice "No matching object can be found, please re-enter the voice", and the text corresponding to the voice can also be displayed in the viewfinder.

The index data set refers to a collection of second index data including each object. For example, taking the scenario of FIG. 4 as an example, the index data set may include the second index data of object 46 , the second index data of object 47 , the second index data of object 48 , and the second index data of object 49 .

If the first speech recognition result contains only one object, but at least two target objects are matched, the terminal device 100 may prompt the user for further selection. The prompt information may be text prompt information, voice prompt information, or other types of prompt information. For example, the first speech recognition result is "following the little boy playing skateboard", that is, the first speech recognition result only contains the object "little boy", but two target objects are matched. At this time, the mobile phone can let the user execute Click the operation, or let the user re-input the voice to further determine the target object that the user wants to specify from the two target objects.

In the process of searching the target area information corresponding to the first index data, the terminal device 100 uses the area information with the most matching items as the target area information corresponding to the first index data.

For example, taking the scene in FIG. 4 as an example, at this time, the first speech recognition result is "tracking a boy playing skateboard", the index data corresponding to "skateboard" is 001C, and the index data corresponding to "boy" are 001A and 003B. That is, the first index data includes 001C, 001A, and 003B.

Based on the index data set, it can be known that the second index data corresponding to the area information of the object 46 includes 001A, 003B, 001C, 002D, 005E, 008F and 002N, and the second index data corresponding to the area information of the target object 48 includes 001A, 003B , 003C and 002D, 005E, 001F and 002N.

By comparison, it can be seen that in the second index data of the object 46, the target indexes that match the indexes in the first all data include 001C, 001A, and 003B, that is, the number of items that match the first index data is 3. That is, the number of items of the target index is 3; and in the second index data of the object 48, the target index that matches the indexes in the first all data only includes 001A and 003B, and does not include 001C, that is, the first index data The number of matching items is 2.

Similarly, it can be known that the number of matching items of object 47 and object 49 are both 0.

That is to say, the number of items of the target index of the object 46 is the largest, so the area information of the object 46 is used as the target area information, and the object 46 is used as the target object. Referring to Table 5 above, it can be known that the target area information includes the coordinate information of the bounding box selection area of the original image collected by the wide-angle camera, and the coordinate information of the bounding frame selection area of the original image collected by the main camera.

In some embodiments, after determining the target object specified by the user's voice, the terminal device 100 may use a corresponding mark to identify the target object, so as to let the user know the target object. The marking method can be, but not limited to, a rectangular frame. For example, referring to FIG. 4 , after the mobile phone 41 collects the voice "tracking the boy playing skateboard", performs voice recognition on the voice, and determines the target area information, it can determine that the target object specified by the user is the object 46, and according to the target The area information can determine the area where the object 46 is located in the original image, and a rectangular frame can be used to mark the object 46, that is, the rectangular frame 411 is used to frame the object 46 in the video preview interface 410, and the target object designated by the user's voice is prompted as the object 46.

Of course, in some other embodiments, after determining the target object and the target area information, the mobile phone 41 may not use the corresponding mark to identify the target object.

Step S310: Determine the target image from the N original images, and then crop a sub-image from the target image according to the target area information, the sub-image includes the target object, and the target image refers to the image with the smallest field of view in the image set.

The image set includes at least one original image that can completely include the bounding box selection area of the target object, that is, images that can completely include the bounding box selection area of the target object are selected from the N original images, and these images are formed into an image set.

The terminal device 100 can first select images that can completely contain the bounding box selection area of the target object from the N original images to form an image set, and then select the one with the smallest field of view from the image set according to the field of view angle from small to large. The original image is used as the target image. Wherein, the bounding box selection area of the target object is usually a bounding rectangle area.

For example, as shown in Table 5 above, the target object is the object 46, and the original image that can completely include the bounding box selection area of the object 46 includes the original image captured by the wide-angle camera and the original image captured by the main camera, and the original image captured by the telephoto camera. Object 46 is not included in the original image. At this time, the image set includes the original image collected by the wide-angle camera and the original image collected by the main camera. Since the field of view of the wide-angle camera is larger than the field of view of the main camera, the original image collected by the main camera is determined as the target image. Then, according to the target area information of the object 46 in the original image collected by the main camera, a sub-image including the object 46 is collected.

After the terminal device 100 determines the target image, it can expand the bounding box selection area of the target object by a certain ratio (for example, 5%), and then according to the preset image width and height ratio, according to the target area information of the target image, from the target image. Crop out the sub-image. The preset image aspect ratio can be set according to actual needs.

It should be noted that a sub-image includes a target object. When there are multiple target objects, multiple sub-images are obtained by cropping. For example, the voice collected by the terminal device 100 is "tracking a little boy playing a skateboard and a little boy running". At this time, the target objects include "tracking a little boy playing a skateboard" and "a little boy running", according to the "tracking little boy" According to the target area information corresponding to the "little boy playing skateboard", the sub-image containing "tracking the little boy playing skateboard" is obtained by cropping. subimage.

Step S311 , generating an output image based on the sub-image, and then displaying the output image in the viewfinder.

It should be noted that when there is only one sub-image, the terminal device 100 can reduce or enlarge the single sub-image to a preset resolution to obtain an output image, and then display the output image in the viewfinder according to the expected display mode. When there are multiple sub-images, the terminal device 100 may stitch the multiple sub-images into one image, and then reduce or enlarge the image obtained by splicing to obtain an output image, or may also reduce or enlarge a single sub-image After that, the output image is obtained by splicing, and then the output image is displayed in the viewfinder according to the expected display mode. For example, referring to FIG. 4 , after the mobile phone 41 collects the voice "tracking the little boy playing skateboard", it recognizes the voice information, and according to the voice recognition result, finds the corresponding target area information; The target object is selected, and the target object designated by the user's voice is prompted as the object 46 . At this time, the user can follow the shooting of the object 46 by clicking the shooting button. After receiving the click operation on the shooting button, the mobile phone 41 can crop a sub-image from the target image, and then generate an output image based on the sub-image, and display the output image in the viewfinder frame, that is, the display interface 412 .

Of course, the user does not need to manually click the shooting button to trigger the tracking shooting. At this time, the mobile phone 41 recognizes that the voice contains the keyword "tracking", and determines that the target object is the object 46, which can be automatically displayed after the display interface 410. Interface 412.

In other embodiments, the mobile phone 41 can directly display the interface 412 after the phone 41 collects the voice "tracking the little boy playing skateboard" and determines through voice recognition that the object 46 needs to be tracked and photographed, that is, the mobile phone 41 may not display the interface 410, but after recognizing the target object and the shooting action, the object 46 is directly tracked and shot.

In the tracking shooting mode, when the target object moves, the terminal device 100 stably tracks the target object. For example, please refer to FIG. 5 , which is a schematic diagram of stable tracking provided by an embodiment of the present application, and the scenarios in FIG. 4 and FIG. 5 are the same. As shown in FIG. 5 , the mobile phone 41 collects an image 51 through a wide-angle camera, a main camera or a telephoto camera, and the image 51 includes a plurality of objects; target recognition is performed on the image 51 to obtain object information of each object in the image 51; The second index data of the attribute information of each object is extracted from the database, and an association relationship between the index data of each object and the area information is established.

When detecting the voice 52 input by the user, the mobile phone 41 performs voice recognition on the voice 52 in combination with the object information, that is, the mobile phone 41 performs voice recognition on the voice 42, and obtains the voice recognition result "tracking the boy playing skateboard", and then according to the index data and the area The relationship between the information is determined to determine the target object corresponding to the speech recognition result and the target area information corresponding to the target object. At this time, the mobile phone 41 determines that the target object specified by the voice 52 is the object 53 , and according to the region information of the object 53 , a sub-image including the object 53 is obtained by cropping, and the output image 54 is generated based on the sub-image.

When the object 53 moves, the cell phone 41 captures the image 55 . Compared to image 55 and image 51, the position of target object 53 has changed in the image. The mobile phone 41 performs target recognition on the image 55 to obtain the object information, and then according to the area information of the object 53 , a sub-image including the object 53 is obtained by cropping, and an output image 56 is generated based on the sub-image.

It should be noted that in the sub-image cropped according to the area information of the object 53, in addition to the object 53, other background information may also be included. For the convenience of description, in FIG. 4, FIG. 5 and subsequent drawings, it may be omitted. Some background information, only the information related to the target object is shown. In the embodiment of the present application, the terminal device first performs target recognition on the images of each field of view to obtain the object information of each object, and then extracts the index data corresponding to the attribute information of the object in the database, and compares the index data and the area information. Then, after detecting the voice information, identify the voice information, and determine the target object specified by the voice information and the target area information of the target object according to the association relationship, and finally, according to the target area information, Shoot the target object. In this way, the user does not need to manually select the object of the scene frame to designate the target object, but can conveniently and accurately designate the tracking target through voice, which improves the convenience of shooting.

The embodiments of the present application can implement single-target tracking, and can also implement multi-target tracking.

Referring to FIG. 6 , FIG. 6 is a schematic diagram of tracking and shooting of multi-target tracking according to an embodiment of the present application. As shown in FIG. 6 , after the mobile phone 61 receives the click operation on the camera 62 , the mobile phone 61 starts the camera application and displays the captured image in the viewfinder. After entering the video recording mode, the mobile phone 61 displays a video recording preview interface 63, and the video recording preview interface 63 includes shooting subjects such as object 64, object 65, object 66, and object 67.

After the mobile phone 61 enters the voice shooting mode or enters the tracking shooting mode, it collects original images of different viewing angles through cameras with different viewing angles, and performs target recognition on each original image to obtain object information of each object. And, based on the pre-established database, the index data of the attribute information of each object is extracted, and the association relationship between the index data and the area information is established.

The mobile phone 61 collects the voice “Two screens to track the boy playing skateboard and the girl playing hula hoop” through the microphone, and recognizes the voice to obtain the voice recognition result; and then extracts the index data of the voice recognition result from the database, based on the index data and The relationship between the area information determines the target object corresponding to the speech recognition result and the target area information.

It should be noted that, when the speech specifies at least two target objects, the first index data of the speech recognition result includes the indices of the at least two target objects. At this time, for the index data of each target object, the target area information is determined based on the association relationship between the index data and the area information.

For example, taking the scene in FIG. 6 as an example, the voice collected by the mobile phone 61 includes two target objects, namely "a boy playing a skateboard" and "a girl playing a hula hoop". The first index data corresponding to the speech recognition result includes the index data corresponding to the "boy playing skateboard" and the index data of "girl playing hula hoop"; then, according to the index data corresponding to the "boy playing skateboard", based on the association relationship The target area information of the "boy playing skateboard" is determined, and according to the index data of the "girl playing hula hoop", the target area information of the "girl playing hula hoop" is determined based on the association relationship.

At this time, the mobile phone 61 determines that the target objects corresponding to the voice "two-screen tracking of a boy playing a skateboard and a girl playing a hula hoop" are the object 64 and the object 66 . The mobile phone 61 uses the circumscribed rectangle 69 to frame the object 66 according to the area information of the object 64 and the area information corresponding to the object 66 , and uses the circumscribed rectangle 610 to frame the object 64 . That is, after the mobile phone 61 collects the voice and determines the target object specified by the voice, the video preview interface 69 is displayed, and in the pre-preview interface 68, the target objects specified by the voice are prompted to be the object 64 and the 66.

Through voice recognition on the mobile phone 61, it can be determined that the corresponding shooting targets of the voice "Two-screen tracking of a boy playing a skateboard and a girl playing a hula hoop" are

objects

64 and 66, and can also be based on the extracted keywords "two screens" and "" "Tracking" to determine the shooting action as tracking shooting, and the display mode is two-screen display.

At this time, after determining the target object specified by the voice, the mobile phone 61 determines the target image that can completely include the frame selection area of the object 64 and the frame selection area of the object 66 and has the smallest field of view. At this time, the object 64 and the object The target image for 66 may or may not be the same. Specifically, for the object 64, the target image of the object 64 is determined; for the object 66, the target image of the object 66 is determined; and then according to the target area information, the first sub-image corresponding to the object 64 is cropped from the target image of the object 64 , and the second sub-image corresponding to the object 66 is cropped from the target image of the object 66 . Finally, the first sub-image and the second sub-image are spliced to obtain an output image, and a viewfinder is displayed.

As shown in FIG. 6 , the mobile phone 61 displays a tracking shooting interface 611 . Among them, the left half area of the tracking shooting interface 611 tracks the display object 64 , and the right half area tracks the display object 66 .

In other embodiments, after the mobile phone 61 collects the voice and determines the target object corresponding to the voice, the mobile phone 61 can directly display the tracking shooting interface 611 without displaying the video preview interface 68 .

It should be noted that when at least two sub-images are included, they can be displayed sequentially according to the order in which the target objects appear in the user's voice. screen status, from top to bottom. For example, as shown in Figure 6, in the user's voice "Two screens to track the boy playing a skateboard and the girl playing a hula hoop", the object 64 appears before the object 66, and the mobile phone 61 is in the horizontal screen state, and the object 64 is displayed on the In the left area, the object 66 is displayed in the right area.

In addition, the user's voice may not specify two-screen tracking, and when the mobile phone 61 determines that there are two target objects, it automatically uses two screens to track the two target objects. For example, the collected speech is "tracking a boy on a skateboard and a girl on a hula hoop".

It can be understood that the multi-target tracking scenario is illustrated by taking two targets as an example in FIG. 6 . When more than three target objects are included, the process is similar to the process in FIG. 6 , and details are not repeated here. For example, the collected voices are "tracking a boy playing a skateboard, a girl playing a hula hoop, and a boy running." When the mobile phone 61 determines that there are three target objects specified by the voice, it uses three-screen tracking to display the three target objects.

In some embodiments, when the terminal device 100 determines the target object specified by the voice, and when the target object is tracked and photographed, if it is detected that the target object is close to the edge of the original image with the largest field of view, a prompt may be displayed on the interface information to prompt the user to adjust the pose of the camera or terminal to improve user experience.

Specifically, in the image with the largest field of view, the terminal device 100 detects the distance between the edge of the bounding box selection area of the target object and the edge of the image with the largest field of view. When the distance corresponding to the edge with the smallest distance is less than a certain distance When the distance threshold is reached, it can be considered that the target object is about to exceed the maximum capture range of the camera. At this time, in order to continue to track and shoot the target object, prompt information can be displayed in the viewfinder frame, and the prompt information can be used to prompt the user to adjust the camera pose, and further, can also be used to prompt the camera to adjust the direction. For example, the prompt information can be text information and symbol information, that is, the text "Please adjust the camera direction" is displayed in the viewfinder, and an arrow is used to indicate the direction of camera adjustment.

In some embodiments, the terminal device 100 determines the target object specified by the voice, and when the target object is tracked and photographed, if the second voice information is collected, the second voice information is subjected to voice recognition to obtain a second voice recognition result . When it is determined according to the second voice recognition result that the second voice information is a voice command for adjusting the display mode of the target object, the cropping position of the sub-image of the target object is adjusted according to the target area information of the target object. The display manner of the target object includes the display position and/or display size of the target object.

For example, when the second voice information is "to the right", the terminal device 100 adjusts the cropping position of the sub-image so that the target object in the output image deviates from the center of the screen and deviates to the right. Typically, when cropping sub-images, keep the target object in the center of the frame as much as possible.

Wherein, when it is determined according to the second voice recognition result that the second voice information is a voice command used to enlarge or reduce the target object, that is, when the second voice information is used to adjust the display size of the target object, it can be selected from different In the field-of-view image, an appropriate field-of-view image is determined as the target image, and then a sub-image is obtained by cropping the target image. For example, when the second voice information is "zoom in to the upper body", and the terminal device 100 determines that the second voice information is a voice command for zooming in on the target object according to the second voice recognition result, it selects the image corresponding to the telephoto camera as the target image, And crop the sub-image from the original image collected by the telephoto camera.

During the tracking shooting process, the user can adjust the display position and/or display size of the target object through voice, which further improves the convenience of shooting.

Exemplarily, the terminal device 100 is a mobile phone, the mobile phone includes a main camera, a wide-angle camera, and a telephoto camera, and the viewing angles from large to small are: a wide-angle camera, a main camera, and a telephoto camera. FIG. 7A is a schematic diagram of an image processing process in a tracking shooting scene, and FIG. 7B is a schematic interface diagram in a tracking shooting scene.

As shown in Figure 7A, the mobile phone collects images of different field of view through three cameras, and performs target recognition on the images of each field of view to obtain the object information of each object; based on the attribute information and area information of each object, establish index data and The relationship between regional information. After the mobile phone detects the voice "tracking the boy playing skateboard", it performs voice command recognition in combination with the object information, and determines that the target object specified by the voice is object 72; and it is assumed that the images collected by the main camera and the wide-angle camera can completely contain the object at this time. 72, the image captured by the main camera is used as the target image, and then according to the target area information of the object 72, the image captured by the main camera is cropped according to the default scale and position to obtain the sub-image of the object 72; The sub-image generates an output image 73, and displays the image 73 on the tracking shooting interface 78. For details, please refer to FIG. 7B.

The mobile phone continues to track and photograph the object 72, and at a certain moment collects the voice "go to the right", the mobile phone recognizes the voice, and determines that the voice is a voice command for adjusting the display position of the target object; then, the mobile phone according to the object 72 corresponds to the target area information, adjust the cropping position, and crop the sub-image from the image collected by the main camera, so that the object 72 is displayed on the right side of the screen, that is, by adjusting the cropping position, let the sub-image completely include the frame selection area of the object 72, And let the object 72 be offset from the center of the screen in the output image, and offset to the right. As shown in FIG. 7A , by adjusting the cropping position, the mobile phone obtains a sub-image by cropping, and based on the sub-image, an image 74 is obtained, and the image 74 is displayed on the tracking shooting interface 79 in FIG. 7B .

During the tracking shooting process, when the target object moves to the edge of the original image captured by the main camera, when the mobile phone determines that the original image captured by the main camera cannot completely include the frame selection area of the object 72, it selects the frame selection area that can completely include the object 72. The wide-angle camera image of the frame selection area, that is, the image collected by the wide-angle camera is determined as the target image; then according to the target area information corresponding to the object 72, the sub-image of the object 72 is obtained by cropping from the image collected by the wide-angle camera, and then based on the sub-image An image 75 is generated and displayed on the tracking capture interface 710 as shown in FIG. 7B .

The object 72 continues to move. When the mobile phone detects that the target object 72 is close to the edge of the wide-angle camera image, that is, it is determined that the tracking target is about to exceed the maximum capture range of the camera, prompt information can be generated to prompt the user to adjust the camera pose. As shown in FIG. 7A and FIG. 7B , the mobile phone generates an image 76 and displays the image 76 on the tracking shooting interface 711 . The image 76 includes a text prompt message "Please adjust the camera direction", and an arrow prompt to prompt the user to move the mobile phone to the right.

At a certain moment, after the mobile phone detects that the voice is "zoomed into the upper body", it performs voice recognition on the voice to obtain a voice recognition result. Wherein, the pre-established database includes key words of body parts, for example, the pre-established database includes upper body, lower body, head, eyes, mouth, upper limbs, etc.; according to the speech recognition result, it is determined that the speech is used to amplify the target For the voice command of the object, the image captured by the telephoto camera is determined as the target image from images of three different viewing angles, and a sub-image is obtained by cropping the target image, and an image 77 is generated based on the sub-image. Finally, the image 77 is displayed on the tracking capture interface 712 as shown in Figure 7B.

In some embodiments, when the terminal device 100 is tracking and photographing the target object, the target object can be changed, the target object can be added, and the target object can be decreased through voice commands.

For example, referring to a schematic diagram of replacing the target object shown in FIG. 8 , the scene in FIG. 8 is the same as that in FIG. 4 . As shown in FIG. 4 and FIG. 8 , after the mobile phone 41 displays the tracking shooting interface 412 , the mobile phone 41 continues to perform tracking shooting of the object 46 . At a certain moment, the mobile phone 41 collects the voice "change to a running boy", and then the mobile phone 41 performs voice recognition on the voice to determine that the user needs to replace the tracking object from object 46 to object 48; extract the index of the voice After the data, based on the relationship between the index data and the area information, the target area information of the object 48 is determined; A sub-image of the object 48 is cropped out of the target image, and an output image is generated based on the sub-image, and the output image is displayed, that is, the tracking shooting interface 413 is displayed. The tracking object displayed on the tracking shooting interface 413 is the object 48 , that is, the tracking object is changed from the object 46 to the object 48 .

After the tracking object is changed, the mobile phone 41 continues to track and photograph the object 48 . At a certain moment, the mobile phone 41 collects the voice "add another screen to track the girl playing hula hoop", and then the mobile phone 41 performs voice recognition on the voice to obtain a voice recognition result; when the voice is determined to be used according to the voice recognition result When adding the voice command of the tracking object, after extracting the index data of this voice recognition result, based on the correlation between the index data and the area information, determine that the object designated by this voice is the target area information of the object 47 and the object 47; Then, according to the target area information corresponding to the object 47, the sub-image of the object 47 is obtained by cropping; finally, the sub-image of the object 48 and the sub-image of the object 47 are spliced to obtain an output image, and the output image is displayed on the tracking shooting interface 414. In the tracking shooting interface 414, the object 48 is tracked in the left area, and the object 47 is tracked in the right area.

After adding the tracking object, the mobile phone 41 can continuously track the object 48 and the object 47 through the two screens. At a certain moment, the mobile phone 41 collects the voice "remove the running boy", and then, the mobile phone 41 recognizes the voice to obtain a voice recognition result; when it is determined according to the voice recognition result that the voice is a voice command for reducing the tracking object When, after extracting the index data of the speech recognition result, based on the correlation between the index data and the region information, determine that the object designated by the speech is the object 48, that is, determine that the object to be removed is the object 48, and then based on the target of the object 47 According to the region information, a sub-image of the object 47 is obtained by cropping, an output image is generated based on the sub-image, and the output image is displayed on the tracking shooting interface 415 .

For another example, FIG. 9A is a schematic diagram of an image processing process in a tracking shooting scene, and FIG. 9B is a schematic diagram of an interface corresponding to FIG. 9A . 9A and 9B are the same as the scenario of FIG. 6 .

As shown in FIG. 9A , the mobile phone collects images of different field of view through cameras with different field of view, performs target recognition on the images of each field of view, and obtains object information of each object; based on the attribute information and area information of each object , to establish the relationship between index data and regional information.

After the mobile phone detects the voice "two-screen tracking of a boy playing a skateboard and a girl playing a hula hoop", it performs voice command recognition combined with the object information, and determines that the tracking objects specified by the voice are object 92 and object 93; Select object 92 and object 93, specifically as shown in image 91, which also includes object 94 and object 95; then determine the frame selection area that can completely include object 92 and object 93, and the target with the smallest field of view Then, according to the target area information of object 92 and the target area information of object 93, the sub-image of object 92 and the sub-image of object 93 are obtained by cropping from the target image; Finally, two sub-images are spliced to generate image 96, The image 96 is displayed on the tracking shooting interface 99 shown in FIG. 9B . The left region of the tracking shooting interface 99 tracks the display object 92 , and the right region tracks the display object 93 .

Exemplarily, the mobile phone includes three cameras: a main camera, a telephoto camera, and a wide-angle camera, and a set of original images collected by the mobile phone is shown in FIG. 9C . The original image set includes the image 91 collected by the main camera, the image 912 collected by the telephoto camera, and the image 913 collected by the wide-angle camera. By comparison, it can be seen that, compared with the image 91 of the main camera, the images in the original image set 913 with a larger field of view show more objects, while the images in the original image 912 with a smaller field of view show fewer objects .

Based on the original image set shown in FIG. 9C , for the object 92 , the images captured by the main camera, wide-angle and telephoto all include the object, but only the images captured by the main camera and wide-angle can completely include the bounding frame of the object 92 The selected area, that is, the image set of the object 92 includes the image 91 and the image 913; then, the image with the smallest field of view is determined from the image set as the target image, and at this time, the image set of the object 92 includes the image 91 and the image 913, Since the field of view of the image 91 is smaller than that of the image 913, it can be determined that the image 91 is the target image.

As can be seen from the schematic diagram of the sub-image shown in FIG. 9D , after the mobile phone determines that the target image of the object 92 is the image 91 , the sub-image 914 of the object 92 is cropped from the image 91 according to the predetermined aspect ratio.

Similarly, the sub-image of the object 93 is obtained by cropping, and then the sub-image 914 of the object 92 and the sub-image of the object 93 are zoomed and displayed, and the output image 96 is obtained by splicing.

The mobile phone continuously tracks and shoots object 92 and object 93, and at a certain moment collects the voice "remove the boy playing skateboard and replace it with a boy who runs". The mobile phone recognizes the voice and determines that the voice is used to replace the tracking object. Then, determine the target area information of the object 93 and the object 94 again, and cut out the sub-image from the target image according to the target area information of the object 93 and the object 94, the sub-image is spliced, and the image 97 is obtained, and the The image 97 is displayed on the tracking capture interface 910 shown in FIG. 9B . The left region of the tracking shooting interface 910 tracks the display object 94 , and the right region tracks the display object 93 .

The mobile phone continuously tracked and photographed object 93 and object 94, and at a certain moment collected the voice "add another screen to track the little girl in red". The mobile phone recognized the voice and determined that the voice was used to replace the tracking object. Then, determine the target area information corresponding to object 93, object 94 and object 95, and according to the target area information corresponding to object 93, object 94 and object 95 from A sub-image is obtained by cropping the target image, and the three sub-images are spliced to obtain an image 98 , and the image 98 is displayed on the tracking shooting interface 911 shown in FIG. 9B . The left region of the tracking shooting interface 911 tracks the display object 94 , the middle region tracks the display object 93 , and the right region tracks the display object 95 .

It should be noted that, in the process of tracking and photographing the tracking object, the terminal device 100 can perform target recognition on the images of each field of view collected in real time, obtain the object information of each object, and establish the region information and index data of the object. relationship between. Based on this, after detecting the user's voice, the voice is recognized to obtain a voice recognition result, and based on the establishment of the relationship between the area information and the index data, the information contained in the voice is determined, for example, the target object corresponding to the voice is determined, and the Actions contained in the voice, etc.; finally, the actions corresponding to the voice commands are performed.

In the embodiment of the present application, in the process of tracking and photographing the tracking object, the user can change the tracking object, increase the tracking object, and reduce the tracking object through voice, and can also adjust the display size and display position of the tracking object through the voice, which further improves the performance of the tracking object. The ease and accuracy of tracking shots.

In the above embodiments, tracking shooting is exemplarily introduced in a video recording scene. In specific applications, the technical solutions provided in the embodiments of the present application can also be applied to photographing scenes.

For example, see the schematic diagram of tracking shooting in the shooting scene shown in FIG. 10 . As shown in FIG. 10 , the main interface 102 of the mobile phone 101 includes multiple applications such as the camera 103 , smart life, settings, calendar, clock, and gallery. After the mobile phone 101 receives the click operation on the camera 103, the mobile phone 101 responds to the click operation and displays the photo preview interface 104, and the viewfinder of the photo preview interface 104 includes

objects

105, 106, 107, and 108, etc. Shoot the subject.

After the mobile phone 101 enters the voice shooting mode, it performs target recognition on images of different viewing angles to obtain object information, and extracts index data of the attribute information of the object from the database, and establishes an association relationship between the index data and the area information.

After the mobile phone 101 collects the voice "following the running boy", the mobile phone 101 displays the photo preview interface 109 . In this process, the mobile phone 101 performs speech recognition on the speech, obtains the speech recognition result, and extracts the index data of the speech recognition result; then, according to the correlation between the index data and the area information, find the target corresponding to the speech recognition result object, and the target area information of the target object, at this time, it is determined that the target object corresponding to the voice is the object 107; then, the frame selection area that can completely include the object 107 is determined from the images with different viewing angles, and the viewing angle The smallest target image is then cropped from the target image according to the target area information of the object 107 to obtain a sub-image, the sub-image is zoomed and displayed, an output image is obtained, and the photo preview interface 109 is displayed.

During the movement of the object 107 , the mobile phone 101 continuously tracks the object 107 and displays a photo preview interface 1010 . During the tracking shooting process, the user can click the shooting button 1011 to take a picture of the object 107 . After receiving the click operation on the shooting button 1011, the mobile phone 101 captures a photo of the object 107 in response to the click operation. When the mobile phone 101 receives the click operation on the button 1012 , the mobile phone 101 displays the photo preview interface 1013 in response to the click operation, and the photo preview interface 1013 displays the photo of the object 107 .

It should be noted that in the shooting scene, the user can change the tracking object, increase the tracking object, and reduce the tracking object through voice commands, and can also adjust the display position of the tracking object, zoom in or out through the voice command, and track the object. When reaching the edge of the image with the largest field of view, the user is prompted to adjust the camera pose.

In order to better introduce the solutions provided by the embodiments of the present application, the following description will be given with reference to specific examples. At this time, the terminal device 100 is a mobile phone, and the mobile phone includes a wide-angle camera, a main camera, and a telephoto camera, and the three cameras have different fields of view.

Referring to the schematic block diagram of the shooting process shown in FIG. 11 , as shown in FIG. 11 , a photo or video of the tracking target can be obtained by shooting in two ways, one of which is the traditional way, and the other is the way provided by the embodiment of the present application.

In the traditional way, after the mobile phone enters the camera's photo preview or video recording mode, the target direction is checked; the user manually moves the mobile phone to find the target object; after finding the target object, manually zoom in and out to adjust the display ratio and focal length of the target object in the image, etc. ; Then, when the target object moves, manually track the moving target, that is, manually move the mobile phone, so that the target object is located in the center of the screen as much as possible; finally, a photo or video of the target object is generated.

In the conventional manner, the user needs to manually operate to determine the target object, manually adjust the display scale of the target object, and, when the target object moves, the mobile phone needs to be moved manually.

In the method provided by the embodiment of the present application, the mobile phone enters the camera to take a photo preview or video, and after comparing the target direction, the wide-angle camera, the main camera, and the telephoto camera are used to collect the wide-angle image, the main camera image, and the telephoto image, respectively. The field of view image is used for target recognition to obtain the object information of each object, and the index data of the attribute information of the object is extracted from the database, and the association relationship between the index data and the area information is established.

Users can send tracking commands by voice, and the mobile phone can accurately identify the voice command according to the relationship between the index data and the area information and the pre-stored voiceprint features, and determine the target object corresponding to the voice command and the target area information of the target object; Then, according to the target area information of the target object and the expected display mode, a sub-image is obtained by cropping from the field of view image, that is, a single target area or multiple target areas are generated. The intended display manner may include the image aspect ratio. Finally, according to the size of the target object, the focal length is adjusted, that is, the cropped image is zoomed and displayed to achieve the target display ratio, and a photo or video of the tracked object is generated.

For the manner provided by the embodiments of the present application, reference may also be made to the schematic flowchart of the voice shooting shown in FIG. 12 . As shown in Figure 12, after the mobile phone enters the camera to take pictures and preview or record the scene, the user can manually adjust the camera pose to aim at the target direction. At this time, the mobile phone can collect wide-angle images, main camera images and telephoto images in real time, and perform target recognition on each image to obtain object information. At the same time, the mobile phone will also collect the user's voice command in real time through the microphone, based on the object information and pre-stored voiceprint features, accurately identify the user's voice command, determine the target object corresponding to the voice command, and the information contained in the voice command; then, according to The target area information of the target object and the expected display mode are cropped from the field of view image to obtain a sub-image, that is, a single target area or multiple target areas are generated. The intended display manner may include the image aspect ratio. Finally, according to the size of the target object, adjust the focal length to achieve the target display ratio, and generate a photo or video of the tracked object.

When the detected target object deviates, that is, when the target object is detected to be close to the edge of the image with the largest field of view, the user can be prompted to adjust the pose on the preview interface.

In the manner provided by the embodiment of the present application, the user can specify the tracking object by voice, and continuously track the tracking object without manual operation by the user. In comparison, the method provided by the embodiment of the present application can allow the user to conveniently and accurately specify the tracking object, thereby improving the convenience of tracking and shooting.

The embodiment of the present application also provides a method for determining a target object, so that the user can specify the target object by voice, which improves the convenience and accuracy of determining the target object.

Please refer to FIG. 13 , which is a schematic flowchart of a method for determining a target object provided by an embodiment of the present application. The method can be applied to the terminal device 100, and the method can include the following steps:

Step S1301: Acquire a to-be-processed image, where the to-be-processed image includes at least one object.

The above-mentioned images to be processed may include at least two original images with different viewing angles. For example, the images to be processed include an original image captured by a telephoto camera and an original image captured by a wide-angle camera; it may also include an original image with one viewing angle.

Step S1302: Determine the object information of each object in the image to be processed, the object information includes area information and attribute information, and the area information is used to describe the position of the object in the image to be processed.

In some embodiments, object recognition may be performed on the image to be processed to determine object information of each image in the image to be processed.

Step S1303: Extract second index data of each attribute information from a pre-established data set, where the data set includes attribute key words and attribute indexes.

Step S1304: For each object, associate the second index data with the area information to obtain an association relationship between the index data and the area information.

Step S1305, acquiring first voice information.

Step S1306: Extract the first index data of the first voice information from the data set based on the attribute keywords corresponding to the second index data in the data set.

Step S1307: Determine target area information corresponding to the first index data based on the association relationship and the second index data, wherein the object corresponding to the target area information is the target object.

Wherein, for the similarities between FIG. 13 and the above embodiment, reference may be made to the corresponding content of the above embodiment, and details are not repeated here.

It should be noted that, the method for determining a target object provided by the embodiments of the present application may be applied to the above-mentioned tracking video scene and the photographing scene, but is not limited to the above-mentioned scene. For the convenience of description, for the same points in FIG. 13 as the above embodiments, reference may be made to the corresponding contents of the above embodiments, which will not be repeated here.

In the embodiment of the present application, the object information of each object in the image is identified in advance, and based on the pre-established data set, the association relationship between the index data of each object and the area information is established; when the voice command is collected, based on the association The relationship identifies the object in the image that corresponds to the voice command, allowing users to easily and quickly specify the target object.

Please refer to FIG. 14 , which is another schematic flowchart of the method for determining a target object provided by an embodiment of the present application. The method can be applied to the terminal device 100, and the method can include the following steps:

Step S1401 , displaying a picture captured by the camera in the viewfinder, the picture including at least one object.

The terminal device 100 may include one or at least two cameras with different viewing angles. The terminal device 100 may display the output image in the viewfinder based on the original image captured by one of the cameras.

Step S1402: Acquire first voice information, where the first voice information is used to specify a target object.

Step S1403: Determine the object corresponding to the first voice information in the screen as the target object.

The terminal device 100 may, after collecting the original image, perform target recognition on the original image to obtain object information of each object in the original image; then, based on the object information and the pre-established data set, establish a relationship between the index data and the area information After collecting the first voice information, the first voice information can be recognized, and based on the voice recognition result and the association relationship, the object corresponding to the first voice information is determined, and the object is determined as the target object.

For the process of establishing the association relationship and the process of determining the object corresponding to the first voice information, reference may be made to the above embodiments, and details are not described herein again.

In the tracking shooting scene, after the object corresponding to the first voice information in the screen is determined as the target object, the output image of the target object can be displayed. For example, in the scene in FIG. 4 , after the tracking object is determined to be the object 46 , the interface 412 is displayed.

During the tracking shooting process, the user can adjust the display method of the target object by voice, increase or decrease the target object, replace the target object, etc. For details, please refer to the above embodiment, which will not be repeated here.

In the embodiment of the present application, the user can conveniently and quickly specify the target object through voice.

Please refer to FIG. 15 , which is another schematic flow diagram of the photographing method provided by the embodiment of the present application. The method may be applied to the terminal device 100, and the method may include the following steps:

Step S1501: Display a picture captured by the camera in the viewfinder frame, and the picture includes at least one object.

Step S1502: Receive a first instruction, where the first instruction is used to specify at least two target objects.

The above-mentioned first instruction may be a voice instruction or a non-voice instruction. For example, the user may input the above-mentioned first instruction by finger-pointing or touching.

Step S1503: Determine at least two objects in the screen corresponding to the first instruction as at least two target objects.

When the first command is a voice command, the terminal device can first establish the association relationship between the index data of each object and the region information based on the object information of each object and the pre-established data set, and then based on the association relationship and the voice recognition result , to determine the object corresponding to the first instruction in the image. For the specific process, reference may be made to the above embodiments, and details are not described herein again.

When the first instruction is a non-voice instruction, the terminal device can determine the target object designated by the user according to the user's touch position. For example, the user can input the first instruction by clicking the object 64 and the object 66 in FIG. 6 successively, and the terminal device determines the object corresponding to the user's clicked position as the target object according to the display position of each object in the image.

Step S1504: Display the picture of each target object in sub-regions in the viewfinder frame, wherein one region displays the picture of one target object.

When the first command is a voice command, the terminal device can cut out sub-images based on the target area information of the target object, and then splicing the sub-images to display in an expected display manner.

When the first command is a non-voice command, after determining the display area of the target object, the terminal device may also cut out sub-images respectively, and then display the sub-images after splicing.

For example, referring to the scenario shown in FIG. 6 , the first instruction is a voice instruction. The terminal device specifies two target objects in the collected voice "Two-screen tracking of a boy playing a skateboard and a girl playing a hula hoop", and after recognizing the voice and determining the target object based on the association relationship, the interface 611 is displayed. The left area of the interface 611 displays the picture of the object 64 , and the right area displays the picture of the object 66 . When the first instruction is used to designate three target objects, the display result of the terminal device may refer to the interface 911 in FIG. 9B .

In the embodiment of the present application, when the user specifies at least two target objects, the terminal device displays the at least two target objects in different regions, which improves the convenience of tracking and shooting.

Corresponding to the method for determining a target object in the above embodiment, FIG. 16 shows a schematic block diagram of an apparatus for determining a target object provided by an embodiment of the present application. For convenience of description, only parts related to the embodiment of the present application are shown.

Referring to Figure 16, the apparatus may include:

an image acquisition module 161, configured to acquire a to-be-processed image, where the to-be-processed image includes at least one object;

The first object information determination module 162 is used to determine the object information of each object in the image to be processed, the object information includes area information and attribute information, and the area information is used to describe the position of the object in the image to be processed;

The first extraction module 163 is configured to extract the second index data of each attribute information from a pre-established data set, where the data set includes attribute key words and attribute indexes;

The first establishment module 164 is used for associating the second index data with the area information for each object, so as to obtain the association relationship between the index data and the area information;

a first voice information obtaining module 165, configured to obtain the first voice information;

The second extraction module 166 is configured to extract the first index data of the first speech information from the data set based on the attribute keywords corresponding to the second index data in the data set;

The first target object determination module 167 is configured to determine target area information corresponding to the first index data based on the association relationship and the second index data, wherein the object corresponding to the target area information is the target object.

In some embodiments, the above-mentioned first target object determination module is specifically configured to: for each second index data, respectively match each index in the second index data with each index in the first index data, and determine The number of items of the target index, the target index is the index in the second index data that matches the index in the first index data; based on the association relationship, the area information corresponding to the second index data with the largest number of items is determined as the target area information.

In some embodiments, the above-mentioned first extraction module is specifically configured to: for each attribute information, find the first keyword word matching the attribute information from the data set; use the index data of the first keyword word in the data set as the second keyword index data.

In some embodiments, the above-mentioned second extraction module is specifically configured to: perform speech recognition on the first speech information to obtain a first speech recognition result; and determine, according to the first speech recognition result, that the first speech information is used for specifying the target object A voice command; searching for a second key word matching the first speech recognition result from attribute key words corresponding to the second index data; using the index data of the second key word in the data set as the first index data.

In some embodiments, the above-mentioned second extraction module is specifically configured to: extract the voiceprint feature of the first voice information; determine the similarity between the voiceprint feature and the pre-stored voiceprint feature; when the similarity is greater than or equal to a preset threshold When the voiceprint feature and the first speech information are input into the semantic understanding model, the first speech recognition result output by the semantic understanding model is obtained.

In some embodiments, the above-mentioned first object information determination module is specifically configured to: input the image to be processed into the target recognition model; and obtain the object information output by the target recognition model.

In some embodiments, the first voice information is a voice command for specifying the target object for the first time, or a voice command for increasing or decreasing the target object, or a voice command for replacing the target object.

In some embodiments, the image to be processed includes N original images with different viewing angles, where N is a positive integer greater than or equal to 1. The above device may also include:

The first display module is configured to obtain the first output image of the target object according to the target area information based on the original image, and display the first output image in the viewfinder frame.

In some embodiments, the above-mentioned first display module is specifically configured to: determine a first target image from N original images, the first target image is the image with the smallest field of view in the image set, and the image set includes at least one image that can completely contain The original image of the bounding box selection area of the target object; the first sub-image of the target object is cropped from the first target image according to the target area information; the first output image is generated based on the first sub-image.

In some embodiments, when there are at least two target objects, and each target object corresponds to a first sub-image, the above-mentioned first display module is specifically configured to: display the first sub-image corresponding to each target object Stitching to obtain the first output image.

In some embodiments, the above-mentioned apparatus further comprises:

A second voice information acquisition module, configured to acquire second voice information;

a first recognition module, configured to perform speech recognition on the second speech information to obtain a second speech recognition result;

a first determining module, configured to determine, according to the second voice recognition result, that the second voice information is a voice command for adjusting the display mode of the target object;

a second determining module, configured to determine the second target image from the original image;

a first cropping module for cropping out the second sub-image of the target object from the second target image;

The second display module is configured to generate a second output image based on the second sub-image, and display the second output image in the viewfinder.

In some embodiments, the above-mentioned apparatus may further include:

The first detection module is used to determine the distance between each edge of the bounding box selection area of the target object and the corresponding image edge in the third target image, and the third target image is the image with the largest field of view in the image set ;

The first prompt module is configured to display prompt information in the viewfinder if the minimum value of the multiple distances is less than or equal to the preset distance threshold, and the prompt information is used to prompt adjustment of the camera pose.

In some embodiments, the attribute information includes at least one of: type, gender, age group, activity information, hair length, hair color, clothing type, clothing color, and posture.

The above-mentioned device for determining the target object has the function of realizing the above-mentioned method for determining the target object, and the function can be realized by hardware, and can also be realized by executing corresponding software through hardware, and the hardware or software includes one or more modules corresponding to the above-mentioned functions, Modules can be software and/or hardware.

It should be noted that the information exchange, execution process and other contents between the above-mentioned devices/modules are based on the same concept as the method embodiments of the present application. For specific functions and technical effects, please refer to the method embodiments section. It is not repeated here.

Corresponding to the method for determining a target object in the above embodiment, FIG. 17 shows a schematic block diagram of an apparatus for determining a target object provided by an embodiment of the present application. For convenience of description, only parts related to the embodiment of the present application are shown.

Referring to Figure 17, the apparatus may include:

The first picture display module 171 is used to display the picture captured by the camera in the viewfinder frame, and the picture includes at least one object;

a first obtaining module 172, configured to obtain first voice information, and the first voice information is used to specify a target object;

The second target object determination module 173 is configured to determine the object corresponding to the first voice information in the picture as the target object.

In some embodiments, the second target object determination module is specifically configured to: extract the first index data of the first voice information from a pre-established data set, where the data set includes attribute key words and attribute indexes; based on the association relationship of each object , determine the target area information corresponding to the first index data, and the object corresponding to the target area information is the target object;

In some embodiments, the above-mentioned apparatus may further include:

The second object information determination module is used to determine the object information of each object in the original image, and the object information includes area information and attribute information;

The third extraction module is used to extract the second index data of the attribute information of each object from the data set;

The second establishment module is used for associating the second index data with the area information for each object, so as to obtain the association relationship between the index data and the area information.

In some embodiments, the above-mentioned third extraction module is specifically configured to: for each attribute information, find the first keyword word matching the attribute information from the data set; use the index data of the first keyword word in the data set as the second keyword index data.

In some embodiments, the second object information determination module is specifically configured to: input the original image into the target recognition model; and obtain the object information output by the target recognition model.

In some embodiments, the second target object determination module is specifically configured to: for each second index data, respectively match each index in the first index data with each index in the second index data, and determine the target The number of items in the index, and the target index is the index in the second index data that matches the index in the first index data; based on the association relationship, the area information corresponding to the second index data with the largest number of items is determined as the target area information.

In some embodiments, the second target object determining module is specifically configured to: perform voice recognition on the first voice information to obtain a first voice recognition result; and determine the first voice information as a target object for specifying the target object according to the first voice recognition result searching for the second key word matching the first speech recognition result from the attribute key words corresponding to the second index data; using the index data of the second key word in the data set as the first index data.

In some embodiments, the second target object determination module is specifically configured to: extract the voiceprint feature of the first voice information; determine the similarity between the voiceprint feature and the pre-stored voiceprint feature; when the similarity is greater than or equal to a preset When the threshold is set, the voiceprint feature and the first speech information are input into the semantic understanding model, and the first speech recognition result output by the semantic understanding model is obtained.

In some embodiments, the cameras include N cameras with different viewing angles, where N is a positive integer greater than or equal to 1. The above device also includes:

The third display module is configured to obtain the first output image of the target object according to the target area information based on the original images of N different viewing angles, and display the first output image in the viewfinder frame.

In some embodiments, the above-mentioned third display module is used to: determine a first target image from N original images, the first target image is the image with the smallest field of view in the image set, and the image set includes at least one image that can completely contain the target The original image of the bounding box selection area of the object; the first sub-image of the target object is cropped from the first target image according to the target area information; the first output image is generated based on the first sub-image.

In some embodiments, when there are at least two target objects, and each target object corresponds to a first sub-image, the above-mentioned third display module is specifically configured to: display the first sub-image corresponding to each target object Stitching to obtain the first output image.

In some embodiments, the above-mentioned apparatus further comprises:

a second obtaining module, configured to obtain second voice information;

A second recognition module, configured to perform speech recognition on the second voice information to obtain a second voice recognition result;

a third determining module, configured to determine, according to the second voice recognition result, that the second voice information is a voice command for adjusting the display mode of the target object;

a fourth determining module, used for determining the second target image from the original image;

The second cropping module is used for cropping out the second sub-image of the target object from the second target image;

The fourth display module is configured to generate a second output image based on the second sub-image, and display the second output image in the viewfinder.

In some embodiments, the above-mentioned apparatus may further include:

The second detection module is used to determine the distance between each edge of the bounding box selection area of the target object and the corresponding image edge in the third target image, where the third target image is the image with the largest field of view in the image set ;

The second prompt module is configured to display prompt information in the viewfinder if the minimum value among the multiple distances is smaller than the preset distance threshold, and the prompt information is used to prompt adjustment of the camera pose.

Corresponding to the photographing method of the above embodiment, FIG. 18 shows a schematic block diagram of the photographing apparatus provided by the embodiment of the present application. For the convenience of description, only the part related to the embodiment of the present application is shown.

Referring to Figure 18, the apparatus may include:

The second picture display module 181 is configured to display the picture captured by the camera in the viewfinder, and the picture includes at least one object;

a receiving module 182, configured to receive a first instruction, where the first instruction is used to specify at least two target objects;

a third target object determination module 183, configured to determine at least two objects in the screen corresponding to the first instruction as at least two target objects;

The sub-area display module 184 is configured to display the picture of each target object in sub-areas in the viewfinder, wherein one area displays the picture of one target object.

In some embodiments, the first instruction is a voice instruction.

In some embodiments, the third target object determination module is specifically configured to: extract the first index data of the first instruction from a pre-established data set, where the data set includes attribute key words and attribute indexes; Determine the target area information corresponding to the first index data, and the object corresponding to the target area information is the target object;

In some embodiments, the above-mentioned apparatus may further include:

The third object information determination module is used to determine the object information of each object in the original image, and the object information includes area information and attribute information;

a fourth extraction module, used for extracting the second index data of the attribute information of each object from the data set;

The third establishing module is used for associating the second index data with the area information for each object, so as to obtain the association relationship between the index data and the area information.

In some embodiments, the fourth extraction module is specifically configured to: for each attribute information, find the first keyword word matching the attribute information from the data set; use the index data of the first keyword word in the data set as the second index data.

In some embodiments, the third object information determination module is specifically configured to: input the original image into the target recognition model; and obtain the object information output by the target recognition model.

In some embodiments, the third target object determination module is specifically configured to: for each second index data, respectively match each index in the first index data with each index in the second index data, and determine the target The number of items in the index, and the target index is the index in the second index data that matches the index in the first index data; based on the association relationship, the area information corresponding to the second index data with the largest number of items is determined as the target area information.

In some embodiments, the third target object determination module is specifically configured to: search for the second keyword matching the first instruction from the attribute keywords corresponding to the second index data; The index data is used as the first index data.

In some embodiments, the cameras include N cameras with different field of view, where N is a positive integer greater than or equal to 1; the above-mentioned sub-area display module is specifically configured to: for each target object, determine the number of images from the N original images. A target image, the first target image is the image with the smallest field of view in the image set, and the image set includes at least one original image that can completely contain the bounding box selection area of the target object; for each target object, according to the target area information, from The first sub-image of the target object is cropped from the first target image; the first sub-images corresponding to each target object are stitched to obtain a first output image; and the first output image is displayed in the viewfinder.

In some embodiments, the above-mentioned apparatus further comprises:

a third acquiring module, configured to acquire the second voice information;

a third recognition module, configured to perform speech recognition on the second speech information to obtain a second speech recognition result;

a fifth determining module, configured to determine, according to the second voice recognition result, that the second voice information is a voice command for adjusting the display mode of the target object;

a sixth determining module, configured to determine the second target image from the original image;

The third cropping module is used for cropping out the second sub-image of the target object from the second target image;

The fifth display module is configured to generate a second output image based on the second sub-image, and display the second output image in the viewfinder frame.

In some embodiments, the above-mentioned apparatus may further include:

The third detection module is used to determine the distance between each edge of the bounding box selection area of the target object and the corresponding image edge in the third target image, where the third target image is the image with the largest field of view in the image set ;

The third prompt module is configured to display prompt information in the viewfinder if the minimum value of the multiple distances is smaller than the preset distance threshold, and the prompt information is used to prompt adjustment of the camera pose.

The above-mentioned photographing device has the function of realizing the above-mentioned photographing method, and the function can be realized by hardware or by executing corresponding software in hardware. The hardware or software includes one or more modules corresponding to the above-mentioned functions, and the modules can be software and/or software. or hardware.

The terminal device provided by the embodiments of the present application may include a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method in any of the foregoing method embodiments is implemented.

Embodiments of the present application further provide a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the foregoing method embodiments can be implemented.

The embodiments of the present application provide a computer program product, when the computer program product runs on a terminal device, so that the terminal device can implement the steps in the foregoing method embodiments when executed.

An embodiment of the present application further provides a chip system, where the chip system includes a processor, the processor is coupled to a memory, and the processor executes a computer program stored in the memory, so as to implement the methods described in the foregoing method embodiments. method. The chip system may be a single chip, or a chip module composed of multiple chips.

In the foregoing embodiments, the description of each embodiment has its own emphasis. For parts that are not described or described in detail in a certain embodiment, reference may be made to the relevant descriptions of other embodiments. It should be understood that the size of the sequence numbers of the steps in the above embodiments does not mean the sequence of execution, and the execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application. Furthermore, in the description of the specification of the present application and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be construed as indicating or implying relative importance. References in this specification to "one embodiment" or "some embodiments" and the like mean that a particular feature, structure or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in other embodiments," etc. in various places in this specification are not necessarily All refer to the same embodiment, but mean "one or more but not all embodiments" unless specifically emphasized otherwise.

Finally, it should be noted that: the above are only specific embodiments of the present application, but the protection scope of the present application is not limited to this, and any changes or replacements within the technical scope disclosed in the present application should be covered by the present application. within the scope of protection of the application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

A method for determining a target object, characterized in that the method comprises:

acquiring an image to be processed, the image to be processed includes at least one object;

determining object information of each of the objects in the image to be processed, the object information includes area information and attribute information, and the area information is used to describe the position of the object in the image to be processed;

Extracting second index data of each of the attribute information from a pre-established data set, the data set including attribute key words and attribute indexes;

For each of the objects, associating the second index data with the area information to obtain an association relationship between the index data and the area information;

obtain the first voice information;

extracting the first index data of the first voice information from the data set based on attribute keywords corresponding to the second index data in the data set;

Based on the association relationship and the second index data, target area information corresponding to the first index data is determined, wherein the object corresponding to the target area information is a target object.
The method according to claim 1, wherein determining the target area information corresponding to the first index data based on the association relationship and the second index data, comprising:

For each of the second index data, each index in the second index data is matched with each index in the first index data, and the number of items of the target index is determined, and the target index is an index in the second index data that matches an index in the first index data;

Based on the association relationship, the area information corresponding to the second index data with the largest number of items is determined as the target area information.
The method according to claim 1, wherein extracting the second index data of each of the attribute information from a pre-established data set comprises:

For each of the attribute information, search the data set for a first keyword matching the attribute information;

The index data of the first keyword in the data set is used as the second index data.
The method according to claim 1, wherein extracting the first index data of the first voice information from the data set based on attribute keywords corresponding to the second index data in the data set, comprising: :

performing voice recognition on the first voice information to obtain a first voice recognition result;

According to the first voice recognition result, determine that the first voice information is a voice command for specifying a target object;

Searching for a second keyword matching the first speech recognition result from attribute keywords corresponding to the second index data;

The index data of the second keyword in the data set is used as the first index data.
The method according to claim 4, wherein the voice recognition is performed on the first voice information to obtain a first voice recognition result, comprising:

extracting the voiceprint feature of the first voice information;

determining the similarity between the voiceprint feature and the pre-stored voiceprint feature;

When the similarity is greater than or equal to a preset threshold, the voiceprint feature and the first voice information are input into a semantic understanding model to obtain the first voice recognition result output by the semantic understanding model.
The method according to claim 1, wherein determining the object information of each of the objects in the to-be-processed image comprises:

inputting the image to be processed into a target recognition model;

Obtain the object information output by the target recognition model.
The method according to any one of claims 1 to 6, wherein the first voice information is a voice command for specifying the target object for the first time, or a voice command for increasing or decreasing the target object, or a voice command for replacing The target object's voice command.
The method according to any one of claims 1 to 7, wherein the to-be-processed image comprises N original images with different viewing angles, and N is a positive integer greater than or equal to 1;

After determining the target area information corresponding to the first index data, the method further includes:

Based on the original image, a first output image of the target object is obtained according to the target area information, and the first output image is displayed in a viewfinder.
The method according to claim 8, wherein, based on the original image, obtaining the first output image of the target object according to the target area information, comprising:

A first target image is determined from the N original images, the first target image is the image with the smallest field of view in the image set, and the image set includes at least one frame selection area that can completely contain the target object the original image;

According to the target area information, crop a first sub-image of the target object from the first target image;

The first output image is generated based on the first sub-image.
The method according to claim 9, wherein when there are at least two target objects, and each target object corresponds to one of the first sub-images, obtaining the first sub-image based on the first sub-image The first output image, including:

The first sub-images corresponding to each of the target objects are stitched to obtain the first output image.
The method according to claim 9, wherein after displaying the first output image in the viewfinder, the method further comprises:

obtain second voice information;

Perform voice recognition on the second voice information to obtain a second voice recognition result;

According to the second voice recognition result, determine that the second voice information is a voice command for adjusting the display mode of the target object;

determining a second target image from the original image;

Crop out a second sub-image of the target object from the second target image;

A second output image is generated based on the second sub-image, and the second output image is displayed in the viewfinder.
The method according to any one of claims 8 to 11, wherein the method further comprises:

In the third target image, determine the distance between each edge of the bounding box selection area of the target object and the corresponding image edge, and the third target image is the image with the largest field of view in the image set;

If the minimum value among the plurality of distances is less than or equal to the preset distance threshold, prompt information is displayed in the viewfinder, and the prompt information is used to prompt adjustment of the camera pose.
The method of claim 1, wherein the attribute information includes at least one of the following: type, gender, age group, activity information, hair length, hair color, clothing type, clothing color, and posture.
A method for determining a target object, characterized in that it is applied to a terminal device, the method comprising:

displaying a picture captured by the camera in the viewfinder, the picture including at least one object;

obtaining first voice information, the first voice information is used to specify a target object;

An object in the picture corresponding to the first voice information is determined as a target object.
The method according to claim 14, wherein determining the object corresponding to the first voice information in the picture as the target object comprises:

Extracting the first index data of the first voice information from a pre-established data set, the data set including attribute key words and attribute indexes;

Determine the target area information corresponding to the first index data based on the association relationship of each of the objects, and the object corresponding to the target area information is the target object;

The association relationship is a mapping relationship between the second index data of the attribute information of the object and the area information, where the area information is used to describe the position of the object in the original image collected by the camera.
The method according to claim 15, wherein before determining the target area information corresponding to the first index data based on the association relationship of each of the objects, the method further comprises:

determining object information of each of the objects in the original image, where the object information includes the region information and the attribute information;

Extracting second index data of the attribute information of each object from the data set;

For each of the objects, the second index data is associated with the area information to obtain an association relationship between the index data and the area information.
The method according to claim 16, wherein extracting the second index data of the attribute information of each object from the data set comprises:

For each of the attribute information, searching the data set for a first keyword matching the attribute information;

The index data of the first keyword in the data set is used as the second index data.
The method according to claim 16, wherein determining the object information of each of the objects in the original image comprises:

inputting the original image into the target recognition model;

Obtain the object information output by the target recognition model.
The method according to any one of claims 15 to 18, wherein determining the target area information corresponding to the first index data based on the association relationship of each of the objects, comprising:

For each of the second index data, each index in the first index data is matched with each index in the second index data, and the number of items of the target index is determined, and the target index is an index in the second index data that matches an index in the first index data;

Based on the association relationship, the area information corresponding to the second index data with the largest number of items is determined as the target area information.
The method according to claim 15, wherein extracting the first index data of the first voice information from a pre-established data set comprises:

performing voice recognition on the first voice information to obtain a first voice recognition result;

According to the first voice recognition result, determine that the first voice information is a voice command for specifying a target object;

Searching for a second keyword matching the first speech recognition result from attribute keywords corresponding to the second index data;

The index data of the second keyword in the data set is used as the first index data.
The method according to claim 15, wherein performing speech recognition on the first speech information to obtain a first speech recognition result, comprising:

extracting the voiceprint feature of the first voice information;

determining the similarity between the voiceprint feature and the pre-stored voiceprint feature;

When the similarity is greater than or equal to a preset threshold, the voiceprint feature and the first voice information are input into a semantic understanding model to obtain the first voice recognition result output by the semantic understanding model.
The method according to any one of claims 15 to 21, wherein the camera comprises N cameras with different field of view, and N is a positive integer greater than or equal to 1;

After determining the target area information corresponding to the first index data based on the association relationship of each of the objects, the method further includes:

Based on the original images of N different viewing angles, a first output image of the target object is obtained according to the target area information, and the first output image is displayed in the viewfinder.
The method according to claim 22, wherein obtaining the first output image of the target object according to the target area information based on the original images of N different viewing angles, comprising:

A first target image is determined from the N original images, the first target image is the image with the smallest field of view in the image set, and the image set includes at least one frame selection area that can completely contain the target object the original image;

According to the target area information, crop a first sub-image of the target object from the first target image;

The first output image is generated based on the first sub-image.
The method according to claim 23, wherein when there are at least two target objects, and each target object corresponds to one of the first sub-images, obtaining the first sub-image based on the first sub-image The first output image, including:

The first sub-images corresponding to each of the target objects are stitched to obtain the first output image.
The method according to claim 22, wherein after displaying the first output image in a viewfinder, the method further comprises:

obtain second voice information;

Perform voice recognition on the second voice information to obtain a second voice recognition result;

According to the second voice recognition result, determine that the second voice information is a voice command for adjusting the display mode of the target object;

determining a second target image from the original image;

Crop out a second sub-image of the target object from the second target image;

A second output image is generated based on the second sub-image, and the second output image is displayed in the viewfinder.
The method according to any one of claims 22 to 25, wherein the method further comprises:

In the third target image, determine the distance between each edge of the bounding box selection area of the target object and the corresponding image edge, and the third target image is the image with the largest field of view in the image set;

If the minimum value among the plurality of distances is smaller than the preset distance threshold, prompt information is displayed in the viewfinder, and the prompt information is used to prompt adjustment of the camera pose.
A photographing method, characterized in that it is applied to a terminal device, the method comprising:

displaying a picture captured by the camera in the viewfinder, the picture including at least one object;

receiving a first instruction, where the first instruction is used to specify at least two target objects;

determining at least two objects in the picture corresponding to the first instruction as the at least two target objects;

The picture of each target object is displayed in sub-areas in the viewfinder frame, wherein one area displays a picture of the target object.
The method of claim 27, wherein the first instruction is a voice instruction.
The method according to claim 28, wherein determining at least two objects in the picture corresponding to the first instruction as the at least two target objects comprises:

extracting the first index data of the first instruction from a pre-established data set, the data set including attribute key words and attribute indexes;

Determine the target area information corresponding to the first index data based on the association relationship of each of the objects, and the object corresponding to the target area information is the target object;

The association relationship is a mapping relationship between the second index data of the attribute information of the object and the area information. The area information is used to describe the position of the object in the original image collected by the camera.
The method according to claim 29, wherein before determining the target area information corresponding to the first index data based on the association relationship of each of the objects, the method further comprises:

determining object information of each of the objects in the original image, where the object information includes the region information and the attribute information;

Extracting second index data of the attribute information of each object from the data set;

For each of the objects, the second index data is associated with the area information to obtain an association relationship between the index data and the area information.
The method according to claim 30, wherein extracting the second index data of the attribute information of each object from the data set comprises:

For each of the attribute information, search the data set for a first keyword matching the attribute information;

The index data of the first keyword in the data set is used as the second index data.
The method according to claim 30, wherein determining the object information of each of the objects in the original image comprises:

inputting the original image into the target recognition model;

Obtain the object information output by the target recognition model.
The method according to any one of claims 29 to 32, wherein determining the target area information corresponding to the first index data based on the association relationship of each of the objects, comprising:

For each of the second index data, each index in the first index data is matched with each index in the second index data, and the number of items of the target index is determined, and the target index is an index in the second index data that matches an index in the first index data;

Based on the association relationship, the area information corresponding to the second index data with the largest number of items is determined as the target area information.
The method according to claim 29, wherein extracting the first index data of the first instruction from a pre-established data set comprises:

Searching for the second keyword matching the first instruction from the attribute keyword corresponding to the second index data;

The index data of the second keyword in the data set is used as the first index data.
The method according to any one of claims 29 to 34, wherein the camera comprises N cameras with different viewing angles, and N is a positive integer greater than or equal to 1;

The picture of each target object is displayed in sub-areas in the viewfinder, including:

For each target object, a first target image is determined from the N original images, the first target image is an image with the smallest field of view in an image set, and the image set includes at least one image that can completely contain the Describe the original image of the bounding box selection area of the target object;

For each target object, according to the target area information, crop a first sub-image of the target object from the first target image;

Stitching the first sub-images corresponding to each of the target objects to obtain a first output image;

The first output image is displayed in the viewfinder.
The method according to any one of claims 29 to 35, wherein after displaying the picture of each target object in sub-areas in the viewfinder frame, the method further comprises:

obtain second voice information;

Perform voice recognition on the second voice information to obtain a second voice recognition result;

According to the second voice recognition result, determine that the second voice information is a voice command for adjusting the display mode of the target object;

determining a second target image from the original image;

Crop out a second sub-image of the target object from the second target image;

A second output image is generated based on the second sub-image, and the second output image is displayed in the viewfinder.
The method according to any one of claims 35 to 36, wherein the method further comprises:

In the third target image, determine the distance between each edge of the bounding box selection area of the target object and the corresponding image edge, and the third target image is the image with the largest field of view in the image set;

If the minimum value among the plurality of distances is smaller than the preset distance threshold, prompt information is displayed in the viewfinder, and the prompt information is used to prompt adjustment of the camera pose.
A terminal device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, when the processor executes the computer program, the process according to claim 1 to The method of any one of 13 or 14 to 26 or 27 to 37.
A computer-readable storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, any one of claims 1 to 13 or 14 to 26 or 27 to 37 is implemented method described in item.