CN115426456B

CN115426456B - Method, apparatus, electronic device, and storage medium for controlling image pickup device

Info

Publication number: CN115426456B
Application number: CN202211366523.9A
Authority: CN
Inventors: 裘添烨; 艾国; 杨作兴
Original assignee: Shenzhen MicroBT Electronics Technology Co Ltd
Current assignee: Shenzhen MicroBT Electronics Technology Co Ltd
Priority date: 2022-11-03
Filing date: 2022-11-03
Publication date: 2023-06-09
Anticipated expiration: 2042-11-03
Also published as: CN115426456A

Abstract

The embodiment of the invention provides a method, a device, an electronic device and a storage medium for controlling an image pickup device. The method comprises the following steps: a system on a chip (SOC) and an image sensor when the trigger source is detected; transmitting first configuration information to an image sensor; receiving first image data acquired based on first configuration information from an image sensor, wherein the first image data is acquired before the SOC completes initialization of an Image Signal Processing (ISP) module; storing the first image data in a storage medium; the second image data and the first image data acquired from the storage medium are combined into third image data by the SOC, and the second image data is acquired by the image sensor after the SOC is initialized. Before the initialization of the ISP module is completed, the embodiment of the invention realizes image acquisition based on the configuration information issued by the Microcontroller (MCU), advances the configuration time of the image sensor, and advances the drawing time.

Description

Method, apparatus, electronic device, and storage medium for controlling image pickup device

Technical Field

The invention belongs to the technical field of electronic equipment, and particularly relates to a method and a device for controlling image pickup equipment, electronic equipment and a storage medium.

Background

The low-power-consumption camera device is a battery-powered camera device which can provide environmental change monitoring for users for a long time. The technical requirements of such devices mainly include: the low power consumption ensures long standby time and fast drawing when a trigger source is detected.

The low power consumption image pickup apparatus includes a System-on-Chip (SOC), a microcontroller (Microcontroller Unit, MCU), and an image sensor. The SOC and image sensor are typically in a powered-off state and the MCU is typically in a normally-on state. When a trigger source (such as a person walking over) is detected, the MCU controls the SOC and the image sensor to power up. After the SOC is powered up, the image signal processing (Image Signal Processing, ISP) module is initialized. After the ISP module is initialized, ISP is performed on the image data collected by the image sensor, thereby completing the map.

However, the initialization process of the ISP module requires time, and the ISP module is not mapped until the initialization is completed, resulting in a delay in the mapping time (i.e., the earliest acquisition time of image data).

Disclosure of Invention

The embodiment of the invention provides a method, a device, an electronic device and a storage medium for controlling an image pickup device.

The technical scheme of the embodiment of the invention is as follows:

A method of controlling an image pickup apparatus including an MCU, an SOC including an ISP module, and an image sensor, the method being adapted to the MCU, the method comprising:

powering up the SOC and the image sensor when a trigger source is detected;

transmitting first configuration information to the image sensor;

receiving first image data acquired based on the first configuration information from the image sensor, wherein the first image data is acquired before the SOC completes initialization of the ISP module;

storing the first image data in a storage medium;

and combining second image data and the first image data acquired from the storage medium into third image data by the SOC, wherein the second image data is acquired by the image sensor after the SOC completes the initialization.

In an exemplary embodiment, the transmitting the first configuration information to the image sensor includes:

establishing a first I2C connection between the MCU and the image sensor through a general purpose input/Output (GPIO) port of the MCU;

the first configuration information is sent to the image sensor via the first I2C connection.

In an exemplary embodiment, further comprising:

receiving a notification message from the SOC informing that the initialization is completed;

disconnecting the first I2C connection based on the notification message;

after the first I2C connection is disconnected, the SOC establishes a second I2C connection with the image sensor, the second I2C connection being adapted to carry second configuration information sent by the SOC to the image sensor, wherein the image sensor collects the second image data based on the second configuration information.

A method of controlling an image pickup apparatus including an MCU, an SOC including an ISP module, and an image sensor, the method being applicable to the SOC, the method comprising:

initializing the ISP module after being electrified by the MCU;

receiving second image data from the image sensor, wherein the second image data is image data acquired by the image sensor after the initialization of the SOC is completed;

acquiring first image data from a storage medium, wherein the first image data is acquired by the image sensor based on first configuration information sent by an MCU before the initialization of the SOC is completed, and the MCU stores the first image data into the storage medium;

And combining the first image data and the second image data into third image data.

In an exemplary embodiment, further comprising:

after the initialization is completed, sending a notification message for notifying that the initialization is completed to the MCU, wherein after the MCU receives the notification message, the MCU is disconnected with a first I2C (inter-integrated circuit) of the image sensor, wherein the first I2C is used for bearing the first configuration information;

establishing a second I2C connection with the image sensor;

transmitting second configuration information to the image sensor based on the second I2C connection;

wherein the image sensor acquires the second image data based on the second configuration information.

In an exemplary embodiment, the combining the first image data and the second image data into the third image data includes:

performing an ISP on the first image data based on the ISP module;

performing an ISP on the second image data based on the ISP module;

and combining the first image data after the ISP is executed and the second image data after the ISP is executed into the third image data.

An apparatus for controlling an image pickup device including an MCU, an SOC including an ISP module, and an image sensor, the apparatus being adapted for the MCU, the apparatus comprising:

A power-up module configured to power up the SOC and the image sensor when a trigger source is detected;

a transmission module configured to transmit first configuration information to the image sensor;

a receiving module configured to receive first image data acquired based on the first configuration information from the image sensor, wherein the first image data is acquired before the SOC completes initialization of the ISP module;

a storage module configured to store the first image data into a storage medium;

An apparatus for controlling an image pickup device including an MCU, an SOC including an ISP module, and an image sensor, the apparatus being adapted to the SOC, the apparatus comprising:

an initialization module configured to initialize the ISP module after being powered on by the MCU;

a first acquisition module configured to acquire second image data from the image sensor, wherein the second image data is image data acquired by the image sensor after the initialization module completes the initialization;

A second acquisition module configured to acquire first image data from a storage medium, the first image data being image data acquired by the image sensor based on first configuration information sent by an MCU before the initialization module completes the initialization, wherein the MCU stores the first image data to the storage medium;

and a combination module configured to combine the first image data and the second image data into third image data.

An image pickup apparatus comprising an image sensor, an MCU, and an SOC including an ISP module, wherein:

the MCU is configured to power up the SOC and the image sensor when a trigger source is detected, and send first configuration information to the image sensor;

the SOC is configured to initialize the ISP module after being electrified by the MCU;

the image sensor is configured to acquire first image data based on the first configuration information, wherein the first image data is acquired before the initialization of the ISP module is completed by the SOC;

the MCU is further configured to store first image data into a storage medium;

the SOC is further configured to receive second image data from the image sensor, wherein the second image data is image data collected by the image sensor after the SOC completes the initialization; acquiring first image data from a storage medium; and combining the first image data and the second image data into third image data.

In an exemplary embodiment, the MCU is configured to establish a first I2C connection with the image sensor through a GPIO port of the MCU; transmitting the first configuration information to the image sensor via the first I2C connection; disconnecting the first I2C connection when receiving a notification message from the SOC informing that the initialization has been completed; the SOC is configured to establish a second I2C connection with the image sensor through a GPIO port of the SOC after the first I2C connection is disconnected, and send second configuration information to the image sensor via the second I2C connection, wherein the image sensor collects the second image data based on the second configuration information.

An electronic device, comprising:

a memory;

a processor;

wherein the memory has stored therein an application executable by the processor for causing the processor to perform the method of controlling an image pickup apparatus as set forth in any one of the above.

A computer readable storage medium having stored thereon computer readable instructions which, when executed by a processor, cause the processor to perform the method of controlling an image pickup apparatus as claimed in any preceding claim.

As can be seen from the above technical solution, in the embodiment of the present invention, when a trigger source is detected, the SOC and the image sensor are powered on; transmitting first configuration information to an image sensor; receiving first image data acquired based on first configuration information from an image sensor, wherein the first image data is acquired before the initialization of the ISP module is completed by the SOC; storing the first image data in a storage medium; the second image data and the first image data acquired from the storage medium are combined into third image data by the SOC, and the second image data is acquired by the image sensor after the SOC is initialized. Therefore, before the initialization of the ISP module is completed, the image sensor realizes image acquisition based on the configuration information issued by the MCU, does not need to wait for the completion of the initialization of the ISP module, and advances the configuration time of the image sensor.

In addition, the first image data before the ISP module finishes initialization is saved, and the drawing time is remarkably advanced before the ISP module is waited for to be attached to the second image data after finishing initialization.

Drawings

Fig. 1 is a block diagram of a low-power consumption image pickup apparatus of the related art.

Fig. 2 is a control timing diagram of a related-art low-power-consumption image pickup apparatus.

Fig. 3 is an exemplary configuration diagram of a low power consumption image capturing apparatus according to an embodiment of the present invention.

Fig. 4 is an exemplary schematic diagram of control timing of the low power consumption image capturing apparatus according to the embodiment of the present invention.

Fig. 5 is a first exemplary flowchart of a method of controlling an image capturing apparatus according to an embodiment of the present invention.

Fig. 6 is a second exemplary flowchart of a method of controlling an image capturing apparatus according to an embodiment of the present invention.

Fig. 7 is a first exemplary configuration diagram of an apparatus for controlling an image capturing device according to an embodiment of the present invention.

Fig. 8 is a second exemplary configuration diagram of an apparatus for controlling an image capturing device according to an embodiment of the present invention.

Fig. 9 is an exemplary structural diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

The present invention will be described in further detail with reference to the accompanying drawings, in order to make the objects, technical solutions and advantages of the present invention more apparent.

For simplicity and clarity of description, the following description sets forth aspects of the invention by describing several exemplary embodiments. Numerous details in the embodiments are provided solely to aid in the understanding of the invention. It will be apparent, however, that the embodiments of the invention may be practiced without limitation to these specific details. Some embodiments are not described in detail in order to avoid unnecessarily obscuring aspects of the present invention, but rather only to present a framework. Hereinafter, "comprising" means "including but not limited to", "according to … …" means "according to at least … …, but not limited to only … …". The term "a" or "an" is used herein to refer to a number of components, either one or more, or at least one, unless otherwise specified.

Hereinafter, terms related to the embodiments of the present disclosure will be explained.

MCU: the microcomputer system is formed by integrating functions such as a Central Processing Unit (CPU), a Random Access Memory (RAM), a Read Only Memory (ROM), various I/O ports, an interrupt system, a timer/counter and the like (which can also comprise circuits such as a display driving circuit, a pulse width modulation circuit, an analog multiplexer, an A/D converter and the like) with data processing capability by adopting a very large scale integrated circuit technology.

SOC: refers to the technique of integrating a complete system on a single chip, grouping all or part of the necessary electronic circuitry into packets.

I2C is a two-wire serial bus for connecting the microcontroller and its peripheral devices.

GPIO: the PIN can be used by a user, and the PIN can be used as general input, general output or general input and output.

For low power consumption image pickup apparatuses, it is necessary to ensure that standby time is long and that a drawing can be performed quickly when a trigger source is detected.

Fig. 1 is a block diagram of a low-power consumption image pickup apparatus of the related art. In fig. 1, the low power consumption image pickup apparatus includes an SOC, an MCU, and an image sensor. The low power consumption image pickup apparatus may further include a battery, a Passive InfraRed detector (PIR), a radar sensor, or the like (not shown in fig. 1). The SOC and image sensor are typically in a powered-off state and the MCU is typically in a normally-on state. The SOC also includes an ISP module (not shown in fig. 1). The ISP modules may be implemented as hardware modules having a circuit architecture, as well as software modules containing functional algorithms.

Fig. 2 is a control timing diagram of a related-art low-power-consumption image pickup apparatus. When a trigger source (such as someone walking over, someone pressing a doorbell or an animal passing over, etc.) is detected, the PIR or radar sensor triggers the MCU to interrupt, and the MCU sends a power-on instruction to the SOC and the image sensor, respectively. The image sensor completes the power-on reset faster than the SOC, and therefore the image sensor needs to wait for the SOC.

When the SOC is powered on and reset, the system kernel is loaded first, and then the image sensor is configured (e.g., the address values and initial values of registers in the image sensor are configured), where the initial values may include an image resolution initial value, an image gain initial value, an image exposure time initial value, an image frame rate initial value, or an image sensor clock value, etc.). After the image sensor is configured, image data can be continuously collected.

After the SOC configures the image sensor, the ISP module is initialized. When the ISP module is initialized, the ISP module performs ISP on the image data acquired by the image sensor at this time to output an image.

It can be seen that: in the prior art, the ISP module needs to wait for initialization before it can be mapped. For the moment before the ISP module is initialized, the image cannot be displayed, so that the technical disadvantage of lag of the display time (namely, the earliest acquisition moment of the image data) is caused. For example, a typical 200-300 milliseconds (ms) is required from a power-on reset to a map of an SOC.

In the embodiment of the invention, the host for configuring the image sensor is converted from the SOC to the MCU, the image sensor can be configured without waiting for the power-on reset of the SOC, the image sensor can be configured faster, the configuration time of the image sensor is advanced, and therefore, the image sensor can acquire earlier image data.

The MCU stores the image data (called first image data) acquired by the image sensor and before the ISP module is successfully initialized to a storage medium so as to wait for the ISP module to be used after the initialization is successful. When the initialization of the ISP module is successful, the SOC takes over the MCU to receive the image data (called second image data) acquired by the image sensor and after the initialization of the ISP module is successful. The SOC acquires first image data from the storage medium, executes ISP on the first image data and the second image data using the ISP module, and concatenates the first image data after executing ISP to the second image data after executing ISP to generate combined image data (referred to as third image data). Then, a subsequent operation such as encoding may be performed on the third image data, and the image may be output to the user.

It can be seen that the image time is significantly advanced because the output image contains image data before the ISP module is successfully initialized.

The embodiments of the present invention can be implemented to various types of image capturing apparatuses. Preferably, the embodiment of the present invention is implemented to a low power consumption image pickup apparatus. The following describes an embodiment of the present invention by taking a low power consumption image pickup apparatus as an example.

Fig. 3 is an exemplary configuration diagram of a low power consumption image capturing apparatus according to an embodiment of the present invention. In fig. 3, the low power consumption image pickup apparatus includes an SOC, an MCU, and an image sensor. The low power consumption image pickup apparatus may further include a battery, a passive infrared detector, a radar sensor, or the like (not shown in fig. 3). The SOC and image sensor are typically in a powered-off state and the MCU is typically in a normally-on state. The SOC also includes an ISP module (not shown in fig. 3). The ISP modules may be implemented as hardware modules having a circuit architecture, as well as software modules containing functional algorithms.

The MCU and the SOC may each establish a serial bus connection (e.g., an I2C connection carried by a GPIO port) with the image sensor. For example, the I2C interface of the image sensor is configured in a multi-master-slave mode, wherein the MCU and the SOC are respectively the master and the image sensor is the slave. A serial bus connection, such as an I2C connection, may be established between the MCU and the SOC. The image sensor also has a data transmission connection with the MCU and the SOC, respectively, via a data transmission interface, such as a Mobile Industry Processor Interface (MIPI). The MCU and the SOC are commonly connected to a storage medium.

When a trigger source (such as a person walks, a person presses a doorbell or an animal passes, and the like) is detected, the PIR or the radar sensor triggers the MCU to interrupt, the MCU sends a power-on instruction to the SOC and the image sensor, and then the image sensor waits for the completion of power-on reset. After the SOC is powered on and reset, the system kernel needs to be loaded, and then the ISP module is initialized.

The image sensor may complete a power-on reset faster than the SOC. After the image sensor is powered on and reset, the MCU is connected with a first I2C of the image sensor, and transmits first configuration information to the image sensor to configure the image sensor. Wherein: the first I2C connection of the MCU with the image sensor may be established through the GPIO port of the MCU. The first configuration information is used for configuring image acquisition parameters of the image sensor so that the image sensor can acquire image data, for example, the first configuration information comprises address values and initial values of registers in the image sensor, wherein the initial values can comprise: an image resolution initial value, an image gain initial value, an image exposure time initial value, an image frame rate initial value, or an image sensor clock value, and the like. After the image sensor is configured, image data can be continuously collected. It can be seen that: the MCU in the normal power state is utilized to configure the image sensor, which is different from the prior art that the image sensor is configured after the system kernel is powered on, reset and loaded by the medium SOC, so that the image sensor can acquire image data earlier.

The MCU stores image data (called first image data) acquired by the image sensor before the ISP module is successfully initialized into a storage medium through a data transmission connection with the image sensor.

When the initialization of the ISP module is successful, the SOC takes over the MCU to receive the image data (called second image data) acquired by the image sensor and after the initialization of the ISP module is successful. For example, after the initialization of the ISP module is completed, the SOC sends a notification message to the MCU for notifying that the initialization is completed, and the MCU disconnects the first I2C connection after receiving the notification message.

After the SOC takes over the MCU, the second configuration information may be issued to the image sensor to change the image acquisition parameters of the image sensor set based on the first configuration information. The SOC may establish a second I2C connection with the image sensor, and send second configuration information to the image sensor based on the second I2C connection. The second configuration information is typically a fine adjustment to the first configuration information, such as adjusting image gain, image exposure time, and image frame rate, etc. Examples: when it is desired to change the image gain, the second configuration information contains an image gain update value, the image sensor replaces the image gain initial value set based on the first configuration information with the image gain update value, and then the image sensor acquires the second image data based on the image gain update value.

In the embodiment of the invention, the first configuration information issued by the MCU to the image sensor is used for enabling the image sensor to acquire image data. After the SOC takes over the MCU, the SOC may issue second configuration information to the image sensor when an environment (e.g., ambient brightness) changes or an image demand (e.g., image frame rate) changes. The image sensor changes the image acquisition parameters configured based on the first configuration information based on the second configuration information.

In one embodiment, the SOC may not issue the second configuration information to the image sensor, so that the image sensor continues to acquire the second image data according to the first configuration information.

In an alternative embodiment, the SOC acquires first image data from the storage medium, executes ISP on the first image data and the second image data, respectively, by the ISP module, and stores the first image data after executing ISP before the second image data after executing ISP to obtain a combined image, and outputs the combined image. In another alternative embodiment, the ISP module performs ISP on the first image data and the second image data simultaneously, and stores the first image data after performing ISP before the second image data after performing ISP to obtain a combined image, and outputs the combined image.

Compared with the prior art, the method and the device save the initialization time of the SOC, the initialization time of the ISP module and the configuration time of the image sensor. It can be seen that the embodiment of the invention can provide the output image timed from the completion of the configuration of the image sensor, and the completion of the configuration of the image sensor is advanced, so that the image time is significantly advanced. For example, compared with the prior art, the graph time of the embodiment of the invention can reach about 150 ms.

An exemplary implementation of an embodiment of the present invention is described below.

Step 1: when the external trigger source trigger is detected, the GPIO port of the MCU receives the trigger signal.

Step 2: the MCU is switched from the sleep mode to the working mode, and the image sensor and the SOC are powered on and reset.

Step 3: after the image sensor is reset first, the MCU issues first configuration information to the image sensor through a first I2C connection with the image sensor, the image sensor completes configuration based on the first configuration information, and image data collection is started. The first configuration information is basic configuration information of the image sensor, and generally includes address values and initial values of registers in the image sensor, wherein the initial values may include: an image resolution initial value, an image gain initial value, an image brightness initial value, and the like.

Step 4: the MCU receives image data acquired by the image sensor through data transmission connection with the image sensor, synchronously stores the image data into a storage medium and records the image data as first image data.

Step 5: after the SOC is reset, the initialization of the CPU and related hardware is completed, and after the system kernel (including a communication driver) is loaded, the ISP module is initialized.

Step 6: the MCU disconnects the first I2C connection (i.e., the SDA line of I2C is pulled to a high impedance state) and simultaneously disconnects the image data transmission connection between the MCU and the image sensor. The SOC takes over the I2C connection and the image transfer connection of the image sensor and controls the configuration of the image sensor. For example, the image sensor may continue to collect image data using the first configuration information. Or the SOC transmits second configuration information for finely adjusting the image acquisition parameters in the first configuration information to the image sensor, and the image sensor continuously acquires the image data by utilizing the adjusted image acquisition parameters (the image acquisition parameters which are not adjusted by the second configuration information are kept unchanged).

Step 7: and the ISP module is successfully initialized, and the SOC receives the image data acquired by the image sensor through data transmission connection with the image sensor and records the image data as second image data. ISP is performed on the second image data using an ISP module, such as Automatic Exposure (AE), linear correction, noise removal, dead pixel correction, automatic White Balance (AWB), automatic focusing, and the like.

Step 8: the SOC is communicated with the storage medium through a communication interface to acquire first image data. After ISP processing is performed on the first image data by the ISP module, the first image data after ISP is executed is saved in front of the second image data after ISP is executed, so that a combined image is generated. Typically, the acquisition time range of the first image data is around 100ms (i.e., the first image data within the time range of 100ms is acquired). It can be seen that the drawing time is significantly advanced.

The implementation process of the real mode of the invention is described below by taking MCU as an execution main body. Fig. 5 is a first exemplary flowchart of a method of controlling an image capturing apparatus according to an embodiment of the present invention. The image pickup apparatus includes an MCU, a system-on-chip SOC including an ISP module, and an image sensor. As shown in fig. 5, the method includes:

step 101: when the trigger source is detected, the SOC and the image sensor are powered on.

Step 102: the first configuration information is sent to the image sensor.

Step 103: first image data acquired based on the first configuration information is received from the image sensor, wherein the first image data is acquired before the initialization of the ISP module is completed by the SOC.

Step 104: storing the first image data in a storage medium; the second image data and the first image data acquired from the storage medium are combined into third image data by the SOC, and the second image data is acquired by the image sensor after the SOC is initialized.

In one embodiment, transmitting the first configuration information to the image sensor includes: establishing a first I2C connection between the MCU and the image sensor through a GPIO port of the MCU; the first configuration information is sent to the image sensor via the first I2C connection.

In one embodiment, the method further comprises: receiving a notification message from the SOC informing that the initialization is completed; disconnecting the first I2C connection based on the notification message; after the first I2C connection is disconnected, the SOC establishes a second I2C connection with the image sensor, the second I2C connection adapted to carry second configuration information sent by the SOC to the image sensor, wherein the image sensor collects second image data based on the second configuration information.

The implementation process of the present invention will be described below with SOC as an execution body. Fig. 6 is a second exemplary flowchart of a method of controlling an image capturing apparatus according to an embodiment of the present invention. The image pickup apparatus includes an MCU, a system-on-chip SOC including an ISP module, and an image sensor. As shown in fig. 6, the method includes:

step 201: after being powered up by the MCU, the ISP module is initialized.

Step 202: second image data is received from the image sensor, wherein the second image data is image data acquired by the image sensor after the initialization of the SOC is completed.

Step 203: and acquiring first image data from the storage medium, wherein the first image data is the image data acquired by the image sensor based on the first configuration information sent by the MCU before the SOC is initialized, and the MCU stores the first image data to the storage medium.

Step 204: the first image data and the second image data are combined into third image data.

In an exemplary embodiment, further comprising: after the initialization is completed, a notification message for notifying that the initialization is completed is sent to the MCU, wherein after the MCU receives the notification message, the MCU is disconnected with a first I2C (inter-integrated circuit) of the image sensor, wherein the first I2C is used for bearing first configuration information; establishing a second I2C connection with the image sensor; transmitting second configuration information to the image sensor based on the second I2C connection; wherein the image sensor acquires second image data based on the second configuration information.

In an exemplary embodiment, combining the first image data and the second image data into the third image data includes: performing an ISP on the first image data based on the ISP module; performing ISP on the second image data based on the ISP module; the first image data after the ISP is executed and the second image data after the ISP is executed are combined into third image data.

Fig. 7 is a first exemplary configuration diagram of an apparatus for controlling an image capturing device according to an embodiment of the present invention. The image pickup apparatus includes an MCU, an SOC including an ISP module, and an image sensor. The apparatus 300 is suitable for use in an MCU, the apparatus 300 comprising: a power-up module 301 configured to power up the SOC and the image sensor when the trigger source is detected; a transmitting module 302 configured to transmit the first configuration information to the image sensor; a receiving module 303 configured to receive, from the image sensor, first image data acquired based on the first configuration information, wherein the first image data is acquired before the initialization of the ISP module is completed by the SOC; a storage module 304 configured to store the first image data into a storage medium; the second image data and the first image data acquired from the storage medium are combined into third image data by the SOC, and the second image data is acquired by the image sensor after the SOC is initialized.

Fig. 8 is a second exemplary configuration diagram of an apparatus for controlling an image capturing device according to an embodiment of the present invention. The image pickup apparatus includes an MCU, an SOC including an ISP module, and an image sensor. The apparatus 400 is adapted for use in an SOC, the apparatus 400 comprising: an initialization module 401 configured to initialize the ISP module after being powered on by the MCU; a first acquisition module 402 configured to acquire second image data from the image sensor, wherein the second image data is image data acquired by the image sensor after the initialization module completes the initialization; a second obtaining module 403 configured to obtain first image data from the storage medium, where the first image data is image data collected by the image sensor based on the first configuration information sent by the MCU before the initialization module completes initialization, and the MCU stores the first image data in the storage medium; the combination module 404 is configured to combine the first image data and the second image data into third image data.

The embodiment of the invention also provides an image pickup device. The image pickup apparatus includes an image sensor, an MCU, and an SOC including an ISP module, wherein: the MCU is configured to power on the SOC and the image sensor when the trigger source is detected, and send first configuration information to the image sensor; the SOC is configured to initialize the ISP module after being electrified by the MCU; an image sensor configured to acquire first image data based on the first configuration information, wherein the first image data is acquired before the initialization of the ISP module is completed by the SOC; an MCU further configured to store the first image data into a storage medium; the SOC is further configured to receive second image data from the image sensor, wherein the second image data is image data acquired by the image sensor after the SOC is initialized; acquiring first image data from a storage medium; the first image data and the second image data are combined into third image data.

In an exemplary embodiment, the MCU is configured to establish a first I2C connection with the image sensor through a GPIO port of the MCU; transmitting first configuration information to the image sensor via a first I2C connection; disconnecting the first I2C connection when receiving a notification message from the SOC informing that the initialization has been completed; and the SOC is configured to establish a second I2C connection with the image sensor through a GPIO port of the SOC after the first I2C connection is disconnected, and send second configuration information to the image sensor through the second I2C connection, wherein the image sensor acquires second image data based on the second configuration information.

Preferably, the image pickup apparatus is a low power consumption image pickup apparatus.

The invention also provides electronic equipment respectively. An electronic device includes: a processor; a memory; in which an application executable by a processor is stored for causing the processor to execute the method of controlling the image pickup apparatus of the above embodiment. The memory may be implemented as various storage media such as an electrically erasable programmable read-only memory (EEPROM), a Flash memory (Flash memory), a programmable read-only memory (PROM), and the like. A processor may be implemented to include one or more central processors or one or more field programmable gate arrays, where the field programmable gate arrays integrate one or more central processor cores. In particular, the central processor or central processor core may be implemented as a CPU, MCU or Digital Signal Processor (DSP).

Fig. 9 is an exemplary structural diagram of an electronic device according to an embodiment of the present invention. Preferably, the electronic apparatus 800 may be implemented as a low power consumption image pickup apparatus.

The electronic device 800 includes: a processor 801 and a memory 802. Processor 801 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 801 may be implemented in hardware in at least one of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), programmable logic array (Programmable Logic Array, PLA). The processor 801 may also include a main processor, which is a processor for processing data in an awake state, also referred to as a central processor (Central Processing Unit, CPU), and a coprocessor; a coprocessor is a low-power processor for processing data in a standby state. In some implementations, the processor 801 may integrate with an image processor (Graphics Processing Unit, GPU) for rendering and rendering of content required for display by the display screen. In some implementations, the processor 801 may also include an AI processor for processing computing operations related to machine learning. For example, the AI processor may be implemented as a neural network processor.

Memory 802 may include one or more computer-readable storage media, which may be non-transitory. Memory 802 may also include high-speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices.

In some embodiments, a non-transitory computer readable storage medium in memory 802 is used to store at least one instruction for execution by processor 801 to implement the methods of controlling an image capture device provided by various embodiments in the present disclosure. In some embodiments, the electronic device 800 may further optionally include: a peripheral interface 803, and at least one peripheral. The processor 801, the memory 802, and the peripheral interface 803 may be connected by a bus or signal line. Individual peripheral devices may be connected to the peripheral device interface 803 by buses, signal lines, or a circuit board. Specifically, the peripheral device includes: at least one of radio frequency circuitry 804, a touch display 805, a camera assembly 806, audio circuitry 807, a positioning assembly 808, and a power supply 809. Peripheral interface 803 may be used to connect at least one Input/Output (I/O) related peripheral device to processor 801 and memory 802. In some implementations, the processor 801, memory 802, and peripheral interface 803 are integrated on the same chip or circuit board; in some other embodiments, either or both of the processor 801, the memory 802, and the peripheral interface 803 may be implemented on separate chips or circuit boards, which is not limited in this embodiment.

The Radio Frequency circuit 804 is configured to receive and transmit Radio Frequency (RF) signals, also known as electromagnetic signals. The radio frequency circuit 804 communicates with a communication network and other communication devices via electromagnetic signals. The radio frequency circuit 804 converts an electrical signal into an electromagnetic signal for transmission, or converts a received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 804 includes: antenna systems, RF transceivers, one or more amplifiers, tuners, oscillators, digital signal processors, codec chipsets, subscriber identity module cards, and so forth. The radio frequency circuitry 804 may communicate with other terminals via at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: metropolitan area networks, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and/or wireless fidelity (Wireless Fidelity, wi-Fi) networks. In some implementations, the radio frequency circuitry 804 may also include circuitry related to near field wireless communication (Near Field Communication, NFC), which is not limited by the present disclosure.

The display 805 is used to display a User Interface (UI). The UI may include graphics, text, icons, video, and any combination thereof. When the display 805 is a touch display, the display 805 also has the ability to collect touch signals at or above the surface of the display 805. The touch signal may be input as a control signal to the processor 801 for processing. At this time, the display 805 may also be used to provide virtual buttons and/or virtual keyboards, also referred to as soft buttons and/or soft keyboards. In some embodiments, the display 805 may be one and disposed on a front panel of the electronic device 800; in other embodiments, the display 805 may be at least two, respectively disposed on different surfaces of the electronic device 800 or in a folded design; in some implementations, the display 805 may be a flexible display disposed on a curved surface or a folded surface of the electronic device 800. Even more, the display 805 may be arranged in an irregular pattern other than rectangular, i.e., a shaped screen. The display 805 may be made of a material such as a liquid crystal display (Liquid Crystal Display, LCD) or an Organic Light-Emitting Diode (OLED).

The camera assembly 806 is used to capture images or video. Optionally, the camera assembly 806 includes a front camera and a rear camera. Typically, the front camera is disposed on the front panel of the terminal and the rear camera is disposed on the rear surface of the terminal. In some embodiments, the at least two rear cameras are any one of a main camera, a depth camera, a wide-angle camera and a tele camera, so as to realize that the main camera and the depth camera are fused to realize a background blurring function, and the main camera and the wide-angle camera are fused to realize a panoramic shooting and Virtual Reality (VR) shooting function or other fusion shooting functions. In some implementations, the camera assembly 806 may also include a flash. The flash lamp can be a single-color temperature flash lamp or a double-color temperature flash lamp. The dual-color temperature flash lamp refers to a combination of a warm light flash lamp and a cold light flash lamp, and can be used for light compensation under different color temperatures.

Audio circuitry 807 may include a microphone and a speaker. The microphone is used for collecting sound waves of users and the environment, converting the sound waves into electric signals, inputting the electric signals to the processor 801 for processing, or inputting the electric signals to the radio frequency circuit 804 for voice communication. For purposes of stereo acquisition or noise reduction, the microphone may be multiple and separately disposed at different locations of the electronic device 800. The microphone may also be an array microphone or an omni-directional pickup microphone. The speaker is used to convert electrical signals from the processor 801 or the radio frequency circuit 804 into sound waves. The speaker may be a conventional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, not only the electric signal can be converted into a sound wave audible to humans, but also the electric signal can be converted into a sound wave inaudible to humans for ranging and other purposes. In some implementations, the audio circuit 807 may also include a headphone jack. The location component 808 is used to locate the current geographic location of the electronic device 800 to enable navigation or location-based services (Location Based Service, LBS). The positioning component 808 may be a positioning component based on the U.S. global positioning system (Global Positioning System, GPS), the beidou system of china, the grainer system of russia, or the galileo system of the european union. The power supply 809 is used to power the various components in the electronic device 800. The power supply 809 may be an alternating current, direct current, disposable battery, or rechargeable battery. When the power supply 809 includes a rechargeable battery, the rechargeable battery may support wired or wireless charging.

Those skilled in the art will appreciate that the above-described structures are not limiting of the electronic device 800 and may include more or fewer components than shown, or may combine certain components, or may employ a different arrangement of components. It should be noted that not all the steps and modules in the above processes and the structure diagrams are necessary, and some steps or modules may be omitted according to actual needs. The execution sequence of the steps is not fixed and can be adjusted as required. The division of the modules is merely for convenience of description and the division of functions adopted in the embodiments, and in actual implementation, one module may be implemented by a plurality of modules, and functions of a plurality of modules may be implemented by the same module, and the modules may be located in the same device or different devices. The hardware modules in the various embodiments may be implemented mechanically or electronically. For example, a hardware module may include specially designed permanent circuits or logic devices (e.g., special purpose processors such as FPGAs or ASICs) for performing certain operations. A hardware module may also include programmable logic devices or circuits (e.g., including a general purpose processor or other programmable processor) temporarily configured by software for performing particular operations. As regards implementation of the hardware modules in a mechanical manner, either by dedicated permanent circuits or by circuits that are temporarily configured (e.g. by software), this may be determined by cost and time considerations.

The present invention also provides a machine-readable storage medium storing instructions for causing a machine to perform a method as herein described. Specifically, a system or apparatus provided with a storage medium on which a software program code realizing the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or apparatus may be caused to read out and execute the program code stored in the storage medium. Further, some or all of the actual operations may be performed by an operating system or the like operating on a computer based on instructions of the program code. The program code read out from the storage medium may also be written into a memory provided in an expansion board inserted into a computer or into a memory provided in an expansion unit connected to the computer, and then, based on instructions of the program code, a CPU or the like mounted on the expansion board or the expansion unit may be caused to perform part or all of actual operations, thereby realizing the functions of any of the above embodiments. Storage medium implementations for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (e.g., CD-ROMs, CD-R, CD-RWs, DVD-ROMs, DVD-RAMs, DVD-RWs, DVD+RWs), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer or cloud by a communications network.

The foregoing is merely a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method of controlling an image capturing apparatus including a microcontroller MCU, a system on chip SOC including an image signal processing ISP module, and an image sensor, the method being applicable to the MCU, the MCU being in a normally-on state, the method comprising:

powering up the SOC and the image sensor when a trigger source is detected;

transmitting first configuration information to the image sensor, wherein the first configuration information is used for configuring image acquisition parameters of the image sensor so that the image sensor can acquire image data;

receiving first image data acquired based on the first configuration information from the image sensor, wherein the first image data is acquired before the SOC completes initialization of the ISP module and is timed at a time when the image sensor completes configuration;

storing the first image data in a storage medium;

Wherein second image data, which is image data acquired by the image sensor after the initialization of the SOC is completed, and the first image data acquired from the storage medium are combined by the SOC into third image data.

2. The method of claim 1, wherein the step of determining the position of the substrate comprises,

the sending the first configuration information to the image sensor includes:

establishing a first I2C connection between the MCU and the image sensor through a general purpose input/output GPIO port of the MCU;

3. The method as recited in claim 2, further comprising:

disconnecting the first I2C connection based on the notification message;

4. A method of controlling an image pickup apparatus including an MCU, an SOC, and an image sensor, the SOC including an ISP module, the method being adapted to the SOC, the MCU being in a normally-on state, the method comprising:

initializing the ISP module after being electrified by the MCU;

acquiring first image data from a storage medium, wherein the first image data is acquired by the image sensor before the initialization of the SOC is completed based on first configuration information sent by an MCU (micro control unit), and is counted at the time when the configuration of the image sensor is completed, wherein the MCU stores the first image data to the storage medium, and the first configuration information is used for configuring image acquisition parameters of the image sensor so that the image sensor can acquire the image data;

5. The method as recited in claim 4, further comprising:

After the initialization is completed, sending a notification message for notifying that the initialization is completed to the MCU, wherein after the MCU receives the notification message, the MCU is disconnected from the first I2C for carrying the first configuration information and the image sensor;

establishing a second I2C connection with the image sensor;

6. The method of claim 4, wherein combining the first image data and the second image data into third image data comprises:

performing an ISP on the first image data based on the ISP module;

performing an ISP on the second image data based on the ISP module;

7. An apparatus for controlling an image pickup device, wherein the image pickup device includes an MCU, an SOC, and an image sensor, the SOC includes an ISP module, the apparatus is adapted to the MCU, the MCU is in a constant state, the apparatus includes:

a transmitting module configured to transmit first configuration information to the image sensor, wherein the first configuration information is used for configuring image acquisition parameters of the image sensor so that the image sensor can acquire image data;

a receiving module configured to receive, from the image sensor, first image data acquired based on the first configuration information, wherein the first image data is acquired before the SOC completes initialization of the ISP module and is timed at a time when the image sensor completes configuration;

8. An apparatus for controlling an image pickup device, wherein the image pickup device includes an MCU, an SOC, and an image sensor, the SOC includes an ISP module, the apparatus is adapted to the SOC, the MCU is in a constant state, the apparatus includes:

a second acquisition module configured to acquire first image data from a storage medium, the first image data being image data acquired by the image sensor before the initialization module completes the initialization based on first configuration information sent by an MCU (micro control unit) and counted at a configuration time of the image sensor, wherein the MCU stores the first image data to the storage medium, and the first configuration information is used for configuring image acquisition parameters of the image sensor so that the image sensor can acquire the image data;

9. An image pickup apparatus, comprising an image sensor, an MCU and an SOC, the SOC comprising an ISP module, the MCU being in a normally-on state, wherein:

The MCU is configured to power up the SOC and the image sensor when a trigger source is detected, and send first configuration information to the image sensor, wherein the first configuration information is used for configuring image acquisition parameters of the image sensor so that the image sensor can acquire image data;

the image sensor is configured to acquire first image data based on the first configuration information, wherein the first image data is acquired before the initialization of the ISP module is completed by the SOC and is timed at the moment the image sensor completes configuration;

the MCU is further configured to store first image data into a storage medium;

10. The image pickup apparatus according to claim 9, wherein,

The MCU is configured to establish a first I2C connection with the image sensor through a GPIO port of the MCU; transmitting the first configuration information to the image sensor via the first I2C connection; disconnecting the first I2C connection when receiving a notification message from the SOC informing that the initialization has been completed;

the SOC is configured to establish a second I2C connection with the image sensor through a GPIO port of the SOC after the first I2C connection is disconnected, and send second configuration information to the image sensor via the second I2C connection, wherein the image sensor collects the second image data based on the second configuration information.

11. An electronic device, comprising:

a memory;

a processor;

wherein the memory has stored therein an application executable by the processor for causing the processor to perform the method of controlling an image pickup apparatus according to any one of claims 1 to 3 or the method of controlling an image pickup apparatus according to any one of claims 4 to 6.

12. A computer readable storage medium having stored thereon computer readable instructions which, when executed by a processor, cause the processor to perform the method of controlling an image pickup apparatus according to any one of claims 1 to 3 or the method of controlling an image pickup apparatus according to any one of claims 4 to 6.