JP7348783B2 - Imaging device, its control method, program and recording medium - Google Patents

Description

The present invention relates to an imaging device, a control method thereof, a program, and a recording medium.

2. Description of the Related Art In recent years, as the performance and functionality of image sensors have improved, image sensors have become capable of generating image signals with high frequency for high-speed continuous shooting of still images and high frame rate shooting of moving images. In order to process image signals that are generated at high speed, a technique is known in which image signals are processed in stages using a plurality of image processing circuits (Patent Document 1). In the technology disclosed in Patent Document 1, first, a first image processing circuit performs preprocessing on the generated image signal at high speed, and then a second image processing circuit performs preprocessing on the output from the first image processing circuit. On the other hand, processing is performed at a frame rate lower than that of the first image processing circuit. At this time, in order to compensate for the difference in processing speed of the circuits, the image signal processed by the first image processing circuit is temporarily stored in the frame memory. Further, Patent Document 1 discloses that in such a configuration, the state of the frame memory is checked to control whether or not high-speed frame rate photography can be performed.

Further, Patent Document 2 discloses a technology that realizes capturing images at a high pixel rate by compressing an image signal in a first image processing circuit and then performing image processing in a second image processing circuit. ing.

Unexamined Japanese Patent Publication No. 2017-192100 Japanese Patent Application Publication No. 2017-153156

Incidentally, an imaging device may be equipped with a plurality of shooting modes, such as high-speed continuous shooting of still images and high frame rate shooting of moving images. In such an imaging device, for example, when trying to balance video shooting, which prioritizes maintaining a high frame rate, and still image shooting, which prioritizes high image quality, it is necessary to use a flexible frame that takes into account aspects other than the frame memory buffer status. Write control to memory is required. In this regard, the above-mentioned conventional technology does not take into consideration writing control to the frame memory that allows both shooting that maintains the frame rate and shooting that maintains high image quality.

SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide a memory write control technique that enables both shooting that maintains a frame rate and shooting that maintains high image quality.

The present invention has been made in view of the above-mentioned problems, and its purpose is to realize a technology that can control writing to a memory in order to achieve both shooting that maintains a frame rate and shooting that maintains high image quality.

In order to solve this problem, for example, the imaging device of the present invention has the following configuration. That is, the imaging device is capable of shooting in a first shooting mode that prioritizes maintaining the frame rate or a second shooting mode that prioritizes image quality, and includes a storage means for buffering image signals, and a storage device that buffers image signals, a first signal processing means for sequentially acquiring image signals from an image sensor and writing the acquired image signals to the storage means for buffering when photographing in the second photographing mode; and second signal processing means for reading out the written image signal and performing image processing, and the first signal processing means is configured to perform processing when the amount of data of the image signal written in the storage means is greater than or equal to a first threshold value. In this case, depending on whether the mode is the first shooting mode or the second shooting mode, the storage means is configured to buffer the image signal obtained by reducing the band of the image signal obtained from the image sensor. The present invention is characterized in that it switches between writing the image signal acquired from the image sensor and interrupting the writing of the image signal acquired from the image sensor into the storage means in order to interrupt the buffering to the storage means .

According to the present invention, it is possible to control writing to a memory in order to achieve both shooting that maintains a frame rate and shooting that maintains high image quality.

A block diagram showing an example of a functional configuration of a digital camera as an example of an imaging device according to Embodiment 1 of the present invention Flowchart showing a series of operations of memory write control processing in the digital camera according to the embodiment Timing chart illustrating the operation of the digital camera according to the present embodiment during still image continuous shooting Timing chart illustrating the operation of the digital camera according to this embodiment when shooting a video at a high frame rate

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the claimed invention. Although a plurality of features are described in the embodiments, not all of these features are essential to the invention, and the plurality of features may be arbitrarily combined. Furthermore, in the accompanying drawings, the same or similar components are designated by the same reference numerals, and redundant description will be omitted.

In the following, a digital camera that processes an image signal output from an image sensor using two signal processing circuits will be described as an example of an imaging device. However, this embodiment is applicable not only to digital cameras but also to other devices capable of processing image signals output from an image sensor using two signal processing circuits. These devices may include, for example, personal computers, mobile phones including smartphones, game consoles, tablet terminals, watch-shaped or glasses-shaped information terminals, medical devices, monitoring systems, in-vehicle system devices, and the like.

(Digital camera configuration)
FIG. 1 is a block diagram showing an example of the functional configuration of a digital camera 100 as an example of the digital camera of this embodiment. Note that one or more of the functional blocks shown in FIG. 1 may be realized by hardware such as an ASIC or a programmable logic array (PLA), or may be realized by a programmable processor such as a CPU or MPU executing software. You can. Alternatively, it may be realized by a combination of software and hardware. Therefore, in the following description, even when different functional blocks are described as operating entities, the same hardware can be implemented as operating entities.

The image sensor 101 has a configuration in which a plurality of pixels each having a photoelectric conversion element are two-dimensionally arranged. The image sensor 101 photoelectrically converts a subject optical image formed by a photographing optical system (not shown) at each pixel, and further converts it from analog to digital using an A/D conversion circuit to obtain a digital signal (image signal) for each pixel. Output. The image sensor 101 may be an image sensor such as a CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

The first signal processing unit 102 includes, for example, an image processing processor, sequentially acquires image signals output from the image sensor 101, and writes the acquired image signals to the main storage unit 107 via the memory controller unit 106. However, according to memory write control processing to be described later, processing for reducing the data amount of the image signal is performed as necessary. The image signal written in the main storage section 107 is read out and processed by the second signal processing section 103. The data processed by the second signal processing unit 103 is then recorded in the recording unit 109.

The bandwidth of the signal that can be processed by the first signal processing section 102 is wider than the bandwidth of the signal that can be processed by the second signal processing section 103. Therefore, the first signal processing unit 102 temporarily buffers the image signal in the main storage unit 107 to absorb the shortage of bandwidth when processing it in the second signal processing unit 103.

For example, in this embodiment, the processing capacity of the first signal processing unit 102 is 8 Gpix/sec, and the processing capacity of the second signal processing unit 103 is 2 Gpix/sec. If the image signal (for example, one screen of a video signal) read from the image sensor 101 is 32 Mpix (8000 horizontal pixels x 8000 vertical lines) and the readout time for one screen is 4 msec, the required band is 8 Gpix/sec. In other words, the second signal processing unit 103 cannot process this image signal within the same time. Therefore, after processing or passing the image signal, the first signal processing unit 102 temporarily buffers the image signal in the main storage unit 107.

The first signal processing unit 102 realizes the functions of functional blocks such as a band reduction unit 104, a system control unit 105, and a memory controller unit 106, for example. The band reduction unit 104 reduces the amount of image signals from the image sensor 101, thereby reducing the band when writing the image signals to the main storage unit 107 via the memory controller unit 106. The band reduction unit 104 reduces the band by performing compression processing on the image signal, for example. However, the present invention is not limited to this example, and the band reduction unit 104 may reduce the band by applying a low-pass filter to the image signal and thinning out (that is, reducing) the signal.

An example will be explained in which the image signal (for example, one screen of a video signal) read out from the image sensor 101 is 32 Mpix (8000 horizontal pixels x 8000 vertical lines) and the readout time is 4 msec. For example, when shooting a video at a high frame rate of 200 images/sec, the second signal processing unit 103 cannot process the image signals completely, and the amount of data accumulated in the buffer area (also simply referred to as a buffer) of the main storage unit 107 decreases. increase In such a case, there is a possibility that the buffer will run out of free space. In this embodiment, in order to reduce the data amount of the image signal to be buffered, the band reduction unit 104 reduces or compresses the image signal to 1/8, for example.

The system control unit 105 uses the bandwidth reduction unit 104 when writing the image signal from the image sensor 101 to the main storage unit 107 (via the memory controller unit 106) according to the shooting mode of the digital camera specified by the user. Controls whether or not to execute processing. In other words, the system control unit 105 switches between providing the image signal via the band reduction unit 104 to the memory controller unit 106 or providing the video signal from the image sensor 101 to the memory controller unit 106 as is. In this embodiment, a case will be described in which the system control unit 105 is arranged in the first signal processing unit 102, but the control unit 108 may also play the role of the system control unit 105.

The memory controller unit 106 controls the write timing and read timing of image signals to the main storage unit 107 . The main storage unit 107 includes a volatile storage medium such as a DRAM, and includes the above-mentioned image signal buffer area. The main storage unit 107 also functions as a work memory when the control unit 108 executes a program stored in the recording unit 109.

The second signal processing unit 103 includes, for example, an image processing processor, and reads the image signal output from the first signal processing unit 102 (and the image signal stored in the main storage unit 107) to improve, for example, image quality. Executes predetermined image processing such as processing and recognition processing.

The control unit 108 includes one or more processors such as a CPU (or MPU), and controls each block of the digital camera 100 by loading and executing programs stored in the recording unit 109 in the main storage unit 107. and control data transfer between each block. Further, the control unit 108 controls each block of the digital camera 100 and controls the shooting mode according to operation signals from the operation unit 110 that accepts operations from the user. The digital camera 100 operates, for example, in a high-speed continuous shooting mode (also simply referred to as still image continuous shooting) in which still images are continuously shot at a predetermined image quality, and in a high-frame rate shooting mode (also referred to simply as still image shooting) in which still images are shot continuously at a predetermined image quality. It is also possible to shoot using video high frame rate shooting.

The recording unit 109 includes a recording medium that is a nonvolatile memory such as a semiconductor memory or a hard disk, and records and holds captured images and the like. Further, the recording unit 109 retains programs (for example, various applications and operating systems) executed by the control unit 108, constants for operation, and the like.

The operation unit 110 includes switches and levers for inputting various operations related to photography, such as a mode switching button, a power button, a still image recording button, and a button for instructing start and stop of video recording. Further, the operation unit 110 includes a menu display button, a decision button, other cursor keys, a pointing device, a touch panel, etc., and transmits an operation signal to the control unit 108 when these keys or buttons are operated by the user.

(Operation overview of memory write control processing)
Next, an outline of the operation of the memory write control process will be explained. The timing chart shown in FIG. 3 shows the state of the amount of data in the buffer and the state of the digital camera 100 when the shooting mode of the digital camera 100 is still image continuous shooting.

Timing t301 indicates the timing at which continuous still image shooting is started by the user. During the period from timing t301 to t302, the amount of image signal writing by the first signal processing unit 102 exceeds the amount of image signal reading by the second signal processing unit 103, so that the data of the image signal buffered in the main storage unit 107 is The amount increases.

Timing t302 indicates the timing at which the buffer of the main storage unit 107 becomes unable to write any further image signals (buffer full state). In addition, in this embodiment, the case where t302 is the timing when the buffer becomes full is explained as an example, but t302 is the timing when the data amount of the image signal in the buffer becomes equal to or greater than a predetermined first threshold value. It may be.

During the period from timing t302 to t303, continuous still image shooting is interrupted (for example, user operation for continuous still image shooting is temporarily not accepted), and the second signal processing unit 103 reads out image signals. Therefore, the data amount of the image signal being buffered decreases.

Timing t303 indicates a state in which the amount of image signal data in the buffer in the main storage unit 107 becomes empty (also referred to as buffer empty). From this timing, the digital camera 100 can resume continuous still image shooting (for example, accept a user operation for continuous still image shooting). In addition, in this embodiment, the case where t303 is the timing when the buffer becomes empty is explained as an example, but t303 is the timing when the data amount of the image signal in the buffer becomes less than a predetermined second threshold value. It may be. Note that the predetermined second threshold is smaller than the predetermined first threshold. For example, if high-speed continuous shooting can be resumed when there is space to write image signals for several to dozens of images (even before the buffer is empty), high-speed continuous shooting can be resumed after it has been interrupted. You will also be able to quickly resume continuous shooting.

In this way, when writing the image signal from the image sensor 101 to the main storage section 107 (via the memory controller section 106), the system control section 105 writes the image signal from the image sensor 101 directly to the memory controller section 106. Control whether to pass or abort.

Next, the outline of the operation of the memory write control process will be further explained with reference to FIG. The timing chart shown in FIG. 4 shows the state of the amount of data in the buffer and the state of the digital camera 100 when the shooting mode of the digital camera 100 is video high frame rate shooting.

Timing t401 indicates the timing at which the user starts capturing a moving image at a high frame rate. During the period from timing t401 to t402, the amount of image signal writing by the first signal processing unit 102 exceeds the amount of image signal reading by the second signal processing unit 103, so the data of the image signal buffered in the main storage unit 107 The amount increases.

Timing t402 indicates the timing at which the main storage unit 107 becomes buffer full. Note that in this embodiment, the timing at which t402 becomes the buffer full state is explained as an example; Good too.

At this time, the system control unit 105 outputs the image signal to the memory controller unit 106 via the band reduction unit 104 when writing the image signal from the image sensor 101 to the main storage unit 107 via the memory controller unit 106. It works like this.

During the period from timing t402 to timing t403, while video high frame rate shooting continues, the amount of data read from the main storage unit 107 exceeds the amount of data when writing the band-reduced image signal to the main storage unit 107. Therefore, the data amount of the image signal being buffered decreases.

Timing t403 indicates the timing when the amount of image signal data in the buffer becomes empty in the main storage unit 107 (that is, the buffer is empty) while video high frame rate imaging continues. After timing t403, the bandwidth deletion process by the bandwidth reduction unit 104 is canceled, and the amount of buffered data increases again as in the case from t401 to t402. Note that t403 may also be a timing when the amount of data of the image signal in the buffer is less than a predetermined second threshold, similar to t303. Note that, as described above, the predetermined second threshold is smaller than the predetermined first threshold.

(Series of operations related to memory write control processing)
Next, with reference to FIG. 2, a series of operations related to memory write control processing in the digital camera 100 will be described.

Note that this process is started, for example, when the power of the digital camera 100 is turned on by a user operation and a still image recording button or a button for instructing the start of video recording included in the operation unit 110 is pressed. Further, this processing is realized by the control unit 108 deploying and executing the program stored in the recording unit 109 in the work area of the main storage unit 107, and controlling each unit such as the first signal processing unit. .

In S201, the control unit 108 receives a user instruction via a still image recording button or a button for instructing the start of video recording, and starts shooting in the shooting mode of still image continuous shooting or moving image high frame rate shooting.

For example, an image signal photoelectrically converted in the image sensor 101 is first read out by the first signal processing unit 102 and buffered in a buffer area of the main storage unit 107 via the memory controller unit 106. Further, at the same time as the image signal is buffered in the main storage unit 107, the image signal buffered in the main storage unit 107 is transmitted to the second signal processing unit 103 via the memory controller unit 106.

In the description of this embodiment, in order to simplify the explanation, an example will be described in which the shooting mode is started as still image continuous shooting or moving image high frame rate shooting. However, the shooting mode may include other modes, and in that case, specific processing corresponding to the other shooting mode may be executed in S203.

In S202, the control unit 108 monitors the state of the buffer in the main storage unit 107 and determines whether the buffer is full (that is, a state in which no further image signal can be written to the buffer). When the control unit 108 determines that the buffer is not in a buffer full state, the control unit 108 returns the process to S202 again. In this case, writing of the image signal to the buffer and reading it to the second signal processing unit 103 are continued. On the other hand, if the control unit 108 determines that the buffer is in a buffer-full state according to the information from the main storage unit 107, the control unit 108 advances the process to S203.

In S203, the control unit 108 determines whether the shooting mode of the digital camera 100 is continuous still image shooting or high frame rate moving image shooting. When the control unit 108 determines that the shooting mode of the digital camera 100 is continuous still image shooting (for example, according to the setting parameters held in the recording unit 109 or the main storage unit 107), the control unit 108 advances the process to S204. On the other hand, if the control unit 108 determines that the shooting mode is video high frame rate shooting, the control unit 108 advances the process to S207.

In S204, the system control unit 105 interrupts still image continuous shooting (because the buffer is full when the shooting mode is high-speed continuous shooting). In this case, the system control unit 105 interrupts writing of the image signal acquired from the image sensor 101 to the main storage unit 107. Further, for example, it is possible to notify the control unit 108 that continuous still image shooting is to be interrupted so that the control unit 108 does not accept user operations for continuous still image shooting. The control unit 108 may display a notification that a user operation for continuous still image shooting is not accepted on a display unit (not shown).

In S205, the control unit 108 monitors the state of the buffer in the main storage unit 107 and determines whether the buffer is empty (that is, the amount of image signal data in the buffer is empty). If the control unit 108 determines that the buffer is empty, the process proceeds to S206, and if not, it repeats the process of S205 while continuing the interrupted still image continuous shooting state.

In S206, the system control unit 105 restarts continuous still image shooting. In this case, the system control unit 105 resumes writing the image signal acquired from the image sensor 101 into the main storage unit 107. Further, the system control unit 105 can, for example, notify the control unit 108 that continuous still image shooting will be restarted, so that the control unit 108 can again accept user operations for continuous still image shooting. The control unit 108 may display a notification indicating that a user operation for continuous still image shooting is accepted on a display unit (not shown). Thereafter, the control unit 108 ends this series of operations in response to a user instruction such as ending continuous shooting.

In S207, the system control unit 105 changes the writing to the main storage unit 107 from uncompressed writing to an image signal with the band removed (because the buffer is full when the shooting mode is video high frame rate shooting). Switch to writing. Specifically, when writing the image signal from the image sensor 101 to the main storage unit 107 via the memory controller unit 106, the system control unit 105 writes the image signal via the band reduction unit 104 to the memory controller unit 106. Switch image writing to pass.

In step 208, the control unit 108 monitors the state of the buffer in the main storage unit 107, and determines whether the buffer is empty (that is, the amount of image signal data in the buffer is empty). If the control unit 108 determines that the buffer status is buffer empty, the process proceeds to S209, and if not, it repeats the process of S208 while continuing the bandwidth reduction writing process.

In S209, the system control unit 105 cancels writing of the image signal whose bandwidth has been reduced. In this case, the system control unit 105 resumes writing to the main storage unit 107 the image signal acquired from the image sensor 101 and for which band deletion has not been performed. At this time, when writing the video signal from the image sensor 101 to the main storage section 107 via the memory controller section 106, image writing is switched so that the video signal from the image sensor 101 is passed as is to the memory controller section 106. Thereafter, the control unit 108 ends this series of operations in response to a user instruction such as to end photographing.

Note that in the above-described embodiments, video high frame rate shooting will be described as an example of shooting that prioritizes maintaining the frame rate, and still image high-speed continuous shooting will be explained as an example of shooting that prioritizes image quality. did. However, the photographing mode is not limited to these. The above-described embodiment may be applied to a case where the digital camera 100 is capable of performing high-speed continuous shooting of two types of still images. In other words, when shooting still images at high speed, priority is given to maintaining the frame rate (maintaining the number of shots per hour), and when shooting still images at high speed, priority is given to image quality. You may. In this case, in high-speed continuous shooting where priority is given to maintaining the frame rate, it is only necessary to switch whether or not to write the image signal whose bandwidth has been reduced to the buffer, depending on the data amount of the image signal in the buffer.

As described above, in this embodiment, the first signal processing unit 102 The image writing to the main storage unit 107 is switched from .

In other words, in the shooting mode that gives priority to maintaining the frame rate, the system control unit 105 switches whether or not to write the image whose bandwidth has been reduced to the buffer, depending on the data amount of the image signal in the buffer. In this way, even if the amount of image signal data in the buffer increases, high frame rate imaging can be continued while reducing the amount of data in the buffer. Furthermore, the image signal whose bandwidth is reduced is used only when the amount of data of the image signal in the buffer increases temporarily. Therefore, while maintaining high frame rate photography, it is possible to limit the images whose image quality deteriorates to a certain range of images.

Furthermore, in a shooting mode that prioritizes image quality, the system control unit 105 controls whether or not to write the image signal acquired from the image sensor to the main storage unit 107 according to the data amount of the image signal in the buffer. did. That is, in the present embodiment, when the shooting mode is still image continuous shooting, and when the amount of image signal data in the buffer of the main storage unit 107 is full (or above a threshold value), the system control unit 105 , temporarily interrupts still image continuous shooting. By doing so, the memory write control process can be controlled so that image quality does not deteriorate in the captured image when still images are captured. In this way, according to the above-described processing, it is possible to control writing to the memory in order to achieve both shooting that maintains the frame rate and shooting that maintains high image quality.

(Other embodiments)
The present invention provides a system or device with a program that implements one or more functions of the embodiments described above via a network or a storage medium, and one or more processors in a computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The invention is not limited to the embodiments described above, and various changes and modifications can be made without departing from the spirit and scope of the invention. Therefore, the following claims are hereby appended to disclose the scope of the invention.

101...Image sensor, 102...First signal processing section, 103...Second signal processing section, 104...Band reduction section, 107...Main storage section

Claims (10)

An imaging device capable of shooting in a first shooting mode that prioritizes maintaining frame rate or a second shooting mode that prioritizes image quality,
storage means for buffering the image signal;
first signal processing for sequentially acquiring image signals from an image sensor and writing the acquired image signals to the storage means for buffering when photographing in the first photographing mode and in the second photographing mode; means and
a second signal processing means for reading out the image signal written in the storage means and performing image processing;
The first signal processing means determines whether the mode is the first photographing mode or the second photographing mode when the amount of data of the image signal written in the storage means is equal to or greater than a first threshold value. Accordingly, an image signal obtained by reducing the band of the image signal obtained from the image sensor is written to the storage means for buffering , or an image obtained from the image sensor is written to the storage means to interrupt buffering to the storage means. An imaging device characterized in that it switches whether or not to interrupt writing of a signal into the storage means. The first signal processing means reduces the band when photographing is being performed in the first photographing mode when the amount of data of the image signal written in the storage means exceeds the first threshold value. 2. The imaging apparatus according to claim 1, wherein the image signal obtained is written into the storage means. The first signal processing means is configured to process data from the image sensor when photographing in the second photographing mode when the amount of data of the image signal written in the storage means exceeds the first threshold value. The imaging apparatus according to claim 1 or 2, wherein writing of the acquired image signal to the storage means is interrupted. The first signal processing means is arranged such that, after the data amount of the image signal written in the storage means becomes equal to or greater than the first threshold, the data amount of the image signal written in the storage means is set to the first threshold. The imaging device according to claim 2 or 3, wherein when the value becomes less than a second smaller threshold, writing of the image signal acquired from the imaging device into the storage means is restarted. The first signal processing means reduces the band of the image signal by compressing the image signal acquired from the image sensor, and writes the image signal with the reduced band into the storage means. 5. The imaging device according to any one of 1 to 4. The first signal processing means reduces the band of the image signal by reducing the image signal acquired from the image sensor, and writes the image signal with the reduced band into the storage means. 5. The imaging device according to any one of 1 to 4. 1 . The first photographing mode is a photographing mode for photographing a moving image at a predetermined frame rate, and the second photographing mode is a photographing mode for photographing a still image at a predetermined image quality. 7. The imaging device according to any one of 6 to 6. A method for controlling an imaging device capable of capturing images in a first imaging mode that prioritizes maintenance of frame rate or a second imaging mode that prioritizes image quality, the imaging device comprising storage means for buffering image signals; The method of controlling the imaging device is
The first signal processing means sequentially acquires image signals from an image sensor of the imaging device and buffers the acquired image signals when photographing in the first photographing mode and when photographing in the second photographing mode. a first signal processing step of writing into the storage means of the imaging device for
a second signal processing step in which the second signal processing means reads out the image signal written in the storage means and performs image processing;
In the first signal processing step, if the amount of data of the image signal written in the storage means is equal to or greater than a first threshold value, the first signal processing step determines whether the mode is the first shooting mode or the second shooting mode. Accordingly, an image signal obtained by reducing the band of the image signal obtained from the image sensor is written to the storage means for buffering , or an image obtained from the image sensor is written to the storage means to interrupt buffering to the storage means. A method for controlling an imaging device, comprising: switching whether to interrupt writing of a signal into the storage means. A program for causing a computer to function as each means of the imaging device according to claim 1 . A recording medium on which a program for causing a computer to function as each means of the imaging apparatus according to claim 1 is recorded.

Images