WO2017022368A1 - Signal processing device and signal processing method - Google Patents

Abstract

[Problem] To provide a signal processing device and a signal processing method that are capable of maintaining moving image visibility and of providing an image suitable for ascertaining a situation. [Solution] A signal processing device comprising: an image acquisition unit that obtains an unprocessed image obtained by image capture and a processed image having a first area and a second area, said first area being the unprocessed image corresponding to at least part of a target display area in a held image obtained by image capture at a point in time prior to the image capture of the unprocessed image, said second area being different from the first area and being the held image; and a display control unit that determines at least either the unprocessed image or the processed image as the display image.

Description

Signal processing apparatus and signal processing method

The present disclosure relates to a signal processing device and a signal processing method.

In recent years, research and development have been conducted on techniques for suppressing the influence of shake of the imaging apparatus during imaging. Such a technique is generally called a shake correction technique and is roughly divided into an optical system and an electronic system.

The optical system is a system that suppresses the influence of shake in imaging by moving an optical system such as a lens or an imaging system such as an imaging surface (image sensor) in accordance with the shake of the imaging apparatus.

In addition, the electronic method captures an image at a wider angle than the display target range, and performs image processing so as to cut out the display target range from an image (hereinafter, also referred to as a video) obtained by the image capturing. This is a method to suppress the influence.

Here, in general, there is an upper limit to the amount of shake that can be corrected by the above-described shake correction technique. For example, in the optical system, when the magnitude of shake of the imaging device exceeds the magnitude of shake corresponding to the movable range of the optical system or the imaging system, an image affected by the shake is generated. In addition, in the electronic method, when the display target range does not fit in the wide-angle imaging range due to shake, an image is not generated for the range that does not fit in the imaging range, and thus a partially missing image is displayed. .

On the other hand, in Patent Document 1, when a shake that is difficult to be corrected by the shake correction technique occurs, a surveillance camera system that switches an image (hereinafter also referred to as a display image or a display image) displayed from a moving image to a still image. The invention which concerns is disclosed. As a result, an image affected by the shake is not displayed, and the viewer can be prevented from feeling tired.

Further, Patent Document 2 discloses an invention relating to an image processing apparatus that uses an electronic image shake correction technique to compensate for a lacking part by shake correction processing with an image obtained by past imaging. As a result, it is possible to prevent an image with a part missing from being displayed.

JP 2006-037662 A JP 2005-027046 A

However, in the inventions disclosed in Patent Documents 1 and 2, it may be difficult for the user to appropriately grasp the situation from the image.

For example, in the invention disclosed in Patent Document 1, since the image is not updated in a situation where the shake is large, it is difficult for the user to observe the image obtained by imaging in the situation.

In the invention disclosed in Patent Document 2, in the shake correction process, an interpolation process is performed not on a past image but on an unprocessed image (hereinafter also referred to as a raw image or a raw video) obtained by imaging. Is supposed to be done. Therefore, it is difficult for the user to observe the raw image.

Therefore, the present disclosure proposes a new and improved signal processing apparatus and signal processing method capable of providing an image suitable for grasping the situation while maintaining the visibility of the video.

According to the present disclosure, the raw image obtained by imaging, and the first area corresponds to at least a part of the display target area in the retained image obtained by imaging at a time before the imaging time of the raw image An image acquisition unit for obtaining a processed image, wherein the second image is a retained image that is a second image different from the first region, and at least one of the raw image and the processed image And a display control unit that determines a display image as a display image.

In addition, according to the present disclosure, at least a part of a display target region in a raw image obtained by imaging and a retained image obtained by imaging the first region before the imaging time of the raw image And a processed image in which a second region different from the first region is the retained image, and at least one of the raw image and the processed image Is determined as a display image. A signal processing method is provided.

As described above, according to the present disclosure, it is possible to provide a signal processing device and a signal processing method capable of providing an image suitable for grasping a situation while maintaining the visibility of an image. Note that the above effects are not necessarily limited, and any of the effects shown in the present specification, or other effects that can be grasped from the present specification, together with or in place of the above effects. May be played.

It is a figure for demonstrating the subject of the conventional shake correction technique. 1 is a system configuration diagram illustrating an example of a schematic physical configuration of a signal processing system according to a first embodiment of the present disclosure. FIG. 2 is a block diagram illustrating an example of a schematic functional configuration of a signal processing system according to the embodiment. It is a figure for demonstrating the production | generation process of the stabilized image | video which concerns on the same embodiment. It is a figure for demonstrating the subject of the object tracking process using a shake correction technique. It is a figure for demonstrating the estimated position display process of the tracking target object in the embodiment. 3 is a flowchart conceptually showing an entire process of the imaging apparatus according to the embodiment. 4 is a flowchart conceptually showing a calculation process of a display target area of the imaging apparatus according to the embodiment. 4 is a flowchart conceptually showing video processing related to object tracking of the imaging apparatus according to the embodiment. 12 is a flowchart conceptually showing a video switching process based on information relating to shake in the first modification of the embodiment. 7 is a flowchart conceptually showing a video switching process based on an operation state of an imaging apparatus according to a first modification of the embodiment. It is a figure showing an example of change of a vector integrated value by operation of an imaging device concerning the 1st modification of the embodiment. It is a figure showing an example of change of a vector integrated value by shake of an imaging device concerning the 1st modification of the embodiment. It is a figure which shows the example of the schematic physical structure of the signal processing system which concerns on the 2nd modification of the embodiment. It is a figure which shows the example of a schematic function structure of the signal processing system which concerns on the 2nd modification of the embodiment. It is a figure which shows the other example of the schematic physical structure of the signal processing system which concerns on the 2nd modification of the embodiment. It is a figure which shows the other example of the schematic function structure of the signal processing system which concerns on the 2nd modification of the embodiment. It is a system configuration figure showing an example of rough physical composition of a signal processing system concerning a 2nd embodiment of this indication. FIG. 2 is a block diagram illustrating an example of a schematic functional configuration of a signal processing system according to the embodiment. 5 is a flowchart conceptually showing a video switching process based on a user operation in the recording apparatus according to the embodiment. 5 is a flowchart conceptually showing a video switching process based on shake in imaging in the recording apparatus according to the embodiment. 6 is a flowchart conceptually showing a video switching process based on a video update status in the recording apparatus according to the embodiment. 5 is a flowchart conceptually showing a video switching process based on occurrence of abnormality in the video in the recording apparatus according to the embodiment. It is a figure which shows the example of a schematic function structure of the signal processing system which concerns on the modification of the embodiment. It is explanatory drawing which showed the hardware constitutions of the signal processing apparatus which concerns on one Embodiment of this indication.

Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

In the present specification and drawings, a plurality of constituent elements having substantially the same functional configuration may be distinguished by adding different numbers after the same reference numerals. For example, a plurality of configurations having substantially the same function are distinguished as necessary, such as the imaging range 10A and the imaging range 10B. However, when it is not necessary to distinguish between substantially the same functional configurations, only the same reference numerals are given. For example, when there is no need to distinguish between the imaging range 10A and the imaging range 10B, they are simply referred to as the imaging range 10.

The description will be made in the following order.
1. Problems of conventional technology First embodiment of the present disclosure (generation of stabilized video)
2-1. Configuration of apparatus 2-2. Technical features 2-3. Processing of apparatus 2-4. Summary of first embodiment 2-5. Modified example 3. Second embodiment of the present disclosure (switching between display of raw video and stabilized video)
3-1. Configuration of apparatus 3-2. Technical features 3-3. Processing of apparatus 3-4. Summary of First Embodiment 3-5. Modified example 4. 4. Hardware configuration of a signal processing device according to an embodiment of the present disclosure Conclusion

<1. Issues of conventional technology>
First, with reference to FIG. 1, the problem of the prior art will be described. FIG. 1 is a diagram for explaining a problem of a conventional shake correction technique.

As described above, shake correction techniques are generally roughly classified into two types: an optical method and an electronic method. In the optical system, when the magnitude of shake of the imaging apparatus exceeds the magnitude of shake corresponding to the optical system or the movable range of the imaging system, it is difficult to suppress the influence of the shake. In the electronic method, if the imaging range changes according to the shake, the display target range is fixed, and if the display target range does not fit in the imaging range, a partially missing video is generated or a video is generated It may not be. Further, when the display target range is not fixed, the display target range moves according to the shake. The latter problem in the electronic system will be described in detail with reference to FIG.

First, the state before the shake occurs will be described. For example, in a state in which no shake has occurred, as shown in the left diagram of FIG. 1, the imaging range 10A is located at the center in the horizontal direction, and the display target range (hereinafter also referred to as a display target region) 20A is imaged. It is located at the center of the range 10A. In this case, an image 30A obtained by extracting the range of the display target region 20A in the imaging range 10A becomes a display image.

Here, when the imaging range 10 </ b> A changes within a range in which the display target area 20 </ b> A can be accommodated by shake, the influence of shake on the display image is suppressed by applying the electronic shake correction technique as described above. The

Next, the state when a shake exceeding the upper limit of shake correction occurs will be described. For example, when shake occurs, as shown in the middle diagram of FIG. 1, the imaging range moves to the imaging range 10B, the display target area is within the imaging range 10B, and the original display target area 20A The display area 20B is moved closer to the area. In this case, 30B obtained by extracting the range of the display target region 20B in the imaging range 10B is a display image.

Next, a description will be given of the state when the shake becomes even larger. For example, when the shake continues, as shown in the right diagram of FIG. 1, the imaging range further moves to the imaging range 10C, and the display target area is within the imaging range 10C and the original display target area 20A or 20B. To the display target region 20C closer to the other region. In this case, 30C obtained by extracting the range of the display target area 20C in the imaging range 10C is the display image.

As described above, when a shake of a size that does not fit the display target area in the imaging range occurs, the conventional shake correction technology cannot completely suppress the influence of the shake on the display video, and the display video is disturbed and the visibility deteriorates. . Furthermore, since the disturbance of the display image increases as the magnitude of the shake increases, the burden on the user who observes the display image increases.

On the other hand, as described above, a technique for preventing or suppressing the disturbance of the display image is disclosed. However, as already described, it is difficult for the user to properly grasp the situation with the display video in these technologies.

Therefore, in the present disclosure, while suppressing deterioration of the visibility of the video due to shake, the situation can be grasped while maintaining the visibility of the signal processing device or the video that allows the user to appropriately grasp the situation from the video. Provided is a signal processing device capable of providing an image suitable for the above. Hereinafter, the signal processing device according to each embodiment of the present disclosure will be described in detail.

<2. First Embodiment of the Present Disclosure (Generation of Stabilized Video)>
In the above, the subject of the prior art was demonstrated. Next, the signal processing system according to the first embodiment of the present disclosure will be described. In the present embodiment, an example in which the signal processing device is an imaging device will be described. For convenience of explanation, the recording apparatus 200 according to the first and second embodiments is distinguished by attaching a number corresponding to the embodiment at the end, like the recording apparatus 200-1 and the recording apparatus 200-2. .

<2-1. Configuration of device>
With reference to FIG. 2, the configuration of the signal processing system according to the first embodiment of the present disclosure will be described. FIG. 2 is a system configuration diagram illustrating an example of a schematic physical configuration of the signal processing system according to the first embodiment of the present disclosure.

The signal processing system according to the present embodiment includes an imaging device 100A, a recording device 200-1A, and a display device 300 as shown in FIG. For example, the signal processing system may be a monitoring system, and the imaging device 100A may be a monitoring camera.

In the signal processing system according to the present embodiment, an image obtained by imaging by the imaging device 100A is transmitted to the recording device 200-1A via communication. There are two types of video to be transmitted, one is a video obtained by imaging and is an unprocessed video, and the other is a processed image generated by a stabilized video generator 107 described later. (Hereinafter also referred to as a stabilized image or a stabilized image). For example, a solid line as shown in FIG. 2 indicates transmission of raw video, and a broken line indicates transmission of stabilized video.

Next, the transmitted video is recorded in the recording device 200-1A. For example, as shown in FIG. 2, both the raw video Data1 and the stabilized video Data2 are recorded on the recording device 200-1A.

Then, the display device 300 displays the video recorded in the recording device 200-1A. For example, as shown in FIG. 2, the stabilized image is provided to the display device 300 and displayed. Note that the video transmitted to the recording device 200-1A may be displayed on the display device as it is through the recording device 200-1A without being recorded.

Subsequently, the functional configuration of the signal processing system according to the present embodiment will be described with reference to FIG. FIG. 3 is a block diagram illustrating an example of a schematic functional configuration of the signal processing system according to the present embodiment.

(Imaging device)
As shown in FIG. 3, the imaging apparatus 100A includes a control unit 101, a lens driving unit 102, an optical system 103, an imager 104, a video signal processing unit 105, a raw video memory 106, a stabilized video generation unit 107, and a retained video. A memory 108, a retained video generation unit 109, a gyroscope 110, and a communication unit 111 are provided.

The control unit 101 generally controls the operation of the imaging apparatus 100A. Specifically, the control unit 101 controls operations of the lens driving unit 102, the imager 104, the video signal processing unit 105, the stabilized video generation unit 107, the retained video generation unit 109, and the communication unit 111. In particular, the control unit 101 controls the operation of the stabilized video generation unit 107 based on information obtained from the gyroscope 110.

The lens driving unit 102 drives a lens included in the optical system 103. For example, the lens driving unit 102 can move the position of the lens using a motor or the like.

The optical system 103 includes a lens, and forms an external image on the imager 104 using the lens. For example, the optical system 103 can include lenses such as a photographing lens and a zoom lens that collect external light.

The imager 104 generates a video signal based on external light. For example, the imager 104 may be a signal conversion element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor).

The video signal processing unit 105 performs a correction process on the video signal generated by the imager 104. For example, the video signal processing unit 105 may be an ISP (Image Signal Processor).

The raw video memory 106 stores the raw video. Specifically, the raw video memory 106 stores information about the video signal obtained from the video signal processing unit 105 (hereinafter, also referred to as video information or data) in units of pixels. For example, in the raw video memory 106, an address corresponding to a pixel is provided, and video information in units of pixels is stored at the address.

The stabilized video generation unit 107 generates a stabilized video based on the raw video and a video obtained by past imaging (hereinafter also referred to as a holding image or a holding video). Specifically, the stabilized video generation unit 107 updates the retained video using the raw video, and cuts out a region to be displayed from the video obtained by the update (hereinafter also referred to as a mixed image or a mixed video). To generate a stabilized video. Then, the stabilized video generation unit 107 provides the communication unit 111 with video information related to the generated stabilized video.

The retained video memory 108 stores the retained video. Specifically, the retained video memory 108 provides the retained video to the stabilized video generation unit 107, and stores the retained video that is generated or updated by the retained video generation unit 109.

The retained video generation unit 109 generates and updates the retained video. Specifically, the holding video generation unit 109 performs holding video processing on the mixed video updated by the stabilized video generation unit 107 using the raw video, and stores the video obtained by the holding video processing. It is stored in the memory 108.

The gyroscope 110 detects the attitude or angular velocity of the imaging device 100A as an inertial sensor. Note that instead of the gyroscope 110, an acceleration sensor may be provided in the imaging apparatus 100A.

The communication unit 111 communicates with the recording device 200-1A. Specifically, the communication unit 111 transmits, as an image providing unit, at least one of the video information related to the stabilized video and the raw video to the recording device 200-1A via communication. For example, the communication unit 111 performs wired communication or wireless communication.

(Recording device)
As illustrated in FIG. 3, the recording apparatus 200-1A includes a control unit 201, a communication unit 202, a storage unit 203, and a video switching unit 204.

The control unit 201 generally controls the operation of the recording apparatus 200-1A. Specifically, the control unit 201 controls operations of the communication unit 202, the storage unit 203, and the video switching unit 204.

The communication unit 202 communicates with the imaging device 100A. Specifically, the communication unit 202 receives a stabilized video and a raw video from the imaging device 100A. Note that the imaging apparatus 100A and the recording apparatus 200-1A may be bi-directional.

The storage unit 203 stores video. Specifically, the storage unit 203 stores both the stabilized video and the raw video received by the communication unit 202.

The video switching unit 204 provides video to the display device 300 as an image providing unit. Specifically, the video switching unit 204 acquires a stabilized video or a raw video received by the communication unit 202 as an image acquisition unit, and provides the acquired video to the display device 300. The video switching unit 204 switches the video provided to the display device 300 based on an instruction from the control unit 201.

(Display device)
The display device 300 displays a video based on the video information provided from the recording device 200-1A. Specifically, the display device 300 includes a display unit that displays a stabilized video or a raw video provided from the recording device 200-1A.

<2-2. Technical features>
Next, technical features of the imaging apparatus 100A according to the present embodiment will be described.

(Generation of stabilized video)
The imaging device 100A generates a stabilized video based on the raw video and the retained video. Specifically, the stabilized video generation unit 107, as an image extraction unit, at least a part of the display target region in the retained video obtained by imaging from the raw video before the raw video is captured. Extract the part corresponding to. Then, the stabilized video generation unit 107 generates a stabilized video that is a raw video from which the first area has been extracted, and a second area that is different from the first area is a retained video. Furthermore, with reference to FIG. 4, the generation process of the stabilized image | video which concerns on this embodiment is demonstrated in detail. FIG. 4 is a diagram for explaining a stabilized video generation process according to the present embodiment.

First, when a video is generated by imaging, the stabilized video generation unit 107 acquires a raw video, and updates the retained video using the acquired raw video. For example, when there is no shake, the stabilized video generation unit 107 generates a raw video obtained by imaging in the imaging range 10D out of the holding video 40D as illustrated in the left diagram of FIG. The part that matches the subject is updated to the raw video.

Further, when shake occurs, the stabilized video generation unit 107 detects the raw video subject obtained by imaging in the imaging range 10E out of the retained video 40E as shown in the middle diagram of FIG. The matching part is updated to the raw video.

In addition, when the shake is further increased, the stabilized video generation unit 107 generates a raw video obtained by imaging in the imaging range 10F out of the retained video 40F as illustrated in the middle diagram of FIG. The part that matches the subject is updated to the raw video.

Next, the stabilized video generation unit 107 extracts a portion corresponding to the display target area from the mixed video. For example, when there is no shake, the stabilized video generation unit 107 extracts a portion corresponding to the display target region 20D as shown in the left diagram of FIG. 4 from the mixed video. In this case, since all of the display target area 20D is within the imaging range 10D, the video 50D, which is all unprocessed video, is extracted.

Further, when shake occurs, the stabilized video generation unit 107 extracts a portion corresponding to the display target area 20E as shown in the middle diagram of FIG. 4 from the mixed video. In this case, since a part of the display target area 20E does not fit from the imaging range 10E, the video 50E including the area 51E that is an unprocessed video and the area 52E that is a retained video is extracted.

Further, when the shake is further increased, the stabilized video generation unit 107 extracts a portion corresponding to the display target area 20F as shown in the middle part of the right diagram of FIG. 4 from the mixed video. In this case, since the display target area 20F is out of the imaging range 10F, the video 50F, which is all retained video, is extracted.

Next, the stabilized video generation unit 107 outputs the extracted video as a stabilized video. For example, the stabilized video generation unit 107 outputs the extracted videos 50D, 50E, and 50F as shown in the lower part of each diagram of FIG. 4 as the stabilized video and displays them as the display video by the display device 300.

Here, the stabilized video is a mode in which a raw video region and a retained video region are distinguished. Specifically, the retained video is processed by the retained video generation unit 109 so as to have a specific visual aspect. The specific visual aspect includes coloring. For example, the retained image 40E is lightly whitened. In addition to the coloring, the specific visual aspect may be a change in brightness or luminance, patterning, emphasis on the outer edge of the retained image, or the like. Further, an object indicating that it is a retained video, for example, a character, a symbol, a figure, or the like may be superimposed on the retained video.

Even if the shake occurs, if the shake is large enough to correct the shake, the shake correction process is performed, and the process when the shake is not generated as a subsequent process is performed. It may be done.

(Display the estimated position of the tracking target object)
The imaging apparatus 100A performs a tracking process for a specific subject appearing in the video. Specifically, the stabilized video generation unit 107 captures a specific subject in the raw video as an image analysis unit. For example, the stabilized video generation unit 107 determines the presence / absence and position of the subject in the raw video based on the feature points of the subject to be identified.

Here, in the object tracking process using the shake correction technique, the observer may lose sight of the tracking target object. A problem in the object tracking process using the shake correction technique will be described in detail with reference to FIG. FIG. 5 is a diagram for explaining the problem of the object tracking process using the shake correction technique.

First, when there is no shake, since the tracking target object is within the imaging range, the position of the tracking target object is captured, and an object indicating the position of the tracking target object is superimposed. For example, as shown in the upper part of the left diagram of FIG. 5, since the person who is the tracking target is within the imaging range 10G, the person is captured by the video analysis process, and the position of the tracking target object is indicated for the human. The object 60G is superimposed. Then, a range corresponding to the display target region 20G is extracted from the imaging range 10G, and the extracted video 30G is displayed as a display video.

On the other hand, when shake occurs, the tracking target object may be out of the imaging range. For example, as shown in the upper part of the right diagram in FIG. 5, since the person who is the tracking target does not fit in the imaging range 10H, the person is not captured. Of course, the person who is the tracking target object is not captured in the display image 30H. As a result, the observer loses sight of the person who is the tracking target object.

Therefore, in the imaging device 100A according to the present embodiment, the stabilized video generation unit 107 estimates when the specific subject to be captured is not included in the unprocessed video area (first area) in the stabilized video. An object indicating the position of a specific subject is superimposed on the stabilized video. This configuration will be described in detail with reference to FIG. FIG. 6 is a diagram for explaining the tracking target object estimated position display processing in the present embodiment.

First, when the tracking target object is within the imaging range and the position of the tracking target object is captured, an object indicating the position of the tracking target object is superimposed. For example, as shown in the upper part of the left diagram of FIG. 6, since the person who is the tracking target is within the imaging range 10I, the person is captured by the video analysis process, and the position of the tracking target object is indicated for the human. An object 60I is superimposed. Then, a range corresponding to the display target area 20I is extracted from the imaging range 10I, and the extracted video 50I is displayed as a display video.

On the other hand, when the tracking target object is out of the imaging range, an object indicating the estimated position of the tracking target object is superimposed.

Specifically, first, the stabilized video generation unit 107 determines whether a tracking target object exists in the raw video region of the stabilized video. For example, the stabilized video generation unit 107 determines whether or not a person who is a tracking target object exists in the unprocessed video region 51J in the video 50J that is a stabilized video as illustrated in FIG.

When it is determined that there is no tracking target object in the unprocessed video area, the stabilized video generation unit 107 estimates the position of the tracking target object based on the position of the past tracking target object. For example, the stabilized video generation unit 107 estimates the position of the tracking target object from the past change in the position of the tracking target object.

Next, the stabilized video generation unit 107 determines whether the estimated position of the tracking target object (hereinafter also referred to as the tracking estimated position) is within the display target area. For example, the stabilized video generation unit 107 determines whether the estimated tracking position is within the display target area 20J, that is, the retained video area 52J.

If it is determined that the estimated tracking position is within the display target area, the stabilized video generation unit 107 superimposes an object indicating the position of the tracking target object on the estimated position. For example, the stabilized video generation unit 107 superimposes the object 60J indicating the position of the tracking target object on the tracking estimation position in the region 52 of the retained video. Then, the stabilized video 50J is displayed as a display video.

In the above description, an example in which the estimated tracking position is within the display target area, that is, the retained video area has been described. However, when the estimated tracking position is outside the display target area, the object indicating the position of the tracking target object is stable. It may be superimposed on the digitized video. For example, when the estimated tracking position is outside the display target area, the stabilized video generation unit 107 can superimpose an object such as an arrow indicating the estimated tracking position.

(Change display area)
The imaging apparatus 100A changes the display target area based on the size of the raw video area in the stabilized video. Specifically, when the size of the raw video area is equal to or smaller than the threshold value, the stabilized video generation unit 107 determines the display target area so that the raw video area becomes large. For example, the stabilized video generation unit 107 determines that the ratio of the size of the raw video area in the display target area is a threshold in a range corresponding to the display target area in the video obtained by updating the retained video using the raw video. In the following cases, the region of the raw video is enlarged by shifting the display target region.

In addition, although the comparison with the threshold value of the size of the unprocessed video has been described as the ratio in the display target area, it may be performed by the total value of the area or the side. In addition, the size of the display target area may be changed.

<2-3. Device processing>
Next, processing of the imaging device 100A according to the present embodiment will be described.

(Flow of the entire process)
First, the overall processing of the imaging apparatus 100A will be described with reference to FIG. FIG. 7 is a flowchart conceptually showing the entire processing of the imaging apparatus 100A according to the present embodiment.

The imaging apparatus 100A detects shake (step S401). Specifically, the control unit 101 detects the presence / absence of shake and the magnitude of shake based on information such as angular velocity obtained from the gyroscope 110.

Next, the imaging apparatus 100A calculates a region of the retained video corresponding to the raw video (Step S402). Specifically, the stabilized video generation unit 107 calculates a region of a retained video in which a subject corresponding to a raw video subject obtained by imaging is captured.

Next, the imaging apparatus 100A converts the address of the raw video into the address of the retained video (Step S403). Specifically, the stabilized video generation unit 107 converts the address of the raw video into the address of the retained video corresponding to the raw video.

Next, the image capturing apparatus 100A updates the address data related to the conversion on the retained video memory 108 to the raw video data (step S404). Specifically, the stabilized video generation unit 107 updates the data stored in the address of the retained video corresponding to the converted raw video address to the raw video data.

Next, the imaging apparatus 100A calculates the address of the display target area (step S405). Details will be described later.

Next, the imaging apparatus 100A reads the data of the calculated address on the retained video memory 108 (step S406). Specifically, the stabilized video generation unit 107 reads out data stored at the address of the retained video corresponding to the calculated display target area address.

Next, the imaging apparatus 100A determines the presence / absence of an object tracking process (step S407). Specifically, the stabilized video generation unit 107 determines whether the object tracking process is continuing or started.

If it is determined that the object tracking process is being performed, the imaging apparatus 100A performs a video process related to the object tracking (step S408). Details will be described later. If it is determined not to perform the object tracking process, the process proceeds to step S409.

Next, the imaging apparatus 100A outputs a stabilized video (step S409). Specifically, the stabilized video generation unit 107 outputs the data read from the holding video memory 108 or the data related to the video obtained by performing the video processing related to the object tracking on the video related to the read data. Provided to the communication unit 111. Then, the communication unit 111 transmits the provided data to the recording device 200-1A.

Next, the imaging device 100A updates the raw video on the held video memory 108 to the held video (step S410). Specifically, the retained video generation unit 109 performs a coloring process on the raw video on the retained video memory 108.

(Flow of display target area calculation processing)
Next, the calculation process of the display target area of the imaging device 100A will be described with reference to FIG. FIG. 8 is a flowchart conceptually showing a calculation process of the display target area of the imaging apparatus 100A according to the present embodiment.

The imaging apparatus 100A determines whether or not the display target area is controlled based on the display ratio (step S421). Specifically, the stabilized video generation unit 107 determines whether or not to change the display target area based on the ratio of the unprocessed video in the display target area, that is, the stabilized video. Of course, the ratio of the retained video may be used instead of the ratio of the raw video.

When it is determined that the display target area is controlled based on the display ratio, the imaging apparatus 100A calculates the ratio of the unprocessed video in the display target area (step S422). Specifically, the stabilized video generation unit 107 specifies a display target area that has already been set in the mixed video. Then, the stabilized video generation unit 107 calculates the ratio of the raw video area in the display target area.

If it is determined not to control the display target area based on the display ratio, the process proceeds to step S425.

Next, the imaging apparatus 100A determines whether or not the ratio of the unprocessed video is equal to or greater than a threshold (step S423). Specifically, the stabilized video generation unit 107 determines whether the calculated ratio of raw video in the display target area is equal to or greater than a threshold value. The threshold value may be set in advance or may be changed afterwards.

If it is determined that the ratio of unprocessed video is equal to or greater than the threshold, the process proceeds to step S425.

When it is determined that the ratio of the unprocessed video is less than the threshold, the imaging device 100A recalculates the position of the display target area (step S424). Specifically, the stabilized video generation unit 107 shifts the display target area so that the ratio of raw video increases.

Next, the imaging apparatus 100A calculates an address on the retained video memory 108 corresponding to the display target area (step S425). Specifically, the stabilized video generation unit 107 calculates the address of the mixed video area specified by the display target area.

(Object tracking process flow)
Next, with reference to FIG. 9, video processing related to object tracking of the imaging device 100 </ b> A will be described. FIG. 9 is a flowchart conceptually showing video processing relating to object tracking of the imaging apparatus 100A according to the present embodiment.

The imaging apparatus 100A determines whether or not the tracking target object exists in the unprocessed video area (step S431). Specifically, the stabilized video generation unit 107 determines whether a tracking target object exists in an unprocessed video area in the video related to the read data. Of course, the stabilized video generation unit 107 may determine whether the tracking target object exists in the raw video using only the raw video.

If it is determined that there is no tracking target object in the unprocessed video area, the imaging apparatus 100A calculates a tracking estimation position (step S432). Specifically, the stabilized video generation unit 107 estimates the position of the tracking target object based on the past change in the position of the tracking target object.

Next, the imaging apparatus 100A determines whether the tracking target object exists in the display target area (step S433). Specifically, the stabilized video generation unit 107 determines whether the tracking estimated position is within the display target area.

If it is determined that the tracking target object is present in the display target area, the imaging device 100A superimposes an object indicating the estimated tracking position (step S434). Specifically, the stabilized video generation unit 107 superimposes an object indicating the estimated tracking position on the video related to the read data.

<2-4. Summary of First Embodiment>
As described above, according to the first embodiment of the present disclosure, the imaging device 100A is obtained as a signal processing device from an unprocessed image obtained by imaging by imaging at a time before the imaging time of the unprocessed image. A portion corresponding to at least a part of the display target region in the held image is extracted. In addition, the imaging apparatus 100A generates a processed image in which the first region is a raw image extracted by the image extraction unit, and a second region different from the first region is a retained image. For this reason, even if the display target area does not fit in the imaging range due to shake, the area where no unprocessed video exists is supplemented with the retained video, thereby suppressing deterioration in the visibility of the video due to shake. It becomes possible and the burden on the user can be reduced. In addition, since the unprocessed video that is not processed from the video obtained by imaging is used as the display video, the user can appropriately grasp the situation from the video.

The processed image includes an image in which at least one of the first region and the second region is discriminated. For this reason, the stabilized video is generated so that the raw video and the retained video are discriminated, so that the user can grasp the situation at the time when the image was captured by observing the region of the raw video. . As a result, it is possible to observe the situation at the time of imaging of the display target area while maintaining at least the visibility.

Also, the second region of the processed image is a specific visual aspect. For this reason, it becomes possible for a user to improve a user's convenience by distinguishing each area | region at a glance.

Also, the specific visual aspect includes a specific coloring. For this reason, the area of the retained video is displayed in such a manner that a human can easily recognize the difference, so that the user can quickly grasp the area of the retained video.

Also, images obtained by imaging are communicated via a network. For this reason, it is possible to generate a stabilized video even for a video obtained by remote imaging.

In addition, the raw image includes an image obtained by imaging in a situation where the display target area does not fit in the imaging range. For this reason, it is possible to provide a display image in which the disturbance is suppressed even when an image obtained by imaging in a situation where shake correction is difficult is acquired.

Further, the imaging apparatus 100A captures a specific subject in the unprocessed image, and when the captured specific subject is not included in the first region, an object indicating the estimated position of the specific subject is used as the processed image. Superimpose. For this reason, even if the tracking target object is not shown in the raw video region of the stabilized video, the tracking estimated position is indicated, so that the user can hardly lose track of the tracking target object.

In addition, when the size of the first area is equal to or smaller than the threshold, the imaging apparatus 100A determines the display target area so that the first area becomes larger. For this reason, it becomes possible to easily grasp the situation at the time of imaging by increasing the area of the raw video in the stabilized video.

Also, both processed images and unprocessed images are recorded. Here, for example, in the surveillance system, the video evidence ability is important. Since the stabilized video includes the retained video, only the raw video area has evidence capability. However, since the stabilized video is a processed video, there is a possibility that the evidence ability is suspected. Therefore, by recording not only the stabilized video but also the raw video as in the present configuration, the stabilized video can be used for monitoring and the raw video can be used as evidence.

<2-5. Modification>
Heretofore, the first embodiment of the present disclosure has been described. In addition, this embodiment is not limited to the above-mentioned example. Below, the 1st and 2nd modification of this embodiment is demonstrated.

(First modification)
As a first modification of the present embodiment, the imaging device 100A may switch the provided video. Here, there are several situations where it is desirable to provide a stabilized video. Specifically, there are cases where the imaging device is installed far from the location to be photographed, or where the imaging device is not fixed.

For example, when the imaging device is installed far away from the location to be photographed, there are cases where a subject such as a volcano, the sea or a river that is difficult for humans to approach is subject to photography, or a disaster site is subject to photography. In addition, there are cases where the imaging device is not fixed, for example, when it is temporarily installed in a police investigation or investigation, or when the installation location is not stable, for example, when it is affected by traffic or wind.

Therefore, in the first modification of the present embodiment, the control unit 101 switches the video transmitted by the communication unit 111 based on information related to the state of the device that performs imaging. For example, the control unit 101 switches between a mode in which a stabilized video is transmitted (stabilization mode ON) and a mode in which an unprocessed video is transmitted (stabilization mode OFF). As a method for switching video, there are video switching based on information relating to shake of the imaging device 100A and video switching based on the operating state of the imaging device 100A.

(Video switching based on shake information)
The control unit 101 switches the video to be transmitted to the communication unit 111 based on the degree of shake in imaging. Specifically, the degree of shake in imaging is estimated based on the movement of the imaging device 100A. Specifically, the movement of the imaging apparatus 100A is estimated based on information obtained from an inertial sensor (hereinafter also referred to as sensor information). For example, the control unit 101 calculates a motion vector based on information such as angular velocity obtained from the gyroscope 110. Then, the control unit 101 switches the stabilization mode ON and OFF based on the calculated magnitude of the motion vector.

Note that the movement of the imaging apparatus 100A may be estimated based on a video obtained by imaging. For example, the control unit 101 calculates a motion vector based on a time-series change of the raw video. Of course, the stored image may be used for estimating the motion of the imaging apparatus 100A.

Further, the control unit 101 may switch the video to be transmitted to the communication unit 111 based on the possibility of the occurrence of shake in imaging. Specifically, the possibility of occurrence of shake in imaging is estimated based on whether or not the subject is enlarged in the video obtained by imaging. For example, the control unit 101 determines the presence or absence of a zoom-in operation in imaging based on the magnitude of the focal length. When the focal length is larger than the threshold value, the control unit 101 sets the stabilization mode to ON because the zoom-in operation is being performed. Note that the threshold for the focal length may be determined in advance or may be changed afterwards.

Further, with reference to FIG. 10, a video switching process based on information related to shake of the present modification will be described. FIG. 10 is a flowchart conceptually showing a video switching process based on information relating to shake in the first modification of the present embodiment.

The imaging apparatus 100A determines whether the focal length can be acquired (step S441). Specifically, the control unit 101 determines whether the focal length of the lens included in the optical system 103 can be acquired.

If it is determined that the focal length can be acquired, the imaging apparatus 100A determines whether the focal length exceeds the threshold L (step S442). Specifically, the control unit 101 acquires the focal length of the lens and determines whether the acquired focal length exceeds a threshold value L.

If it is determined that the focal length exceeds the threshold L, the imaging apparatus 100A sets the stabilization mode to ON (step S443).

If it is determined in step S441 that the focal length cannot be acquired, or if it is determined in step S442 that the focal length does not exceed the threshold L, the imaging device 100A calculates a motion vector (step S444). ). Specifically, the control unit 101 calculates a motion vector of the imaging apparatus 100A based on information obtained from the gyroscope 110 or video analysis.

Next, the imaging device 100A determines whether the stabilization mode is OFF (step S445).

If it is determined that the stabilization mode is OFF, the imaging device 100A determines whether the magnitude of the motion is greater than or equal to the threshold value a (step S446). Specifically, the control unit 101 determines whether the magnitude of the calculated motion vector is greater than or equal to the threshold value a.

If it is determined that the magnitude of the motion is greater than or equal to the threshold value a, the process proceeds to step S443, and the stabilization mode is set to ON. Otherwise, the stabilization mode is not changed and remains OFF.

If it is determined in step S445 that the stabilization mode is ON, the imaging device 100A observes the movement during the predetermined period T1 (step S447). Specifically, the control unit 101 continues to calculate motion vectors periodically during a predetermined period T1.

Next, the imaging apparatus 100A determines whether or not the magnitude of movement in the predetermined period T1 is equal to or less than the threshold value b (step S448). Specifically, the control unit 101 determines whether all the motion vectors calculated during the predetermined period T1 are equal to or less than the threshold value b.

When it is determined that the magnitude of the movement in the predetermined period T1 is equal to or less than the threshold value b, the imaging device 100A sets the stabilization mode to OFF (step S449). Otherwise, the control unit 101 does not change the stabilization mode and keeps it ON. Note that the control unit 101 may switch the stabilization mode based on the length of the period that is equal to or less than the threshold value b.

(Video switching based on the operating state of the imaging device)
The control unit 101 switches the video to be provided according to the operation state for imaging. Specifically, the operation state for imaging includes the presence or absence of an operation related to the movement of the imaging range. For example, operations related to the movement of the imaging range include a horizontal rotation operation in the imaging direction, a so-called pan operation, and a vertical rotation operation in the imaging direction, a so-called tilt operation.

Also, the operation state for imaging includes the presence or absence of an operation related to a change in the size of the subject. For example, there are zoom-in and zoom-out operations related to the change in the size of the subject. The zoom may be either optical or digital.

The operation related to the movement of the imaging range and the operation related to the change in the size of the subject are determined based on the operation information of the imaging apparatus 100A. Note that, instead of the operation information of the imaging apparatus 100A, an operation related to movement of the imaging range and an operation related to a change in the size of the subject may be determined based on the video analysis result.

Furthermore, with reference to FIG. 11, a video switching process based on the operation state of the imaging apparatus of the present modification will be described. FIG. 11 is a flowchart conceptually showing a video switching process based on the operation state of the imaging apparatus in the first modification of the present embodiment. Note that description of processing that is substantially the same as the processing described above is omitted.

The imaging apparatus 100A determines whether the operation information of the imaging apparatus 100A can be acquired (step S451). Specifically, the control unit 101 determines whether operation information related to zoom, pan, and tilt of the imaging apparatus 100A can be acquired.

If it is determined that the operation information of the image capturing apparatus 100A can be acquired, the image capturing apparatus 100A determines whether the zoom operation is being performed (step S452). Specifically, the control unit 101 determines whether the operation information related to zooming indicates that zooming in or zooming out is being performed.

If it is determined that the zoom operation is being performed, the imaging apparatus 100A sets the stabilization mode to OFF (step S453).

If it is determined in step S452 that the zoom operation is not being performed, the imaging device 100A determines whether the panning or tilting operation is being performed (step S454). Specifically, the control unit 101 determines whether operation information related to panning and tilting indicates that a panning or tilting operation is being performed.

If it is determined that the panning or tilting operation is being performed, the imaging apparatus 100A sets the stabilization mode to OFF (step S453), and if not, sets the stabilization mode to ON (step S455).

If it is determined in step S451 that the operation information of the imaging apparatus 100A cannot be acquired, the imaging apparatus 100A calculates a motion vector (step S456).

Next, the imaging apparatus 100A determines whether the size of the subject has changed (step S457). Specifically, the control unit 101 determines whether the size of the subject is increased or decreased by analyzing time-series video.

If it is determined that the size of the subject has changed, the imaging apparatus 100A sets the stabilization mode to OFF (step S458).

If it is determined in step S457 that the size of the subject has not changed, the imaging device 100A observes the movement during the predetermined period T2 (step S459).

Next, the imaging apparatus 100A determines whether or not the integrated value of the magnitude of movement in the predetermined period T2 is equal to or less than a threshold value (step S460). Specifically, the control unit 101 integrates motion vectors calculated during a predetermined period T2, and determines whether a value obtained by the integration is equal to or less than a threshold value. Further, the motion vector integrated value will be described in detail with reference to FIGS. 12A and 12B. FIG. 12A is a diagram illustrating an example of a change in the vector integrated value due to the operation of the imaging apparatus 100A according to the present modification, and FIG. 12B is an example of a change in the vector integrated value due to the shake of the imaging apparatus 100A according to the present modification. FIG.

Since the operation of the imaging apparatus 100A, for example, panning or tilting is performed in a certain direction, the vector integrated value by the operation generally increases linearly as shown in FIG. 12A. On the other hand, the shake direction of the imaging apparatus 100A alternately changes in the direction in which it occurred and in the direction opposite to the direction in which it occurred, so that the integrated values of the vectors due to shake generally cancel each other as shown in FIG. And eventually converge toward 0. Note that the threshold used for the determination in step S460 may be 0 or a value regarded as 0.

Returning to the description of the flowchart of the video switching process based on the operation state of the imaging apparatus with reference to FIG. The apparatus 100A sets the stabilization mode to ON (step S461), and otherwise sets the stabilization mode to OFF (step S458).

In the above description, the stabilization mode is turned off when the zoom operation or the panning or tilting operation is being performed. However, the association between the operation information and the stabilization mode is another association. Also good. For example, the association shown in Table 1 below is assumed.

As described above, according to the first modification of the present embodiment, the imaging device 100A switches the image to be provided to the external device based on the information related to the state of the device that performs imaging. For this reason, by providing an image suitable for the state of the imaging device 100A to the external device, both the visibility of the image and the user's needs for the image can be achieved.

In addition, the imaging apparatus 100A switches images to be provided based on the degree of shake in imaging. For this reason, when the stabilization mode is turned on when a shake of a predetermined magnitude occurs, the processing load can be reduced as compared with the case where the stabilization video generation process is always performed.

Also, the degree of shake in imaging is estimated based on the movement of the device that performs imaging. For this reason, it is possible to improve the accuracy and accuracy of estimation by estimating the degree of shake based on the movement of the imaging apparatus 100A in which shake has occurred.

Also, the movement of the device that performs imaging is estimated based on information obtained from the inertial sensor. For this reason, it is possible to suppress an increase in processing load as compared to a case where video analysis for estimating the motion of the imaging apparatus 100A is performed.

Also, the movement of the apparatus that performs imaging is estimated based on the image obtained by imaging. For this reason, it becomes possible to reduce cost compared with the case where the estimation process based on the information obtained from an inertial sensor is performed.

Further, the imaging apparatus 100A switches the provided image based on the possibility of occurrence of shake in imaging. For this reason, the stabilization mode is turned ON in advance when the possibility of shake is high, so that it is possible to suppress image disturbance compared to the case where the stabilization mode is switched after the occurrence of shake. It becomes.

Also, the possibility of shake during imaging is estimated based on whether or not the subject is enlarged in the image obtained by imaging. Here, during the zoom-in operation, the image is more susceptible to shake than when the zoom-in operation is not performed. Therefore, during the zoom-in operation, the stabilization of the stabilization mode can be suppressed to reduce the visibility of the video.

Also, the imaging apparatus 100A switches the provided image according to the operation state for imaging. For this reason, when the state of the stabilized video estimated from the operation state of the imaging device 100A is not suitable for visual recognition, switching to an unprocessed video can suppress degradation in visibility due to the operation of the imaging device 100A. It becomes possible.

Also, the operation state for imaging includes the presence / absence of operation related to movement of the imaging range. Here, during the panning or tilting operation, since the imaging range moves, there may be a region that has not yet been imaged. In this case, since there is no area of the retained video corresponding to the raw video, a part of the stabilized video may be lost. However, according to this configuration, the stabilization mode is turned off during the panning or tilting operation, that is, the displayed video is switched to the raw video, thereby providing the user with a complete video. It becomes possible.

Also, the operation state for imaging includes the presence or absence of an operation related to a change in the size of the subject. Here, during the zoom operation, the subject that appears in the unprocessed video is enlarged or reduced, so that the held video is also enlarged or reduced accordingly. Thereby, the retained image may be deteriorated. However, according to this configuration, during the zoom operation, the stabilization mode is turned off, that is, the displayed video is switched to the raw video, thereby preventing the video from being deteriorated.

(Second modification)
As a second modification of the present embodiment, another device different from the imaging device may operate as a signal processing device. Specifically, the recording device 200-1B operates as a signal processing device. With reference to FIG. 13, a configuration of a signal processing system according to a second modification of the present embodiment will be described. FIG. 13 is a diagram illustrating an example of a schematic physical configuration of a signal processing system according to a second modification of the present embodiment.

(Recording device that generates stabilized video from video information obtained from the imaging device)
As shown in FIG. 13, the signal processing system includes an imaging device 100B, a recording device 200-1B, and a display device 300. For example, the recording device 200-1B receives the raw video from the imaging device 100B and generates a stabilized video based on the received raw video and the retained video. Next, the recording device 200-1B provides the display device 300 with video information regarding the generated stabilized video. Note that the recording apparatus 200-1B records the raw image Data1 and the stabilized video Data2 as shown in FIG. Then, the display device 300 displays a stabilized video related to the provided video information. Furthermore, the functional configuration of the signal processing system according to the present modification will be described with reference to FIG. FIG. 14 is a diagram illustrating an example of a schematic functional configuration of a signal processing system according to a second modification of the present embodiment. Here, only the recording apparatus 200-1B will be described.

As shown in FIG. 14, in addition to the control unit 201, the communication unit 202, the storage unit 203, and the video switching unit 204, the recording device 200-1B includes a raw video memory 205, a stabilized video generation unit 206, a retained video. A memory 207, a retained video generation unit 208, and a motion detection unit 209 are provided. Note that the functions of the communication unit 202 and the storage unit 203 are substantially the same as the functions described above, and a description thereof will be omitted. The functions of the control unit 201, raw video memory 205, stabilized video generation unit 206, retained video memory 207, and retained video generation unit 208 are the same as the control unit 101, raw video memory 106, and stabilized video generation unit described above. 107, the function of the retained video memory 108 and the retained video generation unit 109 are substantially the same, and the description thereof will be omitted.

The motion detection unit 209 detects the motion of the imaging device 100B based on the raw video. Specifically, the motion detection unit 209 estimates the motion of the imaging device 100B based on the video obtained by imaging, similarly to the processing of the control unit 101 described in the first embodiment. When sensor information of the imaging device 100B at the time of imaging is obtained from the imaging device 100B via the communication unit 202, the motion detection unit 209 may detect the motion of the imaging device 100B based on the sensor information. .

Also, the motion detection unit 209 may estimate the possibility of occurrence of shake during imaging. For example, the motion detection unit 209 estimates the possibility of occurrence of shake in imaging based on the presence / absence of enlargement of the subject in the video, similar to the processing of the control unit 101 described in the first embodiment.

(Recording device that generates stabilized video from recorded video information)
Another recording device 200-1C operating as a signal processing device generates a stabilized video based on the raw video to be recorded and the retained video generated in the process. First, another configuration of the signal processing system according to the present modification will be described with reference to FIG. FIG. 15 is a diagram illustrating another example of the schematic physical configuration of the signal processing system according to the second modification of the present embodiment.

As shown in FIG. 15, the signal processing system includes a recording device 200-1C and a display device 300. For example, the recording device 200-1C generates a stabilized video based on the recorded raw video Data1 and a retained video generated in the process. Next, the recording device 200-1C provides the display device 300 with video information regarding the generated stabilized video. Then, the display device 300 displays a stabilized video related to the provided video information. Furthermore, the functional configuration of the signal processing system will be described with reference to FIG. FIG. 16 is a diagram illustrating another example of the schematic functional configuration of the signal processing system according to the second modification of the present embodiment. Here, only the recording apparatus 200-1C will be described.

As shown in FIG. 16, the recording apparatus 200-1C includes a control unit 201, a storage unit 203, a video switching unit 204, a raw video memory 205, a stabilized video generation unit 206, a retained video memory 207, and a retained video generation unit. In addition to 208 and the motion detector 209, a playback processor 210 is provided. The functions of the control unit 201, the raw video memory 205, the stabilized video generation unit 206, the retained video memory 207, the retained video generation unit 208, and the motion detection unit 209 are substantially the same as those described above. Is omitted.

The playback processing unit 210 controls video playback based on instructions from the control unit 201. Specifically, when the control unit 201 instructs the reproduction processing unit 210 to start reproduction, the reproduction processing unit 210 acquires the video information of the raw video from the storage unit 203 and provides the video information to the raw video memory. In addition, when the control unit 201 instructs the playback processing unit 210 to stop or end playback, the playback processing unit 210 ends or stops the acquisition of video information from the storage unit 203.

Thus, according to the second modification of the present embodiment, the recording apparatus 200 operates as a signal processing apparatus. For this reason, a general imaging device can be employed in the signal processing system. Further, it is possible to display the stabilized video on the display device 300 even when not connected to the imaging device.

<3. Second Embodiment of the Present Disclosure (Switching Display between Raw Video and Stabilized Video)>
The signal processing system according to the first embodiment of the present disclosure has been described above. Next, a signal processing system according to the second embodiment of the present disclosure will be described. In the present embodiment, an example in which the signal processing apparatus is a recording apparatus will be described.

<3-1. Configuration of device>
The configuration of the signal processing system according to the present embodiment will be described with reference to FIG. FIG. 17 is a system configuration diagram illustrating an example of a schematic physical configuration of a signal processing system according to the second embodiment of the present disclosure. Note that a description of a physical configuration that is substantially the same as the signal processing system described above will be omitted.

The signal processing system according to the present embodiment includes a recording device 200-2A and a display device 300 as shown in FIG.

In the signal processing system according to the present embodiment, video information is recorded in advance in the recording device 200-2A. For example, as shown in FIG. 17, both the raw video Data1 and the stabilized video Data2 are recorded in the recording device 200-2A.

Then, the display device 300 displays the video provided from the recording device 200-2A. For example, as shown in FIG. 17, a stabilized video or a raw video is provided to the display device 300, and the display device 300 displays the provided video.

Subsequently, the functional configuration of the signal processing system according to the present embodiment will be described with reference to FIG. FIG. 18 is a block diagram illustrating an example of a schematic functional configuration of the signal processing system according to the present embodiment.

As shown in FIG. 18, the recording device 200-2A includes a control unit 201, a storage unit 203, a video switching unit 204, a reproduction processing unit 210, and a video buffer 211. In addition, description is abbreviate | omitted about the function substantially the same as the function mentioned above.

The video buffer 211 temporarily holds video. Specifically, the video buffer 211 holds an image from the display time point of the displayed video until the time point before the display time point in a reproducible state. More specifically, the video buffer 211 temporarily buffers the raw video provided from the playback processing unit 210 and is more time-sequential than the raw video provided from the playback processing unit 210 to the video switching unit 204. The raw video delayed to the video switching unit 204 is provided to the video switching unit 204. For example, the video buffer 211 provides the video switching unit 204 with a raw video of a predetermined number of frames before the raw video frame provided from the reproduction processing unit 210 to the video switching unit 204 in time series. The time-series delay may be designated from the control unit 201 or the video switching unit 204, or may be a fixed value.

<3-2. Technical features>
Next, technical features of the recording apparatus 200-2A according to the present embodiment will be described.

(A. Switching video)
The recording device 200-2A switches the video to be displayed on the display device 300. Specifically, the control unit 201 determines, as a display control unit, at least one of a raw video and a stabilized video as a display video. Here, the video switching is performed based on various information as described below.

(A-1. Video switching based on user operation)
The control unit 201 determines a display video based on a user operation. Specifically, the control unit 201 determines the display video according to the user's video switching operation on the recording device 200-2A. For example, when a user who feels an abnormality in the video performs a video switching operation on the recording device 200-2A while the stabilized video is being displayed on the display device 300, the control unit 201 causes the video switching unit 204 to The video provided to the display device 300 is switched from the stabilized video to the raw video.

(A-2. Video switching based on camera shake)
The control unit 201 determines a display image based on information related to shake in imaging. Specifically, the information related to shake in imaging includes information related to the degree of shake in imaging. More specifically, the control unit 201 estimates the degree of shake in imaging based on at least one of the raw video and the stabilized video. For example, the control unit 201 calculates a motion vector based on a time-series change of the raw video. Next, the control unit 201 selects a raw image or a stabilized image based on a comparison between the calculated motion vector and a threshold value. Then, the control unit 101 causes the video switching unit 204 to switch the video provided to the display device 300 to the selected video.

When sensor information of the imaging device at the time of imaging is recorded in advance in the recording device 200-2A together with the raw video and the stabilized video, the control unit 201 estimates the degree of shake in imaging based on the sensor information. May be.

Also, the control unit 201 may estimate the possibility of shake during imaging. For example, the control unit 201 estimates the possibility of occurrence of shake in imaging based on the presence / absence of enlargement of the subject in the video, similarly to the processing of the control unit 101 described in the first embodiment.

(A-3. Video switching based on video update status)
The control unit 201 determines the display video based on the display video display update interval. Specifically, the control unit 201 determines a display video based on a comparison between the update interval and a threshold value. For example, the control unit 201 is provided to the display device 300 in the video switching unit 204 when the displayed stabilized video has not been updated for a predetermined period, for example, when the stabilized video includes only the retained video area. To switch to a raw video.

Note that the control unit 201 may cause the video switching unit 204 to switch the display video to the raw video even when the stabilized video includes a raw video region. Specifically, the control unit 201 determines the display video based on the size of the raw video area. More specifically, the control unit 201 determines the display video based on the ratio of the raw video region in the stabilized video. For example, the control unit 201 causes the video switching unit 204 to switch the video provided to the display device 300 to the raw video when the ratio of the raw video region in the displayed stabilized video is equal to or less than the threshold.

Further, the control unit 201 may determine the display video based on the display time of the raw video area. Specifically, the control unit 201 determines the display video based on the ratio of the display time of the raw video area to the display time of the stabilized video. For example, the control unit 201 provides the video switching unit 204 to the display device 300 when the time during which the stabilized video includes an unprocessed video area falls below a predetermined ratio with respect to the display time of the stabilized video. Switch video to raw video.

(A-4. Video switching based on occurrence of abnormality in video)
The control unit 201 determines a display video based on information related to the content of at least one of the raw video and the stabilized video. Specifically, the information related to the content of the video includes information related to whether or not a characteristic object is included in the video. More specifically, whether or not a characteristic object is included in the video is determined based on the analysis of the video. For example, while the stabilized video is displayed, the control unit 201 determines whether an object that is determined to be abnormal is included in the stabilized video by video analysis. Then, when it is determined that the object is included in the stabilized video, the control unit 201 causes the video switching unit 204 to switch the video provided to the display device 300 from the stabilized video to the raw video. In addition to the solid object as described above, the characteristic object may be a liquid such as water or oil, or a gas such as cloud or smoke.

Note that the presence / absence of inclusion of a characteristic object in the video may be determined based on meta information of the video. For example, the meta information of the video includes information indicating the display time point (or frame) of the stabilized video where the abnormality is recognized, and is stored in the storage unit 203 in association with the stabilized video. The control unit 201 causes the reproduction processing unit 210 to start reproduction of the stabilized video and obtains meta information from the storage unit 203. Then, when the display time (or frame) of the stabilized video reaches the time (or frame) indicated by the meta information, the control unit 201 does not change the video provided to the display device 300 to the video switching unit 204 from the stabilized video. Switch to processed video. Note that the meta information may be associated with a raw video.

(B. Pre-processing for switching)
The recording device 200-2A prepares for display in advance before switching the video. Specifically, the control unit 201 makes a state where both the raw video and the stabilized video can be displayed. For example, the control unit 201 causes the playback processing unit 210 to start playback of both the stabilized video and the raw video, and both videos are provided to the video switching unit 204.

(C. Display method after switching)
The recording device 200-2A controls the display time point of the video after switching. Specifically, when the display video is switched, the control unit 201 reproduces the video after switching from a time point before the display time point before switching. For example, when the display video is switched from the stabilized video to the raw video, the control unit 201 provides the video provided to the display device 300 to the video switching unit 204 from the stabilized video from the video buffer 211. Switch to raw video.

Note that, when the display video is switched, the control unit 201 may reproduce the video after switching from the same time as the display time before switching. For example, when the display video is switched from the stabilized video to the raw video, the control unit 201 converts the video provided to the display device 300 to the video switching unit 204 from the stabilized video, from the reproduction processing unit 210. Switch to the raw video provided.

Further, the control unit 201 may select the display time of the video after switching. Specifically, the control unit 201 selects a display video based on setting information from a video delayed in time series from the display time point and a video coinciding with the display time point. For example, the control unit 201 selects a video to be provided to the display device 300 by the video switching unit 204 from the video provided from the video buffer 211 and the video provided from the reproduction processing unit 210 based on the setting information. The setting information may be determined and changed by the user.

<3-3. Device processing>
Next, processing of the recording apparatus 200-2A according to the present embodiment will be described for each video switching method described above.

(A-1. Flow of video switching process based on user operation)
First, a video switching process based on a user operation will be described with reference to FIG. FIG. 19 is a flowchart conceptually showing a video switching process based on a user operation in the recording apparatus 200-2A according to the present embodiment.

The recording device 200-2A outputs a stabilized video (step S501). Specifically, the control unit 201 causes the reproduction processing unit 210 to reproduce at least the stabilized video, and causes the video switching unit 204 to provide the stabilized video to the display device 300. The raw video may be reproduced together with the stabilized video.

Next, the recording apparatus 200-2A determines whether or not there is a video switching operation (step S502). Specifically, the control unit 201 determines whether a video switching operation by the user has been performed on the recording apparatus 200-2A.

When it is determined that the video switching operation has been performed, the recording device 200-2A determines whether or not the display time point is traced back (step S503). Specifically, when a video switching operation is performed, the control unit 201 sets whether to switch the display video to a video whose display time is delayed in time series or a video that matches the display time. Determine based on information.

When it is determined that the display time point is to be traced back, the recording device 200-2A outputs the raw video to be buffered (step S504). Specifically, when it is determined that the display video is switched to a video with the display time point delayed in time series, the control unit 201 provides the video provided from the video buffer 211 to the video switching unit 204 to the display device 300. Let

If it is determined that the display time point is not retroactive, the recording device 200-2A outputs the raw video as it is (step S505). Specifically, when it is determined that the display video is switched to a video having the same display time point, the control unit 201 causes the video switching unit 204 to provide the video provided from the reproduction processing unit 210 to the display device 300.

(A-2. Flow of video switching process based on shake in imaging)
Next, with reference to FIG. 20, a video switching process based on shake in imaging will be described. FIG. 20 is a flowchart conceptually showing a video switching process based on shake in imaging in the recording apparatus 200-2A according to the present embodiment. Note that description of processing that is substantially the same as the processing described above is omitted.

The recording device 200-2A outputs the unprocessed video (step S511). Specifically, the control unit 201 causes the playback processing unit 210 to play the raw video, and causes the video switching unit 204 to provide the raw video to be played to the display device 300. Note that the stabilized video may be reproduced together with the raw video.

Next, the recording apparatus 200-2A determines whether or not the magnitude of shake in imaging is greater than or equal to a threshold value (step S512). Specifically, the control unit 201 calculates a motion vector of the imaging device by analyzing the raw video, and determines whether the calculated motion vector is greater than or equal to a threshold value.

If it is determined that the magnitude of shake in imaging is greater than or equal to the threshold, the recording device 200-2A outputs a stabilized video (step S513). Specifically, when it is determined that the magnitude of the motion vector is a threshold value, the control unit 201 causes the reproduction processing unit 210 to reproduce the stabilized video and the video switching unit 204 to reproduce the stabilized video. The display device 300 is provided. As described above, when the stabilized video is buffered in the video buffer 211, the control unit 201 may switch to the stabilized video provided from the video buffer 211.

Note that if it is determined that the magnitude of shake in imaging is less than the threshold value, the process ends without switching the video.

(A-3. Flow of video switching process based on video update status)
Next, a video switching process based on the video update status will be described with reference to FIG. FIG. 21 is a flowchart conceptually showing a video switching process based on the video update status in the recording apparatus 200-2A according to the present embodiment. Note that description of processing that is substantially the same as the processing described above is omitted.

The recording device 200-2A outputs a stabilized video (step S521).

Next, the recording device 200-2A determines whether or not the display video has not been updated for a predetermined period (step S522). Specifically, the control unit 201 determines whether or not a predetermined period has elapsed since the stabilized video has become only the retained video region.

When it is determined that the display video has not been updated for a predetermined period, the recording device 200-2A determines whether or not the display time point is traced back (step S523).

When it is determined that the display time point is to be retroactive, the recording device 200-2A outputs the buffered raw video (step S524), and when it is determined that the display time point is not retroactive, The processed image is output as it is (step S525).

(A-4. Flow of video switching process based on occurrence of abnormality in video)
Next, a video switching process based on the occurrence of an abnormality in the video will be described with reference to FIG. FIG. 22 is a flowchart conceptually showing a video switching process based on the occurrence of an abnormality in the video in the recording apparatus 200-2A according to the present embodiment. Note that description of processing that is substantially the same as the processing described above is omitted.

The recording device 200-2A outputs a stabilized video (step S531).

Next, the recording apparatus 200-2A determines whether or not an abnormality is detected in the video (step S532). Specifically, the control unit 201 determines whether the display time of the stabilized video has reached the display time indicated by the meta information associated with the stabilized video or the raw video, that is, the time when an abnormality has occurred in the video. To do.

If it is determined that an abnormality in the video has been detected, the recording device 200-2A determines whether or not the display time point is traced back (step S533).

When it is determined that the display time point is to be retroactive, the recording device 200-2A outputs the buffered raw video (step S534), and when it is determined that the display time point is not retroactive, The processed image is output as it is (step S535).

<3-4. Summary of Second Embodiment>
As described above, according to the second embodiment of the present disclosure, the recording apparatus 200-2A determines at least one of the unprocessed image and the processed image as a display image. For this reason, it is possible to provide an image suitable for grasping the situation while maintaining the visibility of the video by displaying a video suitable for display of the raw video and the stabilized video.

In addition, the recording apparatus 200-2A determines a display image based on information related to shake in imaging. For this reason, it is possible to effectively suppress a decrease in the visibility of the video by determining the display video according to the disturbance of the video.

In addition, the information related to shake in imaging includes information related to the degree of shake in imaging. For this reason, by determining the display image according to the degree of the disturbance of the image, it is possible to suppress frequent switching of the image and maintain the visibility of the user.

Also, the degree of shake in imaging is estimated based on at least one of the unprocessed image and the processed image. Therefore, since no additional information is used in the shake estimation process, the configuration for acquiring the additional information is omitted, and the configuration of the recording apparatus 200-2A can be simplified. .

Also, the recording device 200-2A determines the display image based on the display image display update interval. For this reason, it becomes possible to suppress the situation where the amount of information does not increase for the user. In particular, when the configuration of the present embodiment is applied to a monitoring system, monitoring may not be sufficient if the video is not updated. On the other hand, according to this configuration, the reliability of the monitoring system can be improved.

Also, the recording apparatus 200-2A determines the display image based on the comparison between the update interval and the threshold value. For this reason, by determining the lower limit of the update interval in advance, it is possible to prevent the duration of the situation where the amount of information does not increase from being prolonged.

Also, the recording device 200-2A determines the display image based on the size of the first area or the display time. Here, even if the video is updated, if the updated range or time is small, the amount of information desired by the user may not be obtained. On the other hand, according to this configuration, it is possible to reduce the possibility that the amount of information desired by the user will be insufficient.

Also, the recording apparatus 200-2A determines a display image based on information related to the content of at least one of the unprocessed image and the processed image. For this reason, by switching the display video according to the content of the video, it is possible to provide the video according to the user's intention.

Also, the information related to the content of the image includes information related to whether or not a characteristic object is included in the image. For this reason, when the user desires to observe, the display image is switched to an image that can be observed, so that convenience for the user can be improved.

Also, whether or not a characteristic object is included in an image is determined based on the analysis of the image. For this reason, additional information is not required for the characteristic object inclusion determination process, and thus the configuration of the recording apparatus 200-2A can be simplified.

Also, the presence / absence of inclusion of characteristic objects in the image is determined based on the meta information of the image. For this reason, it becomes possible to suppress a processing load compared with the case where image analysis processing is used.

Also, the recording device 200-2A determines a display image based on a user operation. For this reason, it becomes possible to improve a user's convenience by switching a display image according to a user's intention.

In addition, when the display image is switched, the recording device 200-2A displays the image after switching from a time before the display time of the image before switching. For this reason, the user can observe the video from the time before the switching time, and can easily grasp the status of the video at the switching time.

In addition, the recording apparatus 200-2A enables display of both the unprocessed image and the processed image. For this reason, it is possible to suppress the occurrence of the user's waiting time and suppress dissatisfaction by switching the video seamlessly without interruption.

Also, images from the display time point to the time point before the display time point are held in a reproducible state. For this reason, even when a video is displayed from a point in time before the switching point, the video is seamlessly switched without being interrupted due to the retroactive reproduction of the video. Thereby, it becomes possible to improve the ease of grasping the situation at the time of switching while suppressing the occurrence of the waiting time of the user.

<3-5. Modification>
Heretofore, the second embodiment of the present disclosure has been described. In addition, this embodiment is not limited to the above-mentioned example. Below, the modification of this embodiment is demonstrated.

As a modification of the present embodiment, the stabilized image may be generated based on a raw image. Specifically, only the raw image is recorded in the recording device 200-2B, and the recording device 200-2B generates a stabilized image from the recorded raw image. With reference to FIG. 23, the functional configuration of the signal processing system according to the present modification will be described. FIG. 23 is a diagram illustrating an example of a schematic functional configuration of a signal processing system according to a modification of the present embodiment.

As shown in FIG. 23, in addition to the control unit 201, the storage unit 203, the video switching unit 204, the playback processing unit 210, and the video buffer 211, the recording device 200-2B includes a raw video memory 205, a stabilized video generation Unit 206, retained image memory 207, retained image generation unit 208, and motion detection unit 209.

In the recording device 200-2B, since only the raw video is recorded in the storage unit 203, when the control unit 201 instructs the playback processing unit 210 to play back the raw video, the stabilized video generation unit 206 is played back. A stabilized image is generated based on the raw video. Then, the video switching unit 204 displays any one of the raw video to be reproduced, the raw video provided from the video buffer 211, and the stabilized video generated by the stabilized video generating unit 206 as a display video. 300. Note that the stabilized video generation function is substantially the same as the function described in the first embodiment, and a description thereof will be omitted.

Thus, according to the modification of the present embodiment, the stabilized image is generated based on the unprocessed image. Therefore, even when the stabilized video is not stored in advance, a suitable video of the raw video and the stabilized video is provided to the display device 300, so that the user generates the stabilized video. It is not necessary to prepare a separate device, and convenience can be improved.

<4. Hardware Configuration of Signal Processing Device According to One Embodiment of Present Disclosure>
The signal processing apparatus according to each embodiment of the present disclosure has been described above. The processing of the signal processing device described above is realized by cooperation of software and hardware of the signal processing device described below.

FIG. 24 is an explanatory diagram illustrating a hardware configuration of the signal processing devices 100 and 200 (hereinafter also referred to as the signal processing device 100 or the like) according to an embodiment of the present disclosure. As shown in FIG. 24, the signal processing apparatus 100 and the like include a CPU (Central Processing Unit) 132, a ROM (Read Only Memory) 134, a RAM (Random Access Memory) 136, a bridge 138, a bus 140, An interface 142, an input device 144, an output device 146, a storage device 148, a drive 150, a connection port 152, and a communication device 154 are provided.

The CPU 132 functions as an arithmetic processing unit and cooperates with various programs to control the control unit 101, the stabilized video generation unit 107, the retained video generation unit 109, the control unit 201, the video switching unit 204, and the like in the signal processing device 100 and the like. The operations of the stabilized video generation unit 206, the retained video generation unit 208, the motion detection unit 209, and the reproduction processing unit 210 are realized. The CPU 132 may be a microprocessor. The ROM 134 stores programs or calculation parameters used by the CPU 132. The RAM 136 temporarily stores programs used in the execution of the CPU 132 or parameters that change as appropriate during the execution. The raw video memory 106 and the retained video memory 108 and the raw video memory 205, the retained video memory 207 and the video buffer 211 in the signal processing apparatus 100 and the like are realized by the ROM 134 and the RAM 136. The CPU 132, the ROM 134, and the RAM 136 are connected to each other by an internal bus including a CPU bus.

The input device 144 generates an input signal based on an input by the user, such as a mouse, a keyboard, a touch panel, a button, a microphone, a switch, and a lever, and input by the user, and outputs the input signal to the CPU 132. It consists of a control circuit. A user of the signal processing device 100 or the like can input various data or instruct a processing operation to the signal processing device 100 or the like by operating the input device 144.

The output device 146 performs output to a device such as a liquid crystal display (LCD) device, an OLED (Organic Light Emitting Diode) device, or a lamp. Further, the output device 146 may output sound such as a speaker and headphones.

The storage device 148 is a device for storing data as an example of the storage unit 203 such as the signal processing device 100. The storage device 148 may include a storage medium, a recording device that records data on the storage medium, a reading device that reads data from the storage medium, a deletion device that deletes data recorded on the storage medium, and the like. The storage device 148 stores programs executed by the CPU 132 and various data.

The drive 150 is a reader / writer for a storage medium, and is built in or externally attached to the signal processing apparatus 100 or the like. The drive 150 reads information recorded on a mounted removable storage medium such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and outputs the information to the RAM 136. The drive 150 can also write information to a removable storage medium.

The connection port 152 is a bus for connecting to an external information processing apparatus such as the signal processing apparatus 100 or a peripheral device, for example. The connection port 152 may be a USB (Universal Serial Bus).

The communication device 154 is a communication interface composed of a communication device for connecting to a network, for example. Whether the communication device 154 is an infrared communication compatible device, a wireless LAN (Local Area Network) compatible communication device, or an LTE (Long Term Evolution) compatible communication device, wire communication that performs wired communication It may be a device.

<5. Conclusion>
As described above, according to the first embodiment of the present disclosure, even when the display target area does not fit in the imaging range due to shake, the area where no unprocessed video exists is complemented by the retained video, thereby causing the shake. It becomes possible to suppress the deterioration of the visibility of the video, and the burden on the user can be reduced. In addition, since the unprocessed video that is not processed from the video obtained by imaging is used as the display video, the user can appropriately grasp the situation from the video.

In addition, according to the second embodiment of the present disclosure, it is suitable for grasping the situation while maintaining the visibility of the video by displaying the video suitable for display among the raw video and the stabilized video. Images can be provided.

The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, but the technical scope of the present disclosure is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field of the present disclosure can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that it belongs to the technical scope of the present disclosure.

For example, in the above embodiment, only one of the raw video and the stabilized video is provided to the display device 300 and displayed by the display device 300, but the present technology is not limited to this example. For example, both the raw video and the stabilized video may be provided to the display device 300, and the display device 300 may display both provided videos in parallel.

In the above-described embodiment, an example in which the display video is switched during the display of the video has been described. However, each embodiment of the present disclosure may be applied to the determination of the display video at the start of video display.

In the second embodiment, the video buffer 211 has been described as an example of buffering raw video, but the video buffer 211 may buffer stabilized video. In this case, when the display video is switched from the raw video to the stabilized video, the control unit 201 converts the video provided to the display device 300 to the video switching unit 204 from the raw video to the stable provided from the video buffer 211. Switch to the video.

In the second embodiment, an example in which video is played back in time series by being buffered in the video buffer 211 has been described. However, when video is switched, a time point that goes back in time series May be acquired from the storage unit 203 or the like. For example, at the time of video switching, the control unit 201 causes the playback processing unit 210 to play back video of a predetermined number of frames before the frame at the time of video switching from the storage unit 203.

In the above embodiment, an example in which a video (image) is an application target has been described. However, a video with audio may be an application target.

In addition, the effects described in this specification are merely illustrative or illustrative, and are not limited. That is, the technology according to the present disclosure can exhibit other effects that are apparent to those skilled in the art from the description of the present specification in addition to or instead of the above effects.

In addition, the steps shown in the flowcharts of the above-described embodiments are executed in parallel or individually even if they are not necessarily processed in time series, as well as processes performed in time series in the order described. Including processing to be performed. Further, it goes without saying that the order can be appropriately changed even in the steps processed in time series.

The following configurations also belong to the technical scope of the present disclosure.
(1)
An image extraction unit that extracts a portion corresponding to at least a part of a display target region in a holding image obtained by imaging at a time point before imaging of the raw image from a raw image obtained by imaging;
An image generation unit for generating a processed image, wherein the first region is the raw image extracted by the image extraction unit, and the second region different from the first region is the retained image;
A signal processing apparatus comprising:
(2)
The signal processing device according to (1), wherein the processed image includes an image in which at least one of the first region and the second region is determined.
(3)
The signal processing device according to (2), wherein the second region of the processed image has a specific visual aspect.
(4)
The signal processing apparatus according to (3), wherein the specific visual aspect includes a specific coloring.
(5)
The signal processing apparatus according to any one of (1) to (4), wherein an image obtained by the imaging is communicated via a network.
(6)
The signal processing device according to any one of (1) to (5), wherein the raw image includes an image obtained by imaging in a situation where the display target region does not fit in an imaging range.
(7)
An image analysis unit for capturing a specific subject in the raw image;
The image generation unit superimposes an object indicating the estimated position of the specific subject on the processed image when the specific subject captured by the image analysis unit is not included in the first region. The signal processing apparatus according to any one of (1) to (6).
(8)
The image generation unit determines the display target area so that the first area becomes larger when the size of the first area is equal to or smaller than a threshold value. Any one of (1) to (7) 2. The signal processing device according to item 1.
(9)
The signal processing device according to any one of (1) to (8), wherein both the processed image and the unprocessed image are recorded.
(10)
An image providing unit that provides at least one of the processed image and the unprocessed image to an external device;
A control unit that switches an image provided by the image providing unit based on information relating to a state of the apparatus that performs the imaging;
The signal processing device according to any one of (1) to (9), further including:
(11)
The signal processing apparatus according to (10), wherein the control unit switches an image to be provided based on a degree of shake in the imaging.
(12)
The signal processing device according to (11), wherein a degree of shake in the imaging is estimated based on a motion of the device that performs the imaging.
(13)
The signal processing apparatus according to (12), wherein the movement of the apparatus that performs the imaging is estimated based on information obtained from an inertial sensor.
(14)
The signal processing device according to (12), wherein a motion of the device that performs the imaging is estimated based on an image obtained by the imaging.
(15)
The signal processing apparatus according to any one of (10) to (14), wherein the control unit switches an image to be provided based on a possibility of shake occurrence in the imaging.
(16)
The signal processing apparatus according to (15), wherein the possibility of occurrence of shake in the imaging is estimated based on presence or absence of enlargement of a subject in an image obtained by the imaging.
(17)
The signal processing apparatus according to any one of (10) to (16), wherein the control unit switches an image to be provided according to an operation state of the imaging.
(18)
The signal processing apparatus according to (17), wherein the operation state for the imaging includes presence / absence of an operation related to movement of an imaging range.
(19)
The signal processing apparatus according to (17) or (18), wherein the operation state for the imaging includes presence / absence of an operation related to a change in a subject size.
(20)
Extracting from the raw image obtained by imaging a portion corresponding to at least a part of the display target region in the holding image obtained by imaging at a time point before the imaging time of the raw image;
Generating a processed image in which a first region is the extracted raw image, and a second region different from the first region is the retained image;
Including a signal processing method.
(21)
An image extraction unit that extracts a portion corresponding to at least a part of a display target region in a holding image obtained by imaging at a time point before imaging of the raw image from a raw image obtained by imaging;
An image generation unit for generating a processed image, wherein the first region is the raw image extracted by the image extraction unit, and the second region different from the first region is the retained image;
An image providing unit that provides the raw image or the processed image to be generated;
A storage unit for storing an image provided from the imaging device;
An image providing unit that provides the stored image as a display image;
A display device comprising a display unit for displaying a provided display image;
A signal processing system comprising:

The following configurations also belong to the technical scope of the present disclosure.
(1)
Raw images obtained by imaging,
The first area is the raw image corresponding to at least a part of the display target area in the retained image obtained by imaging at a time before the imaging time of the raw image, and is different from the first area An image acquisition unit for obtaining a processed image, wherein the second region is the retained image;
A display control unit for determining at least one of the raw image and the processed image as a display image;
A signal processing apparatus comprising:
(2)
The signal processing apparatus according to (1), wherein the display control unit determines the display image based on information related to shake in the imaging.
(3)
The signal processing apparatus according to (2), wherein the information related to shake in the imaging includes information related to a degree of shake in the imaging.
(4)
The signal processing device according to (3), wherein a degree of shake in the imaging is estimated based on at least one of the unprocessed image and the processed image.
(5)
The signal processing apparatus according to any one of (1) to (4), wherein the display control unit determines the display image based on a display image display update interval.
(6)
The signal processing apparatus according to (5), wherein the display control unit determines the display image based on a comparison between the update interval and a threshold value.
(7)
The signal processing apparatus according to (5) or (6), wherein the display control unit determines the display image based on a size or display time of the first region.
(8)
The display control unit according to any one of (1) to (7), wherein the display control unit determines the display image based on information relating to a content of at least one of the unprocessed image and the processed image. The signal processing apparatus as described.
(9)
The signal processing apparatus according to (8), wherein the information related to the content of the image includes information related to whether or not a characteristic object is included in the image.
(10)
The signal processing apparatus according to (9), wherein whether or not a characteristic object is included in the image is determined based on an analysis of the image.
(11)
The signal processing device according to (9) or (10), wherein whether or not a characteristic object is included in the image is determined based on meta information of the image.
(12)
The signal processing apparatus according to any one of (1) to (11), wherein the display control unit determines the display image based on a user operation.
(13)
The display control unit according to any one of (1) to (12), wherein when the display image is switched, the image after switching is displayed from a time point before the display time point of the image before switching. Signal processing device.
(14)
The signal processing apparatus according to any one of (1) to (13), wherein an image obtained by the imaging is communicated via a network.
(15)
The signal processing apparatus according to any one of (1) to (14), wherein the display control unit is configured to display both the unprocessed image and the processed image.
(16)
The signal processing device according to (15), wherein an image from a display time point to a time point before the display time point is held in the displayable state.
(17)
The signal processing device according to any one of (1) to (16), wherein the processed image is generated based on the unprocessed image.
(18)
Raw images obtained by imaging,
The first area is the raw image corresponding to at least a part of the display target area in the retained image obtained by imaging at a time before the imaging time of the raw image, and is different from the first area Obtaining a processed image, wherein the second region is the retained image;
Determining at least one of the raw image and the processed image as a display image;
Including a signal processing method.
(19)
Raw images obtained by imaging,
The first area is the raw image corresponding to at least a part of the display target area in the retained image obtained by imaging at a time before the imaging time of the raw image, and is different from the first area An image acquisition unit for obtaining a processed image, wherein the second region is the retained image;
A display control unit for determining at least one of the raw image and the processed image as a display image;
An image providing unit that provides the display device with the determined display image;
A display device comprising a display unit for displaying a provided display image;
A signal processing system comprising:

DESCRIPTION OF SYMBOLS 100 Imaging device 101 Control part 102 Lens drive part 103 Optical system 104 Imager 105 Image signal processing part 106 Raw image memory 107 Stabilized image generation part 108 Holding image memory 109 Holding image generation part 110 Gyroscope 111 Communication part 200 Recording apparatus 201 Control unit 202 Communication unit 203 Storage unit 204 Video switching unit 205 Raw video memory 206 Stabilized video generation unit 207 Holding video memory 208 Holding video generation unit 209 Motion detection unit 210 Playback processing unit 211 Video buffer 300 Display device

Claims (18)

1. Raw images obtained by imaging,
   The first area is the raw image corresponding to at least a part of the display target area in the retained image obtained by imaging at a time before the imaging time of the raw image, and is different from the first area An image acquisition unit for obtaining a processed image, wherein the second region is the retained image;
   A display control unit for determining at least one of the raw image and the processed image as a display image;
   A signal processing apparatus comprising:
2. The signal processing apparatus according to claim 1, wherein the display control unit determines the display image based on information related to shake in the imaging.
3. The signal processing apparatus according to claim 2, wherein the information related to shake in the imaging includes information related to a degree of shake in the imaging.
4. The signal processing device according to claim 3, wherein a degree of shake in the imaging is estimated based on at least one of the unprocessed image and the processed image.
5. The signal processing apparatus according to claim 1, wherein the display control unit determines the display image based on a display image display update interval.
6. The signal processing apparatus according to claim 5, wherein the display control unit determines the display image based on a comparison between the update interval and a threshold value.
7. The signal processing apparatus according to claim 5, wherein the display control unit determines the display image based on a size or a display time of the first region.
8. The signal processing apparatus according to claim 1, wherein the display control unit determines the display image based on information relating to a content of at least one of the unprocessed image and the processed image.
9. The signal processing device according to claim 8, wherein the information related to the content of the image includes information related to whether or not a characteristic object is included in the image.
10. 10. The signal processing device according to claim 9, wherein whether or not a characteristic object is included in the image is determined based on an analysis of the image.
11. 10. The signal processing device according to claim 9, wherein whether or not a characteristic object is included in the image is determined based on meta information of the image.
12. The signal processing apparatus according to claim 1, wherein the display control unit determines the display image based on a user operation.
13. 2. The signal processing apparatus according to claim 1, wherein when the display image is switched, the display control unit displays the image after switching from a time point before the display time point of the image before switching.
14. The signal processing apparatus according to claim 1, wherein an image obtained by the imaging is communicated via a network.
15. The signal processing apparatus according to claim 1, wherein the display control unit is configured to display both the unprocessed image and the processed image.
16. The signal processing apparatus according to claim 15, wherein an image from a display time point to a time point before the display time point is held in the displayable state.
17. The signal processing apparatus according to claim 1, wherein the processed image is generated based on the unprocessed image.
18. Raw images obtained by imaging,
    The first area is the raw image corresponding to at least a part of the display target area in the retained image obtained by imaging at a time before the imaging time of the raw image, and is different from the first area Obtaining a processed image, wherein the second region is the retained image;
    Determining at least one of the raw image and the processed image as a display image;
    Including a signal processing method.

Images