JP2011087257A - Semiconductor integrated circuit and imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a semiconductor integrated circuit for recording a vibrant video image by adaptively switching scenes as a sub video image separately from a main video image being captured by an amateur photographer. <P>SOLUTION: A semiconductor integrated circuit includes: a main image storage part 15 for storing a video image captured by an imaging element 3 in a memory 7 as a main video image; a detection part 12 for repeatedly detecting a region of interest (ROI) that is a region to pay attention thereto in the main video image stored in the memory 7; a sub image storage part 16 for storing a video image in the ROI in the video image captured by the imaging element 3 in the memory 7 as a sub video image; an encoder 8 for compressing the main video image and the sub video image stored in the memory 7; and a recording part 9 for recording the main video image and the sub video image compressed by the encoder 8 onto a recording medium. The detection part 12 determines whether the same subject is included in a current ROI for a predetermined period of time and also includes a change part for detecting the ROI again if determined included. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a semiconductor integrated circuit connected to an imaging device and a memory, and an imaging device.

  A digital still camera, digital video camera, etc. that forms a subject on an image sensor with a lens, performs various signal processing on the image data obtained by photoelectric conversion, displays the subject video on a liquid crystal monitor, etc., and records it on a recording medium Such an image pickup apparatus has been put into practical use and widely used. The photographer controls the direction of the camera and the zoom function to determine the composition of the subject and take a picture, such as shooting a scene where a child is playing.

  When an amateur photographer shoots video, generally he / she is not trained to shoot, and often shoots with wrong judgments and operations, so there are cases where the video is not good to enjoy. Many of them are just shot videos, which can be tedious and tedious. In addition, in order to make the video recorded in this way attractive, it is necessary to perform an editing operation after shooting. However, since the editing operation requires technology and time, it is often not performed.

  As an applied technology, a technique has been proposed that monitors the photographer's camera work and assists the photographer to be alerted and shoot a good video if there is a bias in the shooting state, such as when the zoom-up scene continues. (For example, patent document 1). In addition, this applied technology generates metadata based on changes in shooting conditions and records a predetermined number of metadata as incidental information so that only important scenes can be selected and played back during playback. Is also disclosed.

JP 2008-227859 A

  However, in the above conventional imaging apparatus, even if the user is prompted to call attention to assist good camera work, the photographer does not always change the camera work, and a scene with high importance is not selected from the photographed scenes. Since a function for selecting and playing back is provided, the objective of shooting a good image cannot be achieved unless the camera work is actually changed and shot.

  Good video here refers to video that is attractive to watch and does not get tired, and is a short video that lasts from 5 seconds to several tens of seconds for one scene (camera work). Define. Most videos taken by professional photographers, such as television and movies, fall under this rule.

  SUMMARY OF THE INVENTION The present invention solves the above-described conventional problems, and separates a main video image taken by an amateur photographer, adaptively switches scenes as a sub video image, and records a lively video image. The purpose is to provide. It is also included in the object of the present invention to provide an imaging device including such a semiconductor integrated circuit.

  In order to achieve the above object, a semiconductor integrated circuit according to the present invention is a semiconductor integrated circuit connected to an image sensor and a memory, and stores a video captured by the image sensor as a main video in the memory. A storage unit; a detection unit that repeatedly detects a gaze region, which is a region to be watched, of the main video stored in the memory; and a video of the gaze region of the video imaged by the image sensor. A sub-video storage unit that stores video in the memory, an encoder unit that compresses the main video and the sub-video stored in the memory by the main video storage unit and the sub-video storage unit, and compression by the encoder unit A recording unit that records the main video and the sub-video recorded on a recording medium, and the detection unit determines whether or not the same subject is included in the current gaze area for a predetermined period. It determines, when it is determined to be included, to detect the gaze region again.

  With this configuration, even when the photographer shoots with monotonous camera work, it is possible to record a video with changes in the sub-video.

  In addition, the detection unit is configured to detect any one of a person detection unit that detects at least one person region that is a region including a person, and at least one person region that is detected, from an image captured by the image sensor. You may provide the area | region detection part which detects the area | region to include as the said gaze area | region.

  With such a configuration, it is possible to record a sub-video with a composition in which a person who is captured when a person is photographed is watched.

  In addition, the area detection unit may preferentially detect the area including the person area close to the center of the main video among the detected at least one person area as the gaze area.

  With such a configuration, when the composition is determined so that the person who the photographer wants to take is centered on the screen, the sub-picture can be recorded with the composition in which the person who wants to take the picture is watched.

  The region detection unit may preferentially detect a region including a human region having a larger area as the gaze region among the detected at least one human region.

  With such a configuration, when the composition is determined so that the person who the photographer wants to shoot is arranged in front of the screen, the sub-image can be recorded with the composition where the person who wants to shoot is watched.

  The detection unit is further held in the holding unit holding a person information indicating a person, a person included in each of at least one person area detected by the person detection unit, and the holding unit. A collation unit that collates the personal information, and the region detection unit may detect the gaze region based on a collation result by the collation unit.

  Further, the holding unit further holds feature data indicating the characteristics of the person corresponding to the person information, and the collating unit is included in each of at least one person area detected by the person detecting unit. A matching score indicating a similarity between the feature of the person and the feature data is calculated by comparing the feature of the person to be recorded and the feature data held in the holding unit, A region including a person region of a person with a high calculated collation score may be preferentially detected as the gaze region.

  With this configuration, when a person to be photographed is determined in advance, a sub-image can be recorded with a composition in which the person who wants to photograph is watched.

  Further, the holding unit further holds the number of times the person has been detected so far, corresponding to each person information included in the person area detected by the person detecting unit until now, The area detection unit may preferentially detect the area including the person area of the person who is held by the holding unit as the gaze area.

  With this configuration, when a person to be photographed is not determined in advance, a sub-image can be recorded with a composition in which a person who is frequently photographed is watched as a person who wants to photograph. Therefore, the photographer does not need to register a person in advance and can avoid annoyance.

  The detection unit further includes a person direction detection unit that detects a direction of a person included in each of at least one person region detected by the person detection unit, and the region detection unit includes the person direction detection. The gaze area may be detected using the orientation detected by the unit so that the area ahead of the person is larger than the area behind the person.

  With such a configuration, it is possible to gaze at a person to be photographed and determine the composition in a form that also includes a subject in the direction the person is looking at and record a sub-video. In other words, since a video including a region in the line-of-sight direction of a person can be taken as a sub-video, the video can be easily seen and understood.

  The predetermined time is preferably 5 seconds or more and 30 seconds or less.

  With this configuration, it is possible to create an attractive video that does not get tired of watching the sub-video.

  An imaging apparatus according to the present invention includes the semiconductor integrated circuit, an imaging element, and a display device, and stores the main video and the sub video in the memory.

  According to the present invention, even if the composition determined by the photographer is monotonous and tedious camera work, it is possible to record a dynamic image with a variety of changes as a sub-image. Further, since the main image is recorded as the photographer intends, even if there is an erroneous determination in the sub-image area switching, a failure that the scene that should be originally recorded is not recorded does not occur.

It is a block diagram which shows the system configuration | structure of the imaging device of this invention. It is a schematic diagram for demonstrating main image reading. It is a schematic diagram for demonstrating subimage reading. It is the figure which paid its attention to the buffer structure of memory about the data flow until an image is stored in a recording medium. It is the figure which showed the image capture operation | movement of the digital processing part in time series. It is a schematic diagram for demonstrating the encoding operation | movement of a main image and a subimage. It is a block diagram which shows the detailed structure of a detection part. It is a figure for demonstrating operation | movement of a detection part. It is a figure which shows an example of the data structure of a person database. It is a figure which shows an example of the data structure of a person result and a collation result It is a flowchart which shows operation | movement of an area | region determination part.

  An image pickup apparatus according to the present invention is a semiconductor integrated circuit used by being connected to an image pickup device and a memory, wherein a main image storage unit that stores an image picked up by the image pickup device as a main image in the memory, and the memory A detection unit that detects a gaze area that is an area to be watched out of the main video stored in the image, and a video of the gaze area out of the video captured by the imaging device is stored in the memory as a sub-video. A sub-video storage unit; an encoder unit that compresses the main video and the sub-video stored in the memory by the main video storage unit and the sub-video storage unit; the main video compressed by the encoder unit; A recording unit that records a sub-video on a recording medium, and the detection unit further includes a determination unit that determines whether the same subject is included in the gaze area for a predetermined period, and the determination unit If it is determined that the contains the same subject by the parts, and a changing unit for changing the gaze region.

  Hereinafter, an imaging device according to an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram illustrating a system configuration of the imaging apparatus according to the first embodiment.

  The imaging apparatus 100 includes a lens 1 that is a zoomable optical element, a lens driving unit 2 that drives the lens 1 to perform a zoom operation, a zoom switch 11 that specifies the zoom direction / zoom amount of the lens 1, and a lens. An imaging element 3 that photoelectrically converts subject light that has passed through 1 and outputs an image signal; an imaging element drive unit 4 that generates a drive signal for extracting an arbitrary region from the imaging element 3 as an image; An analog processing unit 5 that performs various processes such as noise reduction, gain control, and A / D conversion on the output analog image signal, and Y / C conversion processing and electronic zoom on the digitized image signal A digital processing unit 6 that performs processing, a memory 7 that temporarily stores an image processed by the digital processing unit 6, and an image stored in the memory 7 in a predetermined format An encoder 8 for image compression, a recording unit 9 for storing data compressed by the encoder 8 in a recording medium (recording medium) such as an SD card or an HDD, and an image output from the digital processing unit 6 can be confirmed. A display unit 10 such as an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) to be displayed on the display unit, and a detection unit that detects a gaze region that is an image region to be watched on the image data processed by the digital processing unit 12, a control unit 13 that performs overall control, and a mode switch 14 that receives a mode switching instruction from the photographer.

  In the present embodiment, the imaging apparatus 100 includes the memory 7. However, the present invention is not limited to an imaging apparatus with a built-in memory, and may be an imaging apparatus that is used connected to an external memory. Good. Further, the digital processing unit 6, the encoder 8, the recording unit 9, the detection unit 12, and the control unit 13 that are processing units surrounded by a dotted line may be realized as SoC (System on Chip).

  The digital processing unit 6 includes a main image storage unit 15 and a sub image storage unit 16.

  The main image storage unit 15 is a main video storage unit of the present invention that stores a video captured by the image sensor 3 as a main video in the memory 7. Specifically, the main image storage unit 15 reads out an image signal corresponding to the image of the main image region that is the entire image captured by the image sensor 3 from the analog processing unit 5, and a plurality of main images that are temporally continuous. The main video image is stored in the memory 7.

  The sub-image storage unit 16 is a sub-image storage unit of the present invention that stores, in the memory 7, the video of the gaze area among the video captured by the image sensor 3 as a sub-video. Specifically, the sub-image storage unit 16 reads out an image signal corresponding to the image of the gaze area detected by the detection unit 12 from the entire image formed from the analog processing unit 5 and continuously in time. Are stored in the memory 7 as sub-pictures composed of a plurality of sub-images.

  Next, each component will be described more specifically.

  2 and 3 are diagrams illustrating an operation in which the digital processing unit 6 reads an image from the image sensor 3. To simplify the description, for example, the number of pixels of the image sensor 3 is 256 pixels of 16 pixels × 16 pixels, and the number of pixels of the moving image stored in the memory 7 is 64 pixels of 8 pixels × 8 pixels.

  As shown in FIG. 2, the main image storage unit 15 included in the digital processing unit 6 stores the entire image (image of the main image region) 21 captured by the image sensor 3 as the main image 22 in the memory 7. At this time, the main image storage unit 15 creates the main image 22 by reducing the length and width of the image 21 in the main image area by a factor of 1/2. That is, the number of pixels of the main image 22 is halved in both the vertical and horizontal directions compared to the number of pixels of the image 21 in the main image area.

  As shown in FIG. 3, the sub-image storage unit 16 included in the digital processing unit 6 is an image of an image region to be watched detected by the detection unit 12 among the entire images 31 captured by the image sensor 3. (Gaze area image) 32 is stored in the memory 7 as the sub-image 33. At this time, the sub-image storage unit 16 creates the sub-image 33 at the same magnification without changing the vertical and horizontal magnifications. As described above, the image sensor 3 has a larger number of pixels than the number of pixels of the image that the memory 7 records as the main image and the sub image.

  The main image storage unit 15 may convert the number of pixels by thinning out the reading from the image sensor 3 when creating the main image 22, or may read all the pixels from the image sensor 3. In this case, the number of pixels subjected to digital filter processing may be converted.

  In addition, although the sub-image storage unit 16 has shown the example in which the gaze area image 32 and the sub-image 33 have the same number of pixels when the sub-image 33 is created, the number of pixels may be different or the number of pixels If they are different, pixel readout thinning or pixel number conversion by digital filter processing may be performed.

  Next, a flow until an image formed on the image sensor 3 is recorded on a recording medium by the recording unit 9 will be described.

  FIG. 4 is a diagram showing the data flow until the image formed on the image sensor 3 is stored in the recording medium, focusing on the buffer configuration of the memory 7.

  As shown in the figure, the digital processing unit 6 includes a YC conversion processing unit 6A and a digital zoom unit 6B as hardware configurations. The YC conversion processing unit 6A converts the RGB image signal output from the analog processing unit 5 into Y (luminance) and C (color difference) image signals. The digital zoom unit 6B enlarges / reduces the image converted by the YC conversion processing unit 6A.

  In the example described with reference to FIGS. 2 and 3, the entire image 21 is reduced to ½ times by the digital zoom unit 6 </ b> B and stored in the memory 7 as the main image 22. On the other hand, the image 32 in the gaze area is cut out at the same magnification without being enlarged / reduced by the digital zoom unit 6B and stored in the memory 7 as the sub image 33. That is, the main image storage unit 15 has a hardware configuration of the YC conversion processing unit 6A and the digital zoom unit 6B, and the sub-image storage unit 16 has a hardware configuration of the YC conversion processing unit 6A. The sub-image storage unit 16 may have the hardware configuration of the digital zoom unit 6B, and may generate a sub-image by performing enlargement / reduction processing of the image 32 in the gaze area.

  Next, the encoder 8 performs compression processing using the main image 22 and the sub-image 33 stored in the memory 7 as input images. At this time, the reference image and the encoded moving image necessary for the encoding process are stored in the encoding buffer 7C, and the recording unit 9 acquires the encoded moving image from the encoding buffer 7C and writes it to the recording medium.

  As described above, in the imaging apparatus 100 according to the present embodiment, since the imaging device 3 has a higher number of pixels than the number of moving image pixels, the sub-image does not require digital zoom processing, and the main image is also high. Since reduction processing is performed from a fine image, it is possible to suppress image quality deterioration. Further, since the image converted into a desired size is stored in the memory 7, it is possible to reduce the number of memory accesses and the memory capacity to be used.

  FIG. 5 is a diagram showing the image capturing operation of the digital processing unit 6 in time series. The imaging element 3 is an element capable of imaging at a speed higher than the moving image frame period, and is an imaging element capable of imaging 120 frames per second, for example. Here, the whole images 51a to 51h represent images picked up by the image pickup device 3 at a 1/120 second period. The digital processing unit 6 stores the image in the memory 7 while switching between the main image and the sub image every frame period of the image sensor 3.

  That is, the main image storage unit 15 converts the number of pixels using the whole images 51b, 51d, 51f, and 51h as images of the main image region, and stores them in the memory 7 as the main images 53b, 53d, 53f, and 53h. Further, the sub-image storage unit 16 stores the gaze area images of the whole images 51a, 51c, 51e, and 51g having different times from the whole images 51b, 51d, 51f, and 51h as the sub-images 52a, 52c, 52e, and 52g. To store.

  As a result, the main images 53b, 53d, 53f, and 53h have a 1/60 second cycle, and the sub images 52a, 52c, 52e, and 52g also have a 1/60 second cycle. That is, the digital processing unit 6 generates two different series of images from one series of whole images and stores them in the memory 7.

  FIG. 6 is a schematic diagram for explaining the operation of encoding the main image and the sub-image stored in the memory 7 and recording them on an external recording medium.

  The encoder 8 compresses the main image 61 to generate a main image moving image 63 and also compresses the sub image 62 to generate a sub image moving image 64. The recording unit 9 stores the generated two different moving images in the recording medium 65 as two files. The encoder 8 compresses the main image 61 and the sub-image 62 to generate a multi-angle moving image 66. The recording unit 9 stores the generated single multi-angle video 66 in the recording medium 65 as one file.

  Although an example in which two different images are encoded is described here, the present invention is not limited to two. Further, in the case of recording as a plurality of moving images, it may be possible to give a feature to the attribute of the file name or file management information in order to indicate that the moving images are related.

  In addition, the imaging apparatus determines the readout frame period in accordance with the imaging capability (maximum frame period, number of pixels, etc.) of the imaging element 3 for the input of the main image and the input of the sub-image, and the encoder 8 determines the determined frame. You may make it compress an image by a period.

  Next, a method for detecting a gaze area by the detection unit 12 will be described.

  FIG. 7 is a block diagram illustrating a detailed configuration of the detection unit 12. In the figure, the digital processing unit 6, the control unit 13, and the mode changeover switch 14 are also shown.

  The detection unit 12 repeatedly detects a gaze area that is a gaze area in the main image. The detection unit 12 includes a frame memory 71, a person detection unit 72, a person orientation detection unit 73, a person verification unit 74, a person registration unit 75, a person storage unit 76, an area determination unit 77, an elapsed time timer 78, and an invalid time timer 79. Is provided. Specifically, it is determined whether or not the same subject is included in the current gaze area for a predetermined period. If it is determined that the same gaze area is included, the gaze area is detected again. The person storage unit 76 holds a person database 81, and the area determination unit 77 holds a person detection / collation result 82 and a priority state 83.

  The frame memory 71 stores the main image by storing an image signal corresponding to the image of the main image area output from the main image storage unit 15 of the digital processing unit 6. The frame memory 71 is a partial area of the memory 7, and the detection unit 12 captures a main image from the partial area of the memory 7 via the digital processing unit 6. The frame memory 71 may be a physically different memory from the memory 7.

  The person detection unit 72 detects at least one person area that is an area including a person in the main image. Specifically, the person detection unit 72 holds a person's characteristic pattern in advance, and determines whether or not a person is included in the main image stored in the frame memory 71 using pattern matching. When a person is included, an area including the person is detected as a person area. Further, the person detection unit 72 passes area information (area ID, position, size) indicating the detected person area to the area determination unit 77. Here, the area ID is an ID corresponding to each detected person area on a one-to-one basis, the position is a coordinate indicating the reference point of the person area in the main image, and the size is the person area in the main image. It is a size.

  The person orientation detection unit 73 detects the orientation of a person included in each of at least one person region detected by the person detection unit 72. Specifically, the person orientation detection unit 73 uses the information on the person area detected by the person detection unit 72 to detect which direction the person is facing, and the person orientation information (area ID, Direction and angle) to the region determination unit 77. For example, the image of the person area detected by the person detection unit 72 is extracted from the main image stored in the frame memory 71, the orientation of the person in the person area is specified from the extracted image of the person area, and the person orientation information Is detected.

  The person collation unit 74 is a collation unit of the present invention, and collates the person included in each of at least one person area detected by the person detection unit 72 with the person information held in the person storage unit 76. To do. Specifically, the person collation unit 74 collates the characteristics of the person included in each of the at least one person area detected by the person detection unit 72 with the characteristic data held in the person storage unit 76. Thus, a collation score indicating the similarity between the feature of the person and the feature data is calculated. More specifically, the person collation unit 74 extracts the feature amount of the person using the information on the person area detected by the person detection unit 72 and the person direction information detected by the person direction detection unit 73. Furthermore, the person collation unit 74 collates the extracted feature quantity with the feature quantity of the specific person registered in the person database 81 held in the person storage unit 76, and the similarity that is the probability that the person is a specific person. Information (region ID, collation score) is passed to the region determination unit 77. For example, the person verification unit 74 extracts an image of the person area detected by the person detection unit 72 from the main image stored in the frame memory 71, and features of the person included in the person area from the extracted image of the person area. Extract the amount. Then, by comparing the extracted feature quantity with the feature data of the specific person registered in the person database 81, a matching score indicating the similarity between the feature of the person included in the person region and the feature data Is calculated.

  When the person collated by the person collation unit 74 is not registered in the person database 81, the person registration unit 75 registers new person feature data in the person database 81.

  The person storage unit 76 is a holding unit of the present invention that holds person information indicating a person. Specifically, the person storage unit 76 includes a person database 81 that stores person information and person data indicating the characteristics of the person corresponding to the person information. The data structure of the person database 81 will be described later.

  The area determination unit 77 is an area detection unit of the present invention, and detects an area including any one of at least one person area detected by the person detection unit 72 as a gaze area. Specifically, the area determination unit 77 includes the area information, the person direction information, and the similarity information passed from the person detection unit 72, the person orientation detection unit 73, and the person verification unit 74, and the mode instructed by the photographer. The gaze area is determined according to the above. The area determination unit 77 has a person detection / collation result 82 and a priority state 83.

  The person detection / collation result 82 is a database that stores information indicating the person area and information indicating the person in the person area corresponding to each person area detected by the person detection unit 72. The data structure of the person detection / collation result 82 will be described later.

  The priority state 83 indicates a mode that is a determination criterion for determining an area by the area determining unit 77. Specifically, the priority state 83 indicates one of (A) position priority mode, (B) size priority mode, and (C) collation priority mode. The priority state 83 is switched to any mode by the photographer operating the mode switch 14.

  (A) The position priority mode is a mode in which it is assumed that the composition is determined so that the person who the photographer wants to shoot is reflected in the center. When the priority state 83 is the position priority mode, the region determination unit 77 preferentially focuses on a region including a person region closer to the center of the main image among at least one person region detected by the person detection unit 72. Detect as.

  (B) The size priority mode is a mode in which it is assumed that the photographer decides the composition so that the subject that he / she wants to photograph is displayed closer to the front. When the priority state 83 is the size priority mode, the region determination unit 77 preferentially detects a region including a human region having a larger area as a gaze region among at least one human region detected by the person detection unit 72. To do.

  (C) The collation priority mode is a mode in which it is assumed that the person the photographer wants to take is determined in advance, such as his / her family or acquaintance. The area determination unit 77 detects the gaze area based on the collation result by the person collation unit 74 when the priority state 83 is the collation priority mode. Specifically, an area including a person area of a person with a high matching score calculated by the person matching unit 74 is preferentially detected as a gaze area.

  The elapsed time timer 78 is set with a predetermined elapsed time (for example, 10 seconds) by the area determination unit 77. The elapsed time timer 78 generates an event when the set elapsed time has passed. Specifically, the elapsed time timer 78 prevents the region determination unit 77 from determining the same region for a predetermined elapsed time or more as the gaze region.

  The invalid time timer 79 is set with a predetermined invalid time (for example, 20 seconds) by the area determination unit 77. The invalid time timer 79 generates an event when the set invalid time has passed. Specifically, the invalid time timer 79 is activated when the gaze area is the same area for a predetermined elapsed time or longer, thereby forcing the area determination unit 77 to change the gaze area.

  The detection unit 12 configured as described above can adaptively switch the gaze area according to the mode instructed by the photographer.

  Next, the operation of the detection unit 12 configured as described above will be described with reference to FIGS. 8, 9A, and 9B. Specifically, FIG. 8 is a diagram for explaining the operation of the detection unit 12, FIG. 9A is a diagram illustrating an example of the data structure of the person database 81, and FIG. 9B is data of the person result / collation result 82 It is a figure which shows an example of a structure. Although the description here uses the position and orientation of the face as information for specifying a person, it is not limited to the region determination method using face information, and the region is determined using another method. It doesn't matter.

  The main image 91 shown in FIG. 8 is an image of 16 pixels × 16 pixels, and represents the position with the upper left pixel as the origin coordinate (0, 0). For example, the area information of the area ID 1 is represented as a position (11, 4) and a size 2.

  As shown in FIG. 9A, the person database 81 stores name and feature amount data corresponding to each registered person registration ID. In the person database 81, for example, feature amount data is registered by a photographer.

  The person detection / collation result 82 corresponds to each area ID detected by the person detection unit 72, the position and size of the person area corresponding to the area ID, the orientation of the person included in the person area, A collation score of a person included in the person area and a validity flag indicating whether or not the person area is valid are stored. Specifically, the person detection / collation result 82 is calculated by the person collation unit 74 with the area ID, position, and size detected by the person detection unit 72, the direction detected by the person direction detection unit 73, and the like. A matching score and an effective flag indicating whether or not the person area can be selected as a gaze area are stored.

  First, the person detection unit 72 detects person areas corresponding to the three persons from the main image 91, and passes the detection results to the area determination unit 77. The detection results are, for example, a position (11, 4) as a region ID1, a region of size 2, a position (8, 7) as a region ID2, a region of size 2.5, and a position (2, 9) as a region ID3. , A region of size 3. The area determination unit 77 that has received the detection result from the person detection unit 72 stores the detection result in the person detection / collation result 82.

  Next, the person direction detection unit 73 detects the direction of the person for each of the three person regions detected by the person detection unit 72 and passes the result to the region determination unit 77. The detection results are, for example, 45 ° right as region ID1, 90 ° right as region ID2, and front as region ID3. The area determination unit 77 that has received the detection result from the person orientation detection unit 73 stores the detection result in the person detection / collation result 82.

  Next, the person collation unit 74 collates each of the three person regions with the feature amount data registered in the person database 81 and passes the collation result to the region determination unit 77. As for the collation results, for example, the area ID1 is 90 points as the collation score with the person registration ID1, the area ID2 is 80 points as the collation score with the person registration ID2, and no area ID3 can be collated with the area ID3. The area determination unit 77 that has received the collation result from the person collation unit 74 stores the collation result in the person detection / collation result 82.

  Next, the area determination unit 77 determines a gaze area based on the person detection / collation result 82 and the priority state 83.

  FIG. 10 is a flowchart showing the operation of the area determination unit 77.

  The area determination unit 77 determines whether the priority state 83 is (A) position priority mode, (B) size priority mode, or (C) collation priority mode (step S01).

  When the priority state 83 is (A) position priority mode (position priority mode in step S01), the area determination unit 77 selects an area ID for which the validity flag is valid among the area IDs stored in the person detection / collation result 82. An area ID corresponding to the position closest to the center of the main image is selected from the inside (step S02). For example, in the example of FIG. 8, the area ID of the person area closest to the position (8, 8) is selected. Therefore, the area ID2 is selected in FIGS. 8 and 9B.

  Further, when the priority state 83 is (B) size priority mode (size priority mode in step S01), the area determination unit 77 sets the validity flag among the area IDs stored in the person detection / collation result 82 to be valid. The largest region ID is selected from the region IDs (step S03). For example, the area ID 3 is selected in FIGS. 8 and 9B.

  When the priority state 83 is (C) collation priority mode (collation priority mode in step S01), the region determination unit 77 is a region where the validity flag is valid among the region IDs stored in the person detection / collation result 82. A region ID having the highest matching score is selected from the IDs (step S04). For example, in FIG. 8 and FIG. 9B, the region ID1 is selected.

  Regardless of the mode in which the priority state 83 is set, if there is no area ID for which the validity flag is valid among the area IDs stored in the person detection / collation result 82, the area ID is not selected. In addition, regardless of the mode in which the priority state 83 is set, the area ID is not selected even if there is no person area detected by the person detection unit 72.

  Next, the area determination unit 77 determines whether there is a selected area ID after selecting an area ID corresponding to each mode ((A) position priority mode, (B) size priority mode, and (C) collation priority mode). It is determined whether or not (step S05).

  If there is a selected area ID (YES in step S05), an area 92 including the person area of the selected area ID is set as a gaze area (step S06). For example, in the case of the position priority mode in FIGS. 8 and 9B, the area determination unit 77 sets the area 92 as the gaze area. Here, the area determination unit 77 determines the gaze area so that the person area of the selected area ID becomes the center of the gaze area.

  Since the orientation of the person included in the person area of area ID 2 is 90 ° to the right, rather than determining the gaze area around the person area, there is some other subject in the direction that the person is facing. Therefore, the gaze area may be determined around the area shifted in the direction of the person's direction. For example, in FIG. 8, a region 93 centered on a region where the region ID 2 is shifted may be set as the gaze region. By doing so, the imaging apparatus 100 according to the present embodiment can capture a person's line-of-sight direction as a sub-image, so that it is easy to see and understand.

  By the way, the area determination unit 77 monitors the elapsed time when the same area ID is selected by the elapsed time timer 78 so as not to be monotonous camera work.

  Specifically, after determining the gaze area (step S06), the area determination unit compares the area ID selected last time with the area ID selected this time, and determines whether or not there is a change in the area ID ( Step S07).

  If there is a change in the area ID (YES in step S07), the area determination unit 77 sets an elapsed time in which selection of the same area ID is allowed in the elapsed time timer 78, and starts the elapsed time timer (step S1). S08). In other words, the elapsed time timer 78 is reset to a predetermined time. This predetermined time is, for example, 5 seconds or more and 30 seconds or less. As a result, it is possible to make the video attractive without watching the sub-video.

  On the other hand, if there is no change in the area ID (NO in step S07), or after resetting the elapsed time timer 78, the area determination unit 77 determines whether or not an event occurs by the elapsed time timer 78 ( Step S09).

  When an event occurs by the elapsed time timer 78 (YES in step S09), that is, when the same area ID is continuously selected for a certain period, the area determination unit 77 selects the area ID selected for a certain period for the subsequent certain period. In order to prevent this from happening, the valid flag of the corresponding area ID of the person detection / collation result 82 is cleared. Further, the area determination unit 77 sets the invalid time of the corresponding area ID in the invalid time timer 79 and starts the invalid time timer 79 (step S10). Then, it returns to step S01 and repeats the said process.

  When the photographer changes the camera work or the subject moves, the area ID selected with the priority according to the mode changes and before the event of the elapsed time set in the elapsed time timer 78 occurs. In addition, the elapsed time timer 78 is reset. However, in the case of a scene with a small amount of movement due to monotonous camera work, since the same area ID is continuously selected for a certain period and the elapsed time timer 78 is not reset, an event of the elapsed time set in the elapsed time timer 78 occurs. The validity flag of the corresponding area ID of the person detection / collation result 82 is cleared, and the corresponding area ID cannot be selected for a certain period. When the region determination unit 77 selects the next region ID during the period when the valid flag is cleared (step S02, step S03, or step S04), the remaining region IDs excluding the region ID for which the valid flag is cleared are used. A region ID having a high priority is selected. Thereby, since different person areas are selected at regular intervals, the gaze area is also different at regular intervals. Therefore, the imaging apparatus 100 can change the sub image.

  Here, when a certain period of time has passed with only one person detected in the main image, or when there is no region ID to be selected due to a change in camera work or movement of the subject (NO in step S05), The region of the main image 91 is set as the gaze region, and the elapsed time timer 78 is stopped (step S11). By doing so, it is possible to change the sub-image even when a plurality of persons are not detected.

  The region IDs selected in succession must be returned to the selection target after a certain period. Specifically, when it is determined in step S09 that the event by the elapsed time timer 78 has not occurred (NO in step S09), or after the elapsed time timer 78 is stopped in step S11, the region determination unit 77 The invalid time timer 79 determines whether an event has occurred (step S12).

  When an event is generated by the invalid time timer 79 set in step S10 (YES in step S12), the validity flag of the corresponding area ID of the person detection / collation result 82 is validated to return the corresponding area ID to the selection target. Then, the invalid time timer 79 is stopped (step S13). Then, it returns to step S01 and repeats the said process.

  On the other hand, when the event by the invalid time timer 79 does not occur (NO in step S12), the process returns to step S01 and the above process is repeated.

  As described above, the detection unit 12 adaptively switches the gaze area according to the mode instructed by the photographer. Therefore, the imaging apparatus 100 according to the present embodiment can record a variety of video as a sub-image even when the photographer performs photographing with monotonous camera work.

  As described above, the imaging apparatus 100 according to the present embodiment includes the main image storage unit 15 that stores the image captured by the image sensor 3 as the main image in the memory 7, and the main image stored in the memory 7. A detection unit 12 that repeatedly detects a gaze region that is a region to be watched, a sub-image storage unit 16 that stores a video of the gaze region in the memory 7 as a sub-video among images captured by the image sensor 3, An encoder 8 that compresses the main image and the sub-image stored in the memory 7 by the image storage unit 15 and the sub-image storage unit 16, and a recording unit 9 that records the main image and the sub-image compressed by the encoder 8 on a recording medium; The detection unit 12 determines whether or not the same subject is included in the current gaze area for a predetermined period. When it is determined that the current gaze area is included, the detection unit 12 detects the gaze area again.

  Thereby, the imaging apparatus 100 according to the present embodiment can record a moving image by adaptively switching scenes as sub-images separately from the main image taken by the amateur photographer.

  The physical location of the frame memory 71 is not mentioned and may exist on the memory 7. In addition, when the same area ID is continuously selected for a certain period, the ID is temporarily invalidated and the next candidate is selected. At that time, the area ID may be switched to a different area ID, or the previously selected area ID and The integrated area including both the area IDs selected this time may be set as the gaze area. Further, when switching the gaze area from the area A including one person area to the area B including one other person area, the gaze area may be instantaneously switched from the area A to the area B. The area may be continuously moved so that virtual pan / tilt / zoom is performed from A to the area B.

  Further, the elapsed time value set in the elapsed time timer 78 may be the same every time, and when a low priority area ID is selected, the elapsed time to be set is shortened, and the time is changed in conjunction with the priority. May be. The same applies when invalidating the area ID. The elapsed time timer 78 that measures the elapsed time and the invalid time timer 79 that measures the invalid time may physically exist separately, or may be physically configured so that two systems can be used logically. Good.

  In addition, the contents of the embodiment may be realized as hardware as a semiconductor integrated circuit, or a part of the processing may be realized as software.

  Further, although not described in FIG. 10, the following embodiment may be employed. When a person who is not registered in the person database 81 is detected in the collation priority mode in step S04, the person registration unit 75 registers the person detected this time in the person database 81 and counts the appearance frequency. The area determination unit 77 can select an area including a specific individual as a gaze area using the appearance frequency of the newly registered person and the matching result. That is, the person storage unit 76 holds the number of times that the person has been detected so far, corresponding to each person information of the person included in the person area detected by the person detection unit 72, and the area determination unit 77 The region including the person region of the person who is held in the person storage unit 76 with a large number of times is preferentially detected as the gaze region. In this way, since a person who frequently appears in the imaging apparatus can be photographed as a sub-image, the photographer can avoid the trouble of registering the person in advance.

  Although the imaging apparatus 100 according to the present invention has been described based on the embodiments, the present invention is not limited to these embodiments. Other forms realized by combining arbitrary components in these embodiments, and forms obtained by subjecting these embodiments to modifications conceivable by those skilled in the art without departing from the gist of the present invention are also possible. Are included in the present invention.

  For example, a person motion detection unit may be provided in addition to the person direction detection unit 73, and the frame of the sub-image area may be shifted in consideration of not only the person direction but also the movement direction.

  In the above embodiment, a person is cited as the subject, but the present invention is not limited to this. For example, the subject may be an animal, an airplane or a train.

  The present invention is useful, for example, in the field of consumer video cameras, for example, as an imaging device, particularly as an imaging device that allows an amateur photographer to easily shoot good images.

DESCRIPTION OF SYMBOLS 1 Lens 2 Lens drive part 3 Image pick-up element 4 Image pick-up element drive part 5 Analog processing part 6 Digital processing part 7 Memory 8 Encoder 9 Recording part 10 Display part 11 Zoom switch 12 Detection part 13 Control part 15 Main image storage part 16 Sub image storage Part 21, 31, 51a to 51h Whole image 22, 53b, 53d, 53f, 53h, 61, 91 Main image 32 Gaze region image 33, 52a, 52c, 52e, 52g, 62 Sub image 63 Main image video 64 Sub image Movie 65 Recording medium 66 Multi-angle movie 71 Frame memory 72 Person detection unit 73 Person orientation detection unit 74 Person collation unit 75 Person registration unit 76 Person storage unit 77 Area determination unit 78 Elapsed time timer 79 Invalid time timer 81 Person database 82 Person detection・ Verification result 92, 93 area

Claims (10)

A semiconductor integrated circuit connected to an image sensor and a memory,
A main video storage unit that stores the video imaged by the image sensor as the main video in the memory;
A detection unit that repeatedly detects a gaze region that is a region to be watched out of the main video stored in the memory;
A sub-video storage unit that stores the video of the gaze area in the memory as a sub-video among the video imaged by the imaging device;
An encoder for compressing the main video and the sub video stored in the memory by the main video storage and the sub video storage;
A recording unit that records the main video and the sub-video compressed by the encoder unit on a recording medium;
The detector is
A semiconductor integrated circuit that determines whether or not the same subject is included in the current gaze region for a predetermined period, and detects the gaze region again when it is determined that the subject is included. The detector is
A person detection unit for detecting at least one person region that is a region including a person among videos captured by the imaging device;
The semiconductor integrated circuit according to claim 1, further comprising: a region detection unit that detects a region including any one of the detected at least one person region as the gaze region. The region detection unit
The semiconductor integrated circuit according to claim 2, wherein among the detected at least one person area, an area including a person area near the center of the main video is preferentially detected as the gaze area. The region detection unit
The semiconductor integrated circuit according to claim 2, wherein among the detected at least one person area, an area including a person area having a larger area is preferentially detected as the gaze area. The detection unit further includes:
A holding unit holding person information indicating a person;
A collation unit that collates a person included in each of at least one person area detected by the person detection unit and the person information held in the holding unit;
The region detection unit
The semiconductor integrated circuit according to claim 2, wherein the gaze region is detected based on a collation result by the collation unit. The holding unit further includes:
Corresponding to the person information, it holds feature data indicating the features of the person,
The collation unit
By comparing the feature of the person included in each of the at least one person area detected by the person detection unit with the feature data held in the holding unit, the feature of the person and the feature data are Calculate a matching score indicating the similarity of
The region detection unit
The semiconductor integrated circuit according to claim 5, wherein a region including a person region of a person having a high calculated collation score is preferentially detected as the gaze region. The holding unit further includes:
In correspondence with each person information of the person included in the person area detected by the person detection unit so far, the number of times that the person has been detected so far is held,
The region detection unit
The semiconductor integrated circuit according to claim 5, wherein a region including a person region of a person who is held by the holding unit more frequently is preferentially detected as the gaze region. The detection unit further includes a person direction detection unit that detects a direction of a person included in each of at least one person area detected by the person detection unit,
The region detection unit detects the gaze region using the orientation detected by the person orientation detection unit so that a region ahead of the person is larger than a region behind the person. 2. The semiconductor integrated circuit according to item 1. The semiconductor integrated circuit according to claim 1, wherein the predetermined time is not less than 5 seconds and not more than 30 seconds. A semiconductor integrated circuit according to any one of claims 1 to 9, an imaging device, and a display device,
An imaging apparatus that stores the main video and the sub-video in the memory.

Images