JP4995787B2 - Image storage device, program for image storage device, and image storage system - Google Patents

Description

The present invention relates to an image storage device that organizes and stores videos shot by a shooting terminal.

  As digital video cameras have become smaller and cheaper, it has become common to mount these cameras on mobile objects such as automobiles. In fact, such a camera has been commercialized and marketed as an in-vehicle camera.

  Video data shot by a digital video camera (hereinafter referred to as “video data”) is relatively large in capacity, but with the increase in capacity of SD memory cards that are resistant to vibration, etc. It is also possible to store video data for a considerably long time in the form of a digital data file.

  In addition, along with the downsizing and lowering of the price of GPS (Global Positioning System), it has become common to mount such a device that can identify its own position on a moving body such as an automobile.

  On the other hand, for the captured data, a personal computer (PC) installed in an office / data center or the like in the form of a digital data file in accordance with the increase in capacity and price of a hard disk drive (HDD). ) And server devices, and it is possible to accumulate long-term shooting data.

  With these conventional technologies, when video data shot with an in-vehicle camera and stored in an SD memory card or the like is copied as it is to a HDD such as a server by a photographer as it is, the file name etc. The recorded video data cannot be used effectively unless you know it.

  For this, for example, in Patent Document 1 below, a method of recording position information obtained from GPS in synchronization with image information is considered, and by using this method, an image is taken with an in-vehicle camera. By recording the frame number of the video data and the position information acquired at that time as a set, the video data within the specified distance range from the position information such as latitude and longitude specified by the user is recorded. A system for retrieving the target frame image can be easily configured. The frame number is a number indicating the number of image data in the video data. By specifying the frame number, the image data in the video data is specified.

  In addition, with the development of image processing technology, the number of pixels that can be obtained directly from the video data file, the frame rate (a value indicating how many frames an image per second is composed of), and the pixel value can be easily calculated. In addition to feature quantities such as image brightness, it is possible to extract motion (camera work) in the video, and it is also possible to detect objects by image recognition.

  Further, mainly in paragraphs 0037 to 0047 of Patent Document 2, two types of feature amounts, that is, an overall change amount due to the similarity of the entire image and a local set change amount due to the local similarity between adjacent frames in the video are included. A technique for detecting camera shake and abrupt camerawork by seeking is described.

  In Patent Document 3, mainly in paragraphs 0041 to 0049, feature points are extracted from previously captured images (registration images) to prepare learning maps, which are extracted from input images (recognition images). A technique is described in which image recognition is performed by searching for the feature points that have been performed by performing shift matching processing.

Further, according to the method described in Non-Patent Document 1, it is possible to search for an image including a designated character string. By using this method, a guide sign, signboard (character signboard) in the image can be searched. ) Etc., it is possible to automatically recognize whether or not a specific character string is included.
Japanese Patent Laid-Open No. 7-123348 JP 2000-261757 A JP 2007-299364 A “Recognition of characters in landscape without image extraction for image indexing”, IEICE Technical Report PRMU2004-89 (2004-10), p.37-42

  However, in the conventional technology, consideration is not given to updating of data stored in the server by continuously taking images over a long period of time by a plurality of photographers.

  In other words, there are many industries such as logistics companies, taxi operators, equipment, and road maintenance management contractors that carry out operations that repeatedly run cars in a certain area, such as in the same municipality or sales area. If you take a picture with a digital video camera installed in these cars, you should be able to acquire image data covering the target area. However, simply adding captured image data to the server sequentially and storing them increases the capacity of external storage devices such as HDDs, but increases unilaterally. Exceeds the storage capacity limit.

  Also, as another issue, when searching for an image with a specified position, especially when it is a position where photographers such as arterial roads frequently pass, there is no need for data that matches the search results. There is also the problem of hitting so many.

  As a countermeasure against the above problems, in order to keep the server capacity constant, based on the location information acquired at the same time, if data at the same location (or within a certain threshold distance) already exists In this case, it is possible to delete old data and store only newly taken data. Also, in order to narrow down the search results, it is also possible to hit only those having a new shooting time or update time to the server as a search result.

  However, for example, the image data taken on one day may be clearly displayed on a clear day, and the image data taken from the same point on the next day may be unclear at dusk in the rain. . Considering that there are cases like this, the method of accumulating only the latest data or hitting only the latest data as a search result has the problem that the next day's unclear data will be referenced. Arise.

  In this example, if the photographers for the two days are the same, it is possible for the photographer to determine that the next day's data is not registered in the server, but there are many photographers. In such a case, there is a problem that it is a heavy work burden to confirm whether or not to perform overwriting registration one by one.

  SUMMARY OF THE INVENTION An object of the present invention is to solve the above-described problems and to easily register only data effective for an image user in a server.

  In order to solve the above problem, the present invention provides an evaluation unit that calculates the validity of the data from the received data, and automatically determines whether or not to overwrite and register at the time of server registration according to the evaluation result. to decide.

Specifically, the present invention is an image storage device (data processing server) that receives and stores video shot by a shooting terminal, and includes video data and position data indicating a shooting position associated with the video data, Is used to map the position of the frame of the received video data using a server data receiving unit that receives data including the image data from the photographing terminal and the position data associated with the video data, and to the road section delimited in advance. Corresponding means for dividing the video data so as to obtain unit video data, and storing the unit video data and the validity value of the data calculated by a predetermined evaluation formula for the unit video data an image data storage section, when the, your new unit image data obtained from the received data the received data from the imaging device by the server data receiver The already when the unit video data captured at the same road section is accumulated in the image data storage unit calculates the value of the effectiveness of the data above for the new unit video data according to the predetermined evaluation formula The unit video data stored in the image data storage unit and the validity value of the data are compared with the validity value of the stored data by comparing the new unit video data and the validity of the data. And a data evaluation unit that determines whether or not to update with the degree value.

  As a result, it is possible to avoid storing unnecessary data with a small value stored in the image data storage unit.

The evaluation formula the data evaluation unit is used, the value of the effectiveness of the data, evaluation equation becomes larger as the number of frames the video data is large, or larger the larger the resolution of the unit video data evaluation formula, or An evaluation formula that increases as the average value of the brightness of the pixels included in the unit video data increases, or an evaluation formula that increases as the average value of the brightness difference of each frame in the unit video data increases, or a new shooting date and time increases. Or an evaluation formula that increases as the non-handshake segment ratio in the video segment to be accumulated calculated by a predetermined method increases, or an evaluation formula that combines a plurality of these evaluation formulas.

As a result, clear unit video data can be preferentially stored in the image data storage unit.

Further, the image storage device includes a data recognition unit for recognizing an object reflected in video data in data received from the photographing terminal, and a recognition data storage unit for storing a recognition result recognized by the data recognition unit. The data evaluation unit uses a recognition result stored in the recognition data storage unit in the calculation of the value of the validity of the data, and an evaluation formula that increases the value of the data effectiveness as the recognition rate increases. The data validity value may be calculated by using the data .

Video data with a high recognition rate for any object that is a subject is often clear video data. Therefore, by using the recognition rate to determine the validity of the data , appropriate unit video data is stored in the image data storage unit. It becomes possible to accumulate.

In the present invention, the data evaluation unit sets the number of unit video data captured in the same road section stored in the image data storage unit to a maximum of n (where n is an integer of 1 or more), and the reception If the data validity value is greater than the nth largest value of the data validity including the unit video data shot in the same road section stored in the image data storage unit, Unit video data obtained from the received data by deleting the least effective data from the received data and stored in the image data storage unit, and if smaller, unit video data obtained from the received data Can be rejected without being stored in the image data storage unit.

This makes it possible to store effective unit video data in an amount corresponding to the set value of n.

Further, in the invention, the data evaluation unit, to the image data storage unit, when storing the unit image data for each road segment, the same road section without distinguishing between uplink and downlink direction of the road section It can also be configured to determine whether or not there is.

In the case of video shot with an in-vehicle camera, depending on the application of unit video data stored in the image data storage unit, it may be better to distinguish the up direction and the down direction. Therefore, the amount of unit video data stored in the image data storage unit can be reduced.

In the present invention, the data evaluation unit separately calculates the validity value of the data and the magnitude of the validity value of the data for video data shot at different time periods, days of the week, due dates, or periods. It can also be configured to make a comparison.

Thus, the image photographed in the daytime for example, such as video captured on video, or commuting hours was taken in the evening, a unit video data of different time zones to be able to accumulate separately Become. Alternatively, it is possible to divide by weekday / holiday, and in snowfall areas, it can be divided into the snowing season from December to March and other seasons.

The image storage system includes the image storage device, an image capturing unit that captures an image , a position acquisition unit that acquires position information at the time of image capturing, the captured image data, and the acquired position data. And a terminal storage unit that transmits the data stored in the terminal storage unit to the image storage device.

  In the image storage system, an access point data receiving unit that receives data from the shooting terminal via a wireless LAN interface between the shooting terminal and the image storage device, and transmits the received data to the image storage device. It is good also as a structure provided with the data repeater which consists of an access point data transmission part.

  In the image storage system including this data repeater, the communication cost of the photographing terminal can be reduced as compared with the case where a mobile phone or the like is used as the photographing terminal.

According to the present invention, it becomes possible to accumulate only clear images among images that have been taken in duplicate at the same position, and save data effective for the user of the image with a small storage capacity. Will be able to.

  Embodiments of an image storage apparatus to which the present invention is applied will be described in detail below with reference to the drawings.

  FIG. 1 shows a configuration example of the entire system according to the embodiment of the present invention. In FIG. 1, reference numeral 10 denotes an in-vehicle camera, a camera-equipped mobile phone or other imaging terminal, 20 denotes a data processing server which is an image storage device for storing images taken by the imaging terminal 10, and 30 denotes communication over a wireless LAN over the Internet. It is a data repeater to relay. In the case where the photographing terminal 10 is configured to be able to communicate with the data processing server 20 directly via a telephone network or a communication network, the data relay 30 may not be provided.

  The imaging terminal 10 includes an imaging unit 11, a position information acquisition unit 12, a terminal storage unit 13, and a terminal data transmission unit 14. Details of these will be described later.

  The data processing server 20 includes a server data receiving unit 21, a data evaluation unit 22, an image data storage unit 23, a data recognition unit 24, and a recognition data storage unit 25. These details will also be described later.

  The data repeater 30 includes an access point data receiving unit 31 that receives a communication request from the photographing terminal 10 and an access point data transmitting unit 32 that transmits the requested communication data to the data processing server 20 via the Internet or the like.

  FIG. 2 shows a configuration example of the photographing terminal 10. In this example, a small portable PC (personal computer) 100 having a built-in flash memory 130 is used as the main body of the photographing terminal 10, the digital camera 110 is used as the photographing unit 11 in FIG. 1, and the GPS device 120 is used as the position information obtaining unit 12. The mobile phone 140 for data communication is used as the terminal data transmitter 14. The flash memory 130 is used as the terminal storage unit 13. The digital camera 110, the GPS device 120, and the data communication mobile phone 140 are connected to the small portable PC 100 by a USB cable or the like. Of course, the present invention can also be implemented as an imaging terminal 10 in which these devices are integrated.

  The digital camera 110 is fixedly installed at a position where the outside of the front can be photographed, such as a dashboard of a vehicle or a front grill. The GPS device 120 is also installed on a dashboard that can receive radio waves from GPS satellites, a ceiling outside the vehicle, and the like. The small portable PC 100 and the mobile phone 140 for data communication are also fixedly installed in a place where there is no problem in driving in the vehicle.

  Image data captured by the digital camera 110 and converted into numerical data of RGB values for each pixel (pixel) on the two-dimensional array are sequentially converted into images for each moment in JPEG format, and these data are fixed. It is stored in a storage drive using the flash memory 130 as a motion JPEG format file compiled for a certain amount of time. Alternatively, continuous images are converted into MPEG format files as one video and stored in a storage drive using the flash memory 130. Such means for storing video data as a standard format file such as motion JPEG or MPEG can also be realized by using software on a commercially available PC.

  Also from the GPS device 120, the self-position data calculated by receiving radio waves from GPS satellites is sequentially sent to the small portable PC 100. Since the self-location data is sent as text data in a predetermined format, the latitude / longitude data can be extracted by analyzing the text data with software on the small portable PC 100. At that time, along with the file name and frame number of the video acquired from the digital camera 110 and written as a video file, the latitude / longitude data is stored in a table format image data management table as shown in FIG. Write records one by one. Alternatively, it is possible to write one line at a time in a comma-delimited text file representing data as shown in FIG.

  The image data management table (or text file) shown in FIG. 3 includes data items such as the file name, frame number, shooting time, latitude, and longitude of the video file, and is stored as a file on the flash memory 130. Thus, the terminal storage unit 13 can be configured to store both the image data captured on the flash memory 130 and the acquired position data.

  Using the data communication mobile phone 140 connected to the small portable PC 100, connect this terminal to the Internet and establish a data communication connection to the data processing server 20 also connected to the Internet. Can do. The image data and the position data stored on the flash memory 130 are transferred to the data processing server 20 through the data communication connection thus established. In particular, it is easy to implement software for transmitting data to the HTTP server process on the data processing server 20 on the small portable PC 100 constituting the photographing terminal 10 using the standardized HTTP protocol.

  In this case, by determining a return value indicating normality / abnormality with respect to the HTTP request for data transmission, it is possible to determine whether or not the data processing server 20 has reliably received the data transfer. Since the above software can be determined, it is possible to perform a retry when the process is abnormal and to delete the transmitted data from the flash memory 130 when the process is normal.

  FIG. 4 is a diagram illustrating another configuration example of the photographing terminal 10. The difference from the photographing terminal 10 described in FIG. 2 is the data transmission method from the photographing terminal 10, and the photographing terminal 10 in FIG. 2 uses a mobile phone 140 for data communication as the terminal data transmission unit 14. In contrast, in this embodiment, a wireless LAN card 141 is used. The other digital camera 111, GPS device 121, flash memory 131, and small portable PC 101 are substantially the same as those shown in FIG.

  When the wireless LAN card 141 is used, unlike the mobile phone 140 for data communication, the wireless LAN card 141 can be used only in a limited place within the radio wave range of the wireless LAN access point. The position of the (data repeater 30) is registered, and a data communication connection is established when the position information falls within the range. Alternatively, an embodiment may be considered in which a “communication button” that can be operated by the user is arranged on the photographing terminal 10 and connected when the button is pressed.

  The data repeater 30 relays communication via a wireless LAN interface using the wireless LAN card 141 to the Internet, and a device having such a function is commercially available. Further, since the data transmitted in this way is not different from the case where the photographing terminal 10 of FIG. 2 is used, the data processing server 20 can be implemented with exactly the same configuration.

  The photographing terminal 10 shown in FIG. 4 is advantageous in terms of cost and communication speed because it generally does not use a mobile phone having a high communication cost and a low communication speed.

  FIG. 5 shows a configuration example of the data repeater 30 used when the imaging terminal 10 shown in FIG. 4 is used.

  The data repeater 30 in this embodiment includes a wireless LAN access point 301 from the photographing terminal 10, a network card (Ethernet card) 303 for connecting to the Internet 50, and a control CPU 302 for controlling these. As such a data repeater 30, a commercially available existing one can be used.

  Next, an embodiment of the data processing server 20 will be described. FIG. 6 shows a hardware configuration example of the data processing server 20.

  As shown in FIG. 6, the data processing server 20 is an adapter for connecting to a CPU 201, a magnetic disk device (HDD) 203, a memory 204, and the Internet 50 that fetches and executes instructions stored in the program storage area 202. The server data reception unit 21, the data evaluation unit 22, and the software program stored in the program storage area 202 are configured using a general-purpose computer including a network card 205. An image data storage unit 23, a data recognition unit 24, and a recognition data storage unit 25 are realized.

  Among these, the image data storage unit 23 in the data processing server 20 is realized by using a general-purpose database management system (DBMS) and configuring a table as shown in FIG. That is, the image data storage unit 23 stores unit video data (details will be described later), shooting time, latitude / longitude data of the start and end points, and the like. In this example, the video data itself is stored as one column on the table. However, as another embodiment, the video data is stored in a normal file format, and the file name is stored in the table in the DBMS. It is also possible to implement the present invention in the same manner by managing the above.

  FIG. 8 is a process flowchart of the server data receiving unit 21. The server data receiving unit 21 has a process execution function as shown in FIG. 8 corresponding to the embodiment of the terminal data transmitting unit 14 of the photographing terminal 10 described above.

  The server data receiving unit 21 first receives an HTTP request sent from the terminal data sending unit 14 in step S1, and in step S2, the video data and the position data sent by the HTTP protocol are respectively processed by data processing. The server 20 restores the video file 40 and the position information text file 41 as temporary files.

  Next, in step S3, it is determined whether or not the data can be restored without error. If the data can be restored normally, in step S4, the fact that the file has been received by the data evaluation unit 22 is recorded as event information together with the file name. Notice. Further, the presence or absence of abnormality during the process is returned to the photographing terminal 10 in step S5.

  FIG. 9 is a process flowchart of the data evaluation unit 22. Although an example in which data is evaluated for each divided unit video will be described here, it is apparent from the following description that data can be evaluated more simply for each frame image. is there.

  In step S10, the data evaluation unit 22 receives a notification of the event from the server data reception unit 21 and starts processing. First, in step S11, a process of dividing the video into the unit video from the data of the video file 40 and the position information text file 41 saved as a temporary file is performed. Here, the unit video means a unit obtained by dividing a road in a photographed area in advance.

  FIG. 10 is an explanatory diagram of a unit video section. The video section can be divided at each intersection as shown in FIG. 10A, or can be divided at regular intervals regardless of the intersection as shown in FIG. 10B. Further, as an extreme case, as shown in FIG. 10C, it is conceivable to divide each point (one frame image).

  In order to divide the video stored in the video file 40 into unit videos, position information for each frame in the video is necessary. However, the interval of video acquisition (that is, the frame rate of the video) and the position information acquisition are required. It is possible that the interval is different and there is no position information corresponding to the frame number. However, in that case, the position information for each frame can be given by using the information to which the position information is given among the preceding and following frames and approximating it by a calculation formula such as proportional distribution.

10A and 10B, the road map data of the target area is stored in advance, and the image of each frame in the video is displayed. Search for road location data. At this time, generally, there is an error in the position acquired by the GPS, so that it is mapped to the closest position on the road. If the road section to which the road position mapping the position of a frame belongs is different from the section to which the road position mapping the previous frame position belongs, it can be determined that there is a separation position between the frames. Divide the video with. Such a process as possible out to carry out the image segmentation.

  In the following description of the embodiment, the distance between the divided sections is assumed to be about several tens of meters, and each divided section can be uniquely specified by the latitude and longitude of the start point and the end point (there is no detour between the start point and the end point) The explanation is based on the assumption that the above condition holds. Even when the above conditions are not satisfied, the present invention can be implemented by a method of giving a road ID number or giving a latitude / longitude of an intermediate waypoint.

  Furthermore, the following processing (S121 to S126) is repeated for each section (for each image in the case of dividing each spot). First, in step S121, it is searched whether or not the same data as the target section exists in the image data storage unit 23. This can be realized by searching on the basis of the latitude and longitude of the start point and end point by the function of the DBMS. If it is determined in step S122 that there is no data matching the condition, the process proceeds to step S123, where new segment video segment data is additionally registered, and the process returns to the iterative process. Additional registration can also be realized using the DBMS function.

  If there is data that matches the condition, the process proceeds to step S124, and the effectiveness is calculated for each of the accumulated data and new data. A description will be given assuming that the validity of new data is A and the validity of stored data is B.

  An example of a method for calculating the effectiveness is shown in FIG. In this example, conditions such as high frame rate, high resolution, bright (not shooting in bad weather, dusk, etc.), clear contrast, and recently taken are considered to be effective. The calculation is made by a relatively simple formula such as multiplying by a weighting factor and summing.

In step S30, the number of frames (or frame rate) of the section video is set as the effectiveness A _fr . In step S31, the resolution of the section video is _Ares . The resolution of the section video is the number of vertical pixels × the number of horizontal pixels in one frame. Next, in step S32, the effectiveness _Abr focusing on the brightness is calculated as the average value of the brightness of each pixel of each frame of the section video.

A _{A br = Σ f Σ x Σ} y Y (f, x, y) / (× number of vertical pixels × number of horizontal pixels the number of frames). However, the brightness Y is calculated from the RGB values of the pixels using the following formula called NTSC weighted average method. That is, when the brightness of the (x, y) pixel in the f frame is Y (f, x, y), Y (f, x, y) is calculated by the following equation.

Y (f, x, y) = 0.298912 * R (f, x, y) + 0.586611 * G (f, x, y) + 0.114478 * B (f, x, y)
In the next step S33, the average value of the brightness difference of each frame of the section video is calculated as the effectiveness A _con focusing on the contrast.

A _con = Σ _f {Y _max (f) −Y _min (f)} / number of frames. However, Y _max (f) is the maximum value of lightness in frame f, and Y _min (f) is the minimum value of lightness in frame f.

In step S34, the number of seconds elapsed from a certain reference date and time is set as the effectiveness A _ti for the shooting date and time. The reference date and time may be any time as long as it is older than the oldest shooting date and time.

  Finally, in step S35, the effectiveness A that combines the above is calculated by the following equation.

A = ΣC _i × A _i
Here, A _i is A _fr (effectiveness focused on the number of frames), A _res (effectiveness focused on the resolution), A _br (effectiveness focused on the brightness), A calculated in steps S30 to S34. _con (effectiveness focused on contrast) and A _ti (effectiveness focused on shooting date). C _i is a weighting coefficient for each predetermined evaluation item of A _i .

  If it is determined in step S125 in FIG. 9 that the validity A of the new video data is higher than the validity B of the stored data, the process proceeds to step S126, and the image data storage unit 23 is used using the DBMS function. Is deleted, and a new divided video is registered in the image data storage unit 23. That is, the video data at the same position stored in the image data storage unit 23 is updated to new video data. When new video data is stored in the image data storage unit 23, the validity A may be stored as the validity B of the stored data.

  If the validity B of the stored data is higher in the evaluation of the effectiveness in step S125, the new video data is rejected without being registered in the image data storage unit 23. The above steps S121 to S126 are repeated for each divided video.

  The calculation formula used in the effectiveness calculation method as shown in FIG. 11 can be changed to an appropriate formula according to the specific use purpose of the present invention. The following are specific examples of changes.

For example, with regard to brightness, in the example of FIG. 11, the value of “A _br ” increases as the brightness increases, but an image that is too bright may not always be desirable. Considering this, the difference from the ideal expected value may be calculated as the value of “A _br ”. In this case, since the numerical value becomes smaller as it is closer to the expected value, by taking a negative value as the weighting coefficient, the closer to the expected value, the higher the effectiveness can be evaluated. That is,
A _br = interval deviation of the mean value from the expected value of the brightness of each pixel of each frame of the video _{= Σ f Σ x Σ y abs} (Y (f, x, y) -Y 0) / ( number of frames × vertical pixels Number x number of horizontal pixels)
It is. Here, abs () is a function for obtaining an absolute value, and Y ₀ is an ideal brightness value.

Further, since the contrast is evaluated using only the maximum value and the minimum value in the example of FIG. 11, only a very limited number of pixels may be extremely bright or extremely dark. The value of “A _con ” may be highly evaluated. Therefore, by performing statistical processing on the brightness of the entire pixel, it is possible to calculate a standard deviation that serves as an index of the distribution of brightness. Therefore, it is also preferable to use the index as the value of “A _con ”. That is,
A _con = average value standard value of brightness of each frame of the section video = Σ _f {σ (f)} / number of frames
It is. σ (f) is a standard deviation obtained by statistically processing the lightness distribution Y (f, x, y) in the frame f. Since this value becomes larger as the bias is smaller (that is, the lightness is evenly distributed), the effectiveness becomes higher.

Furthermore, these values can be obtained after performing mathematical calculations instead of using them as they are, depending on their effectiveness. For example, in calculating the value of the effectiveness “A _ti ” for the shooting date and time, the difference from the current date and time can be calculated by a calculation formula that halves the value every month.

A _ti = (1/2) ** (current date and time-shooting date and time)
Here, ** represents the power, and the date / time is calculated in terms of months.

  In other words, compared with a very new image, the calculation formula is such that the effectiveness is 1/2 for an image 1 month ago, 1/4 for an image 2 months ago, and 1/8 for an image 3 months ago. It may be used, but this is a big difference between “current image or image 1 month ago”, but “image 1 year (12 months) ago or image 1 year 1 month (13 months) ago” This is a useful value if the difference has little effect.

In the above example, the value of “A _br ”, the value of “A _con ”, and the like are described based on all the images constituting the video. Therefore, it is considered that there is little change in the large value between adjacent images. Therefore, in order to reduce the amount of calculation, it is also preferable to perform calculation for only a part of images, such as method 1 to method 3 described below.
[Method 1] Only information of an image having a specific frame number such as the beginning or end is used.
[Method 2] Only information of an image having a frame number randomly specified at the time of evaluation is used.
[Method 3] The average value is calculated after the frame rate is lowered (for example, 30 fps video is skipped by 30 frames to obtain 1 fps).

  As another method for determining whether or not to update the image data based on the effectiveness, it is also preferable to use the following method.

  (1) The image data to be stored is not limited to the one with the highest validity, but is set to a maximum of n, and if the validity for a new image is greater than the nth existing validity, a new New image is added and the nth (the least effective one among existing ones) is deleted. As a result, n pieces of image data corresponding to various environmental conditions are stored at the same position, and the probability that appropriate image data required by the user is overwritten is reduced.

  (2) In the above example, since the position information is the latitude and longitude of the starting point and the ending point, the upper and lower sides are actually different even in the same section. This is intentionally identified as the top and bottom, so that only one with high effectiveness is obtained in the up and down directions. By doing so, the amount of image data stored in the storage device can be further reduced.

  (3) It is possible to add not only position information but also time information of photographing time to the image data and divide the stored image data not only by position information but also by time zone. In other words, by providing a column “time zone” on the data and using it as a key, “day” and “night” can be stored separately. Similarly, when there may be different situations in time, such as dividing by weekdays / holidays or by specific due dates or seasons, implementation is also possible to store and manage them separately using the key of time information. Is possible.

  (4) By saving the evaluation value of the validity calculated once in the image data storage unit 23, when new data is transmitted, the validity of the existing data is not recalculated. You can use saved data.

  Next, an embodiment in which the data processing server 20 uses the data recognition unit 24 that recognizes an object in the image data from the image data received from the photographing terminal 10 and accumulates the recognition result in the recognition data accumulation unit 25 will be described. .

  FIG. 12 is a processing flowchart of the data evaluation unit 22 in this embodiment. The process flowchart of FIG. 9 and the outline of the overall process are the same. However, the present embodiment is different in that the recognition result by the data recognition unit 24 is used as the evaluation value in the calculation of the effectiveness in step S224. Below, this difference is explained in detail.

  Many techniques have been devised that extract feature values from image data or video data composed of continuous image sequences and perform effective recognition (Patent Document 2, Patent Document 3, Non-Patent Document 1, etc.). Using these techniques, it is possible to configure a software library that outputs a recognition result when image data is input, and the data recognition unit 24 in this embodiment is implemented using this software library. be able to. Specifically, it can be realized as follows.

  According to Patent Document 2, camera shake and abrupt camera are obtained by obtaining two types of feature amounts, that is, an overall change amount due to the similarity of the entire image and a local set change amount due to the local similarity between adjacent frames in the video. The workpiece can be detected. In the case of video from an in-vehicle camera, shaking when starting or stopping the vehicle, getting over obstacles, or shaking due to inadequate camera installation is considered to be equivalent to “camera shake”. As a result, it is possible to calculate the ratio (%) of the camera shake section in the original video section. By subtracting this value from 100%, it is possible to calculate the non-blurring segment rate (that is, the proportion of valid segments) in the target video segment. Since the method of Patent Document 2 can be realized by software on a computer, it is possible to construct a software library that receives video data as an input and outputs this non-shake interval ratio.

  According to Patent Document 3, a feature map is prepared by extracting a feature point from a previously captured image (registration image), and the feature point extracted from the input image (recognition image) is shifted. Searching and image recognition can be performed by performing matching processing. Therefore, by preparing objects such as traffic signs and signboards of specific shapes (convenience stores, restaurants, etc.) as registration images in advance, whether these objects exist in the recognition image or not. If it exists, it is possible to calculate the position on the image.

  FIG. 13 is a process flowchart of the data recognition unit 24. When the data recognition unit 24 receives the video data 60 with position information for recognition (= continuous image data), the image recognition unit 24 extracts image data frame by frame by iterative processing and determines which object is in which position in each frame. To do. Therefore, first, in step S50, the video data 60 with position information is separated into image data for each frame. Next, in step S51, image recognition is performed using the techniques described in Patent Document 2, Patent Document 3, Non-Patent Document 1, and the like described above, and an object (a recognition target) prepared as a registration image for recognition. If there is a thing), that is the recognition result. In step S52, the recognition result is stored in the recognition result table 61 in the memory area.

  FIG. 14A shows an example of the recognition result table 61. Here, a road sign is an object to be recognized. As shown in FIG. 14A, the recognition result table 61 includes a frame number, a recognized object type (or object name, etc.), an object position indicating the position of the recognized object in the frame, and the vertical and horizontal directions of the object. Information such as the size indicating the size of the file is stored.

  In step S53, it is determined whether or not the recognition has been completed up to the final frame. If the processing has not been completed up to the final frame, the process returns to step S50, and image recognition for each frame is repeated.

If the recognition has been completed up to the last frame, it is next determined in step S54 whether or not the conditions of unnecessary recognition results so far are met. Here, for example, a case where all of the following conditions are satisfied is determined as an unnecessary recognition result.
[Condition 1] Contiguous adjacent frames have the same object type.
[Condition 2] The difference in object positions is less than or equal to a predetermined threshold value.
[Condition 3] The size is not the largest among the objects satisfying Condition 1 and Condition 2.

  If all of the above conditions 1 to 3 are satisfied in the record of the recognition result table 61, the process proceeds to step S55, and the record is deleted from the recognition result table 61. By this processing, for example, in the recognition result table 61 shown in FIG. 14A, the “stop” record of the (N−1) th frame and the “no right turn” record of the Nth and (N + 1) th frames are deleted. Become.

  In step S56, it is determined whether or not the deletion condition up to the last record in the recognition result table 61 has been checked. If there is an unchecked record, the process returns to step S55 and the process is repeated. If all are checked, the process proceeds to step S57.

  In step S57, the frame number of each record remaining in the recognition result table 61 is converted into latitude / longitude information based on the position information added to the video data, and the result is output as the object recognition result 62. FIG. 14B shows an example of the output object recognition result 62. This object recognition result 62 is stored in the recognition data storage unit 25.

  The processes of steps S54 and S55 will be described in more detail according to a specific example.

  FIG. 15 shows an example of the same object moving to an adjacent frame. R in the figure indicates a region recognized as a “right turn prohibition sign”. As shown in FIG. 15, in the case of a video from an in-vehicle camera, since the same sign or the like is generally shown in a plurality of adjacent frames, the recognition result of the same object is displayed in the plurality of adjacent frames. It is output as a position close to the top.

  As described above, for the object having the condition that “the same kind of object and the difference in position on the image is equal to or less than a threshold value” between adjacent frames, the output result captured most greatly, that is, the image of the recognition result. The recognition result in other frames is deleted, leaving the output result having the maximum size. In general, in the case of an in-vehicle camera installed in the front, the object gradually increases as the vehicle travels, and the frame is finally out of the frame, so that the recognition result of the last frame in which the object is photographed remains.

  In this embodiment, since position information such as latitude and longitude is given to the video data, the position of each recognized object (to be precise, road Above, the position where the object looks the largest). In this way, for the input video data with position information, the object or objects that can be extracted from the video is the position of the object (more precisely, the position where the object appears to be the largest on the road). ) Output with data.

  Further, in the image recognition in step S51, according to the method described in Non-Patent Document 1, it is possible to search for an image including a designated character string. It is possible to calculate whether or not a specific character string is present on the image, and if so, in which position it is located on the image. Therefore, for the input video data 60 with position information, the character string such as one or more place names that can be extracted from the video is changed to the position of the character string (more precisely, the character string on the road A software library that is output together with the most visible position) data can be constructed, and the data recognition unit 24 can be configured by the software library.

  FIG. 16 shows a configuration example of the recognition data storage unit 25. The recognition data storage unit 25 is configured by a database management system (DBMS) that manages information such as latitude and longitude indicating the position of a captured image, object type, object position in the image, and object size. .

  FIG. 17 shows an example of the effectiveness calculation process in the data evaluation unit 22 using the recognition result of the data recognition unit 24 described above.

First, at steps S40 to S42, an evaluation value that can be acquired from the video data is obtained. In step S40, the number of frames of the section video is set as the effectiveness A _fr related to the number of frames. In step S41, the resolution of the section video, that is, the number of vertical pixels × the horizontal pixel is set as the resolution effectiveness _Ares . Similarly, in step S42, various effectiveness A _i as described in FIG. 11 is calculated.

Subsequently, in steps S43 and S44, an evaluation value of the effectiveness is obtained using the data recognition unit 24. First, in step S43, the data recognition unit 24 is requested to recognize the camera work, and the data recognition unit 24 detects the camera work using the camera work recognition technology described in Patent Document 2, and detects the detection. The data evaluation unit 22 is notified of the information of the camera work performed. Based on this result, the data evaluation unit 22 calculates the effectiveness A _cw related to the camera work stability from the ratio of the sections without the camera shake section (non-shake section ratio). A _cw is calculated from the following equation.

A _cw = 100 − (% of the motion picture segment is in the segment video)
Next, in step S44, the data recognition unit 24 is requested to recognize signs, signboards, etc., and based on the recognition result, the effectiveness A _rec regarding the recognition result is calculated. The data recognition unit 24 performs object recognition and character recognition using the object recognition technology described in Patent Document 3 and the character recognition technology described in Non-Patent Document 1, and The recognition result is notified to the data evaluation unit 22. The data evaluation unit 22 searches the existing recognition results stored in the recognition data storage unit 25 from the position information of the image, calculates the number of matching recognition results, and determines the number of matching recognition results regarding the recognition level. The effectiveness is _Arec .

That is, the data recognition unit 24 obtains recognition results of objects such as signs and characters from the video data with position information by the above-described recognition processing. Such recognition results are stored in the recognition data storage unit 25, which is also composed of DBMS, “what kind of recognition data existed in which position (latitude / longitude) and in what position on the image”. (See FIG. 14B). When new data is transmitted from the data evaluation unit 22, data recognition is performed on whether the same recognition data exists at the same position (= the same position within a certain threshold range) on the new data. The unit 24 inspects and calculates the number of the same recognition, that is, the matching recognition result as the effectiveness _Arec .

  In general, high image quality is required for many computer recognitions to succeed. In other words, the effectiveness of the image when recognition is successful is high. Therefore, this value is used as one of the evaluation formulas for effectiveness. It should be noted that objects such as store signboards and information signs that are still not recognized (or newly recognized) even when using images with a high recognition success rate are actually differences caused by secular changes in the city. Therefore, the recognition data storage unit 25 also holds the latest information on such objects.

  Thereafter, in step S45, the total effectiveness A is calculated by the following equation.

A = ΣC _i × A _i
However, C _i is the effective index _{A fr, A res, ...,} A cw, A respective evaluation predetermined weighting factor C _fr for items of _rec, C _res, ..., a C _cw, C _rec.

  The processing performed by the above image storage device can be realized by a computer and a software program, and the program can be recorded on a computer-readable recording medium or provided through a network.

It is a figure which shows the example of whole structure concerning embodiment of this invention. It is a figure which shows the structural example of an imaging | photography terminal. It is a figure which shows the example of an image data management table. It is a figure which shows the other structural example of an imaging | photography terminal. It is a figure which shows the structural example of a data repeater. It is a figure which shows the structural example of a data processing server. It is a figure which shows the structural example of an image data storage part. It is a process flowchart of a server data receiving part. It is a processing flowchart of a data evaluation part. It is explanatory drawing of a unit video area. It is a figure which shows an example of the calculation method of effectiveness. It is a process flowchart of the data evaluation part in another Example. It is a process flowchart of a data recognition part. It is a figure which shows the example of a recognition result table and an object recognition result. It is a figure which shows the example of the same object which moves to an adjacent frame. It is a figure which shows the structural example of a recognition data storage part. It is a figure which shows an example of the effectiveness calculation process in a data evaluation part.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Image | photographing terminal 11 Image | photographing part 12 Position information acquisition part 13 Terminal storage part 14 Terminal data transmission part 20 Data processing server 21 Server data reception part 22 Data evaluation part 23 Image data storage part 24 Data recognition part 25 Recognition data storage part 30 Data relay 31 Access point data receiver 32 Access point data transmitter

Claims (9)

An image storage device that receives and stores video shot by a shooting terminal,
A server data receiving unit for receiving data including video data and position data indicating a shooting position associated with the video data from the shooting terminal;
Means for mapping the position of the frame of the received video data using the position data associated with the video data, and dividing the video data so as to correspond to the road section divided in advance to obtain unit video data When,
An image data storage unit for storing the unit video data and a value of the validity of the data calculated by a predetermined evaluation formula for the unit video data;
Wherein when data is received from the imaging device by the server data receiver, for each new unit video data obtained from the received data, stored previously in the unit image data is the image data storage section taken at the same road section If the new unit video data , the image data storage unit calculates the value of the validity of the data according to the predetermined evaluation formula, and compares the value with the value of the validity of the stored data. and characterized in that it comprises a unit video data stored to the the value of the effectiveness of the data and the new unit video data and the data evaluation unit that determines whether to update with a value of effectiveness of the data to Image storage device. The image storage device according to claim 1,
The evaluation formula used by the data evaluation unit is:
The value of the effectiveness of the data, the brightness of the frame number is large enough larger evaluation formula unit video data, or an evaluation formula increases the greater the resolution of the unit video data, or pixels included in the unit image data An evaluation formula that increases as the average value increases, or an evaluation formula that increases as the average value of the brightness difference of each frame in the unit video data increases, or an evaluation formula that increases as the shooting date is new, or a predetermined method. An image storage apparatus comprising: an evaluation formula that increases as the non-blurring section ratio in the stored video segment increases, or an evaluation formula that combines a plurality of these evaluation formulas. The image storage device according to claim 1 or 2,
A data recognition unit for recognizing an object reflected in video data in the data received from the photographing terminal;
A recognition data storage unit for storing a recognition result recognized by the data recognition unit,
The data evaluation unit
In the calculation of the data validity value, the recognition result stored in the recognition data storage unit is used, and the data validity value is calculated using an evaluation formula in which the data validity value increases as the recognition rate increases. An image accumulating device characterized by: In the image storage device according to any one of claims 1 to 3,
The data evaluation unit
The maximum number of unit video data shot in the same road section stored in the image data storage unit is n (where n is an integer of 1 or more), and the validity value of the received data is the image data. Data having the smallest data validity among the accumulated data when the validity is greater than the nth largest value of the data including unit video data shot in the same road section accumulated in the accumulation unit. The unit video data obtained from the received data is saved in the image data storage unit, and the unit video data obtained from the received data is rejected without being stored in the image data storage unit when the data is small An image storage device characterized by that. In the image storage device according to any one of claims 1 to 4,
The data evaluation unit
In the image data storage unit, when storing the unit image data for each road section, characterized by determining whether the same road section without distinguishing between uplink and downlink direction of the road section Image storage device. In the image storage device according to any one of claims 1 to 5,
The data evaluation unit
An image storage device characterized in that, for unit video data shot at different time periods, days of the week, due dates, or periods, the validity value of the data is calculated separately and the magnitudes of the validity values of the data are compared separately .   The computer according to any one of claims 1 to 6, wherein the image data storage device according to any one of claims 1 to 6 includes the server data reception unit, the image data storage unit, and the data evaluation unit, or further according to claim 3. An image storage device program for causing the image storage device to function as the data recognition unit and the recognition data storage unit. An image storage system comprising a photographing terminal and an image accumulating device for accumulating images photographed by the photographing terminal,
The photographing terminal is
A shooting section for shooting images ;
A position acquisition unit for acquiring position information at the time of shooting;
A terminal storage unit that stores the captured video data and the acquired position data in association with each other;
A terminal transmission unit for transmitting data stored in the terminal storage unit to the image storage device;
The image storage device includes:
A server data receiving unit for receiving data including video data and position data indicating a shooting position associated with the video data from the shooting terminal;
Means for mapping the position of the frame of the received video data using the position data associated with the video data, and dividing the video data so as to correspond to the road section divided in advance to obtain unit video data When,
An image data storage unit for storing the unit video data and a value of the validity of the data calculated by a predetermined evaluation formula for the unit video data;
Wherein when data is received from the imaging device by the server data receiver, for each new unit video data obtained from the received data, stored previously in the unit image data is the image data storage section taken at the same road section If the new unit video data , the image data storage unit calculates the value of the validity of the data according to the predetermined evaluation formula, and compares the value with the value of the validity of the stored data. and characterized in that it comprises a unit video data stored to the the value of the effectiveness of the data and the new unit video data and the data evaluation unit that determines whether to update with a value of effectiveness of the data to Image storage system. The image storage system according to claim 8,
Between the photographing terminal and the image storage device,
An access point data receiving unit for receiving data from the photographing terminal through a wireless LAN interface;
An image storage system comprising: a data repeater including an access point data transmission unit that transmits received data to the image storage device.

Images