Method of ranking video data. RU patent 2484529.

FIELD: information technology.

SUBSTANCE: in the method, fragments which correspond to an object and/or an event are selected from initial video data; features are calculated for each fragment, which influence fragment priority estimation and/or are used when searching for fragments in the storage; the priority of each fragment is estimated based on its features; the fragments are sorted each according to its priority and the obtained priority queue of fragments or a fragment with the highest priority is transmitted over communication channels to a user and/or storage.

EFFECT: high quality of processing video data without loss of quality thereof.

26 cl, 4 dwg, 2 tbl

The invention relates to the processing of data, namely, the closed circuit television, CCTV and video Analytics. The invention allows to effectively build geographically distributed video surveillance systems in various industries, including the protection and security, transport and retail sales, sports and entertainment, utilities and social infrastructure. Invention can be used in local and global networks, on the allocated or cloud-based servers. The invention allows to reduce the cost analysis, storage and transfer video, as well as expand the scope of services VSaaS - video security as a service (eng. video surveillance as a service).

One of the significant barriers to the development of territorially-distributed video surveillance systems a significant amount of data from the cameras. Even with the use of modern compression algorithms, such as H.264, camera standard-definition (0.4 megapixels) form the flow of data from 0.5 to 4 Mbit/s, and high definition cameras (1-3 megapixel) from 1 to 10 Mbps (high quality video). For systems with a large number of cameras, the cost of transfer, storage and analysis of data are critical.

In particular, serious barrier for the development of services VSaaS are insufficient throughput capacity of communication channels outside the local network. So, the average speed of the connection of the subscriber to the Internet in the world amounts to 1.8 Mbit/s in 2010. When using asymmetric access technologies (for example, via an ADSL or cable modem) outgoing from the subscriber to the application VSaaS 4-10 times less incoming, and the average is less than 512 Kbit/sec Thus, the service VSaaS to realize remotely view and record video with more cameras, especially high-definition cameras (more than 1 megapixel).

As disclosed in the international application WO 2011041903 published 14.04.2011, video Analytics, implemented on the side of the video source that is built-in video Analytics allows you to reduce the volume of information transmitted via the communication channels. However, in systems with a large number of cameras and a large number of events recorded , this approach is not sufficient to reduce the amount of data transmitted via communication channels and recorded in the archive. On busy sites Analytics generates a continuous stream of events and a real reduction of the amount of transmitted data occurs.

Another problem is the large number of alarm events coming in situational centers and remote remote protection. Operators often do not have time to respond to alarm events generated by the video surveillance system with Analytics, and unable to focus on solving the most important tasks.

Storage of large volumes of video is the high cost. In existing storage facilities videos (DVRs) are used the following methods to increase the efficient use of disk space: (a) removal of old videos with the possibility of manual locking delete individual records (function lock); b) reduction of the volume of the archive of the video due to higher compression, which negatively affects the quality of video (spatial and temporal resolution of the video).

On the solution of the abovementioned problems and directed the invention.

How to rank video, comprising the production of video data with at least one video camera and transfer sorted video channels at least one user and/or, at least, in one store, which is characterized by the fact that at first received from source video distinguish the fragments corresponding to at least one object and/or event, then calculate the characteristics of each fragment that affect the measurement of priority fragment and/or used by search snippets in the repository, then assess the priority of each fragment taking into account its features, then sorts the fragments, in accordance with the priority of each of them and passed in through the channels of communication, at least one user and/or, at least one store received priority queue fragments or one piece with the highest priority.

Fragments may be released with the help of the motion detector.

Fragments may be released with the help of video Analytics, integrated in the network camera or video server.

Fragments may be released with the help of video Analytics server.

Priority evaluation of each fragment can perform by using the regression function.

Construction of regression functions can be based on the statistics of user requests.

Priority evaluation of each fragment can be performed using the statistical classifier.

Training of the statistical classifier can be based on the statistics of user requests.

Priority evaluation of each fragment can perform using rules defined by the user.

Priority evaluation of each fragment can be based on the current user's requests.

Priority evaluation of each fragment the user can install it manually.

Priority evaluation of each fragment can be based on the priority of the camera, which is the source of the fragment of the video.

To assess the priority of each fragment can use one sign, calculated , such as the accuracy of the detection of an object or situation.

For the assessment of priority, you may use the value of the priority, pre-defined by the user in the table for each type of objects or situations.

For assessment of priority fragment can use the sum or maximum estimates of the priorities of individual objects and/or conditions contained in this fragment.

For assessment of priority can use the data obtained from external sensors.

Fragment of the video may be a sequence of frames.

Fragment of a video can be an image - alarming frame or part of the image.

The user can be the operator of video surveillance systems.

Time of storage fragment passed in the repository, may depend on the priority of the fragment.

When sending video via the communication channels with limited bandwidth in the first place can transmit fragments with the highest priority.

Received the priority queue can be displayed on the user interface.

Fragments received priority queue can be displayed on the user interface, color or other graphic indicator that depends on the priority of the fragment.

Can be performed certain actions to attract attention of the operator: the alarm and/or send a text message (eng. Short Message Service short message service) depending on the received assessment of priority.

Each fragment of a priority queue can be delivered in a separate file.

The priority queue can be shared between multiple users.

The present invention is illustrated by figure 1-4. Figure 1 shows the General scheme of the method of ranking the video data. Figure 2 shows a diagram of a CCTV system. Fig.3 shows the thumbnail of the graphic user interface of the system. Video fragments are highlighted and are presented in chronological order. Figure 4 shows a sketch of the graphical user interface of the System. Video fragments are highlighted and presented as a matter of priority.

Way ranked video includes the following steps shown in figure 1:

Step 1. The selection of the fragments of the video using video Analytics

Original video from surveillance cameras, splits into fragments. Fragment can be a sequence of frames, single frame (alarm frame) or part of the frame. Video splitting into fragments occurs so that each fragment meet one item or event, observed camera and of interest to the user. It is possible that in one fragment, there are several objects in case of their simultaneous occurrence.

You can use different types of , for example, motion detector, the detector people detector numbers of cars, fire detector and so on. We recommend the use of tracking algorithms (tracking) for objects with the purpose of completion of formation of a fragment after the disappearance of the object detected. Possible to apply rules for the automatic recognition of predefined situations, for example, crossing the signal line (tripwire), (loitering) and abandoned item.

Examples of fragments is: (a) the person's face, extracted from the video stream; b) alarming frame with the advent of man in the sterile zone; C) a video with the travel of the vehicle at a red light.

It is advisable to limit the size of the fragment to reduce the time of transfer of the most priority fragments via the communication channels with limited bandwidth and more efficient use of warehouse video.

There may be times, especially for lively scenes, when one snippet contains several objects or situations.

Step 2. Calculation of characteristics of fragments

For each piece of calculated characteristics that affect the valuation of priority fragment and/or can be used for subsequent search of fragments in the store. Examples of characteristics that can be used in the Invention, are given in Table 1.

Group signs

Examples of signs

Date and time

Date and time of the beginning of the fragment

Date and time of end of the fragment

The fragment duration

ID and priority camera

ID and priority object detection zone

The presence of certain types of objects (classes)

Number objects

Assessing the accuracy of detecting objects

Situation

Success/fail predefined rules

(Entrance to the zone, the zone exit, stop loitering and others)

Detection of smoke or fire

Signal loss, global changes in the field of view camera

The darkening of the image that the camera can be caused by disconnecting lighting or breakage of the diaphragm

External sensors

The status of the additional sensors that are integrated with the system of video surveillance (, door alarm, alarm, gas etc)

User requests

The number of users of video surveillance system, requesting fragment

Time/relevance requests

On Fig.1 signs fragments identified by lowercase letters: a, b,...

Assessment of priority of each fragment can be done automatically formal or empirical methods, as well as in manual mode.

Among formal methods of assessment of priorities follow noted:

A. The regression function. It is assumed that the priority of the fragment and the sign of the fragment - dependent random variables. Priority fragment depends on the characteristic pieces of information: q=f(x), where f is a function of regression, determines the dependence of the priority q fragment of the vector x contains elements of a fragment or object on the fragment. Priority q fragment can take on continuous values, for example, from 0 (lowest priority) to 1 (highest priority). Can use known methods for constructing the regression function, for example, linear regression or support vector machine (SVM, support vector machine).

B. Statistical classification determines the priority of a fragment of evidence-based fragment: q=C(x), where q is the priority of the fragment With the statistical classifier, x is a vector containing the signs of fragment or object on the fragment. Classifier issues discrete values q, for example, 1 (low priority), 2 (medium priority) and 3 (high priority). Can use the well-known classifications, for example, support vector machine (SVM), nearest neighbor (k-NN) or method (boosting).

In the formal approaches, build regression function or training the classifier can be made: a) at the configuration stage surveillance system; and/or b) continuously in the process of operation of video surveillance system for recording statistics of the events and actions of the user.

On Fig.1 priorities fragments are indicated by capital letters: H (high - high), M (Medium - medium), L (low - low).

Among the possible empirical approaches to the evaluation of priorities fragments are:

A. A trivial estimation to one basis: (q=x or q=-x, where x is a sign of a fragment such as the accuracy of the detection of the object, the number of detected objects, the distance from the object to the camera.

B. Table of priorities: q=Q[x], where x is the type of situation or object type on a fragment, a certain ; Q - reference table of priorities (lookup table), which determines the priority of a slice for each type of situation or object type. Example of the table of priorities is given in table 2.

In case when on the fragment contains many objects or situations, the attributes of each object or situation can be aggregated into the overall assessment of the priority of using such functions as sums or maximum. For example, q=Σ i p i q i /S i p i or q=max i p i q i p i - evaluation of the accuracy of detection (recognition) of the i-th object or situation on a fragment of video, q i - priority i-th object or situation that is determined by reference table, t=1, 2, ... - number of the object or situation. If the algorithm video Analytics does not calculate precision, that relies p i =1.

Users can change the priority of each fragment to control the period of storage fragment of the video in the archive or transfer of fragments in channels with a narrow bandwidth.

Value priorities for typical situations

Examples of situations

Recommended priority q i

Detection of smoke or fire

Overcoming the fence

Human movement towards the building on the protected territory outside working hours

Signal loss or distortion of video quality, significant changes in the field of view camera

Movement of a person from the building on the protected territory

Loitering human Parking

Abandoned object

When motion detector

Running man

Education crowded

Human movement near the building during working hours

Maybe so same job priori priorities for cameras or detectable objects (situations). In this case, the formula for determining priority will be of the form: q=q s f(x), where q s - priority event source (camera priority or surveillance zone).

Step 4. Ranking of all the fragments estimated priority

In step 4 is to rank all of fragments according to the estimates of priority, for the purpose of processing the most priority data first. Ranked fragments can come in a data structure called a priority queue. A particular case of the ranking is to search for one fragment of the highest priority.

For example, a fragment of a video with the detected fire may have higher priority than the fragment with man's appearance in front of the door. A fragment that is transmitted via communication channels, is presented to the operator and is recorded in the repository first.

Fragments of video data with low priority may not be processed, but can be written to a local store in case a user's query. A significant reduction of the load on the channels of communication, storage and operator by filtering out the fragments of low priority.

As new data (the original video), steps 1-4 are repeated.

Thus, the invention of effective transmission not by reducing the video quality, both in terms of products-analogues, and by priority send most important to the user information. Priority transmission of video fragments or part of the frame is determined by the results of work in video Analytics and/or user requests.

The best analogues exercise video by motion detection and transmission of video clips without taking into account their importance, or continuous transmission of video, which leads to the rapid exhaustion of disk storage space and/or bandwidth. Well as the nearest analogues not allow to rank data for the operator, which will increase the required number of operators in monitoring centers.

Figure 2 illustrated one of the possible variants of application of the present invention. Uncompressed video comes with a sensor (camera) on module and video Analytics. The video encoder compresses the video, for example, using the algorithm for H.264. Module video Analytics generates the metadata that describes the objects and the situation on the video. Module ranking data implements this Method of ranking video data, namely implements 4 steps: (a) separates the compressed video fragments on the basis of video Analytics; b) calculates the characteristics of each fragment;) assesses the priorities of each piece of evidence-based; d) ranks (sorts) all fragments according to the estimates of priority. Ranked video comes users (operator) and are recorded in the repository.

The invention can be used in the composition of a local or geographically-distributed video surveillance system. Module ranking of video data may be: a) built-in video transmitter (network camera or network video server); b) be on the receiver video (on the server)is distributed between the transmitter and receiver video.

Way ranked transfer be implemented as a software modules and integrated into such components of the surveillance system, as a network camera, network video server, network video recorder (NVR), video management system (VMS), and a system of online video surveillance. In addition, these software modules can be run on workstations and dedicated servers in the center of data processing (CDP) or cloud hosting provider.

Priority evaluation of the fragment of the video, according to the present Invention may be used for the highlighting (3-4) and ranking fragments at the workplace (operator), video surveillance systems with the aim of presenting the user with the most important fragments as a matter of priority (Fig.4). For example, the disturbing video footage can be displayed and sorted in the descending order of priority evaluation. This significantly increases the productivity of the user (the operator), video surveillance systems by focusing his attention on the most important objects and situations (Fig.4).

Priority evaluation can display not only with the help of colors, and other graphic indicators, such as numbers, columns, lines, circles, that is, similarly as at home devices show the strength or the battery's charge.

User interface that displays ranked video content that can be implemented with the help of specialized software, or using web technologies such as HTML5 or Flash.

As an estimation of priority fragment of video data can be used for storage management video. For the organization of the loop video in the store there is a task to delete old fragments. According to the Invention, removing fragments can be made according to the assessment of their priorities: first of all will be deleted least priority fragments. Storage time of each video segment can be defined by the formula t=t 0 q, where t 0 - basic (maximum) time of data storage in the repository, for example, 30 days, q - priority fragment in the range [0,1]. Thus, the present invention provides efficient use of storage space by storing the most important information.

The invention can be used for a variety of transmission modes video:) continuous fixed speed (transfer the maximum amount of video data in the dedicated bandwidth); b) a continuous variable speed (transfer priority of video data as they become available); in the batch (at the user's request and upon connection between the source and receiver of video).

The invention allows to pass the most important video data after a temporary break the connection between the source and receiver video.

The invention can be used for a ranked transmission of alarm frames with optional downloading the video corresponding to the troubling frame, at the user's request or in delayed mode (e.g. at night when the load on the communication channel is reduced).

How to rank video data can be used not only for a living (conveying) the video coming from the cameras in real time, but on the archived video-recorded in the repository (post processing).

How to rank video can be used in video surveillance systems, built with the use of standard and/or recommendations of the Open forum of the interface of network video (Open Network Video Interface Forum (ONVIF, www.onvif.org) or Alliance compatibility physical security (Physical Security Interoperability Alliance, PSIA, psiaalliance.org). In particular, priority may be given to metadata, messages and/or events in accordance with the recommendations of the ONVIF and/or PSIA. Fragment of the video data can be transmitted in accordance with recommendations of the forum ONVIF and/or PSIA.

Fragments of the video data can also be transmitted in the form of files, for example, in the popular ones like TS, M2TS, MKV, OGV, MP4, AVI and other

How to rank video data can be used for the implementation of the two strategies: (a) processing of all video or events with priority level and above with minimum use of resources (operators, channels of communication, storage); b) handling maximum amount of video data or events, with highest priority given restrictions on resources. In practice, often used hybrid strategy, which combines the above options (a) and (b), the load on the resources may vary on a course of alarm situations within the established limits.

1. How to rank video, including obtaining video with at least one camera (or sensor and transfer sorted video channels of communication, at least one user and/or, at least, in one store, which is characterized by the fact that in the beginning of the received video source select excerpts, appropriate, at least one object and/or event, then calculate the characteristics of each fragment that affect the valuation of priority fragment and/or used by search snippets in the repository, then assess the priority of each fragment taking into account its features, then sorted fragments in accordance with the priority of each of them and passed in through the channels of communication, at least one user and/or, at least, in one store received priority queue fragments or one piece with the highest priority.

2. The method according to claim 1, characterized in that the fragments isolated by a motion detector.

3. The method according to claim 1, characterized in that the fragments isolated by analytic integrated in the network camera or video server.

4. The method according to claim 1, characterized in that the fragments isolated by the server video Analytics.

5. The method according to claim 1, characterized in that the priority evaluation of each fragment is carried out with a help of regression function.

6. The method according to claim 5, which is characterized by the fact that the construction of regression functions carried out on the basis of statistics of user requests.

7. The method according to claim 1, characterized in that the priority evaluation of each fragment is carried out with a help of the statistical classifier.

8. The method according to claim 7, characterized in that the training of the statistical classifier carried out on the basis of statistics of user requests.

9. The method according to claim 1, characterized in that the priority evaluation of each fragment is carried out with a help of rules defined by the user.

10. The method according to claim 1, characterized in that the priority evaluation of each fragment produced with the current user's requests.

11. The method according to claim 1, characterized in that the priority evaluation of each fragment the user manually sets.

12. The method according to claim 1, characterized in that the priority evaluation of each fragment is produced on the basis of priority of the camera, which is the source of the fragment of the video.

13. The method according to claim 1, characterized in that, for assessing the priority of each fragment use one sign, calculated , such as the accuracy of the detection of an object or situation.

14. The method according to claim 1, characterized in that, for assessing the priority of the use value of the priority, pre-defined by the user in the table for each type of objects or situations.

15. The method according to claim 1, characterized in that, for assessing the priority of the fragment use the sum or maximum estimates of the priorities of individual objects and/or conditions contained in this fragment.

16. The method according to claim 1, characterized in that, for assessing the priority use data from external sensors.

17. The method according to claim 1, characterized in that in a fragment of a video is a sequence of frames.

18. The method according to claim 1, characterized in that in a fragment of a video image is an alarming frame or part of the image.

19. The method according to claim 1, characterized in that the operator is the surveillance system.

20. The method according to claim 1, characterized in that fragment passed in the repository depends on the priority of the fragment.

21. The method according to claim 1, characterized in that the transfer of video data via the communication channels with limited bandwidth primarily transmit fragments with the highest priority.

22. The method according to claim 1, characterized in that the priority queue display the user interface in the form of a list obtained fragments and list legend fragments.

25. The method according to claim 1, characterized in that each fragment of a priority queue is passed in a separate file.

26. The method according to claim 1, characterized in that the priority queue are dispersed among several users.