The Surveillance Event Detection (SED) evaluation is a track within of the TRECVid Evaluation. The goal of the evaluation track is to support the development of technologies to detect visual events (people engaged in particular activities) in a large collection of streaming video data. The evaluation is implemented using 150 hours of multi-camera airport surveillance domain data collected by the Home Office Scientific Development Branch (HOSDB), which was ground-truthed by the University of Pennsylvania Linguistic Data Consortium.

The 2010 SED evaluation is the 3rd evaluation of event detection technologies. The evaluation task will be similar in structure to previous evaluations, which were held in 2008 and 2009, however there will be some minor changes which includes:

• The test copora used for the 2009 evaluation will be re-used for the 2010 evaluation.
• The less frequent events of interest, ElevatorNoEntry, OpposingFlow, and TakePicture, will not be included in the evaluation to focus research on improving performance on events with many observations.
• A new sub-task: SED Clip Filtering will be introduced as a low entry barrier task.

We have a mailing list for announcements/discussion -- contact jonathan.fiscus@nist.gov to subscribe.

Evaluation Planning

The evaluation plan is evolving and the following resources are available:

• Evaluation Plans:

2010 SED Evaluation Plan - V1 V1 is essentially unchanged from the 2009-V3 evaluation plan.

• Event Definitions:
  • An annotation guidelines document TRECVid09_Event_Annotation_Guidelines_v1.0.pdf is available.
  • The following paper EventDet-ViPER-format-v4.pdf documents the XML Schema Definition (XSD) proposed for a ViPER XML Representation of Events.
  • The following TrecVid08_XSD_v2.tar includes the XSD and a sample XML file for events.
• Experiment Control Files:
  • The following TRECVid08_ecf_v2.tar includes the XSD and a sample XML file for Experiment Control Files (ECFs).

Software

Evaluation submissions will be scored using the Framework for Detection Evaluations (F4DE) toolkit downloadable from the MIG Tools Web Page.

Dry Run

The dry run period for the TRECVid Event Detection Evaluation will run until July 21, 2010. Dry run submissions will be accepted at any time during the period. The procedure for participating is as follows:

• Sign up as a TRECVid Participant as specified on the TRECVid 2010 web site.
• Obtain the data sets by completing the licensing agreement as specifiec below. The dry run will use part of the development corpus.
• Download the Experiment Control File (ECF): expt_2009_retroED_DEV09_ENG_s-camera_NIST_2.xml
• Run experimental system(s) on the video intervals specified in the ECF
• Package system outputs in ViPER XML format and make sure they validate against the TrecVid08.xsd: TrecVid08_XSD_v2.tar -- see the F4DE software download (above) for a validation script
• Prepare a tar/gzip of all outputs as described in the evaluation plan and submit the archive via FTP
• Notify jonathan.fiscus@nist.gov, martial.michel@nist.gov
• Submissions are due 11:59PM Eastern Time (UTC-4), July 21, 2010
• Results of the dry run will be returned by July 28, 2010 (approx.)

Formal Evaluation

The formal evaluation for the TRECVid Event Detection Evaluation has now commenced. The procedure for participating is as follows:

• Complete the dry run as specified above.
• Download the Experiment Control File (ECF): expt_2009_retroED_EVAL09_ENG_s-camera_NIST_3.xml
• Run experimental system(s) on the video intervals specified in the ECF
• Package system outputs in ViPER XML format and make sure they validate against the TrecVid08.xsd: TrecVid08_XSD_v2.tar -- see the F4DE software download (above) for a validation script
• Prepare a tar/gzip of all outputs as described in the evaluation plan and submit the archive via FTP
• Notify jonathan.fiscus@nist.gov, martial.michel@nist.gov
• Submissions are due as specified in the TRECVid 2010 schedule

Data Resources

• Please contact jonathan.fiscus@nist.gov with questions.

Licensing/Participation Agreements The procedure for licensing the data is as follows: 1. Apply to participate in the TRECVid event detection evaluation by completing the registration forms (NOTE: The deadline has passed for this step). 2. The development data for 2010 evaluation consists of the 2008 Event Detection Training and Testing sets. Follow the instructions in the guidelines www-nlpir.nist.gov/projects/tv2010/tv2010.html#licenses. NIST will acknowledge reciept and then you should proceed to next step. 3. The evaluation set for the 2010 Event Detection evaluation will be the UK Home Office Scientific Development Branch's (HOSDB) i-LIDS MCTTR dataset. This is the same evaluation corpus as used for the 2009 evaluation. The MCTTR data set is available at no cost. To obtain the corpus, participants must complete HOSDB's "MCT application form" and relevant end-user license agreement found below and submit as indicated on the forms. Upon completion, HOSDB will give participants a unique username and password for accessing online resources. MCT Application Form End User License Agreement (EULA) for Commercial Companies End User License Agreement (EULA) for Academic Institutions. Note: Please be sure on the application form to indicate that you are applying in order to participate in the TRECVid evaluation. Development Data Acquisition Download the data by either going through the TRECVid data servers (check server lists in the active participants area) or by shipping a hard drive (at least 250 GB) to: Martial Michel 100 Bureau Drive, Mail Stop 8940 Gaithersburg, MD 20899 USA Annotations for the Development corpora can by found on the data page. Test Data Acquisition There are two ways to obtain the test data: (a) via download from NIST, (b) via hard drive sent to the UK. Download from NIST: To obtain the test data (i.e., the 2009 i-LIDS MCTTR dataset) from NIST, download the MPEG-2 video using an HOSDB-assigned username and password from the URL ir.nist.gov/tv2009/iLIDS/Video/Eval09/. The MPEG-2 video is compatible, (same frame rate, resolution, and de-interlaced) with the Development data set. This MPEG-2 data is 150GB. Via hard drive sent to UK: To obtain the full MCTTR data from the UK you will need to send a 1TB hard drive to HOSDB in the United Kingdom according to the instructions in the MCT application form. The full MCTTR data set consists of the MPEG-2 files above and the original, interlaced video imagery collected from the cameras. In order to maintain compatibility with the 2008 eval, sites should use the MPEG-2 files.

Related

AVSS 2010 Challenge Evaluation

AVSS 2010 Home Page