Wednesday, November 16, 2011

LDC November 2011 Newsletter


New publications:



Spring 2012 LDC Data Scholarship Program

Applications are now being accepted through January 15, 2012 for the Spring 2012 LDC Data Scholarship program! The LDC Data Scholarship program provides university students with access to LDC data at no-cost. This program is open to students pursuing both undergraduate and graduate studies in an accredited college or university. LDC Data Scholarships are not restricted to any particular field of study; however, students must demonstrate a well-developed research agenda and a bona fide inability to pay. The selection process is highly competitive.

The application consists of two parts:

(1) Data Use Proposal. Applicants must submit a proposal describing their intended use of the data. The proposal must contain the applicant's name, university and field of study. The proposal should state which data the student plans to use and contain a description of their research project.

Applicants should consult the LDC Corpus Catalog for a complete list of data distributed by LDC. Due to certain restrictions, a handful of LDC corpora are restricted to members of the Consortium. Applicants are advised to select a maximum of one to two data sets; students may apply for additional data sets during the following cycle once they have completed processing of the initial data sets and publish or present work in some juried venue.

(2) Letter of Support. Applicants must submit one letter of support from their thesis adviser or department chair. The letter must confirm that the department or university lacks the funding to pay the full Non-member Fee for the data and verify the student's need for data.

For further information on application materials and program rules, please visit the LDC Data Scholarship page. Students can email their applications to the LDC Data Scholarship program. Decisions will be sent by email from the same address.
Invitation to Join for Membership Year (MY) 2012
Membership Year (MY) 2012, our 20th Anniversary Year, is open for joining! We would like to invite all current and previous members of LDC to renew their membership as well as welcome new organizations to join the consortium. For MY2012, LDC is pleased to maintain membership fees at last year’s rates – membership fees will not increase. Additionally, LDC will extend discounts on membership fees to members who keep their membership current and who join early in the year.

The details of our early renewal discounts for MY2012 are as follows:
· Organizations who joined for MY2011 will receive a 5% discount when renewing. This discount will apply throughout 2012, regardless of time of renewal. MY2011 members renewing before March 1, 2012 will receive an additional 5% discount, for a total 10% discount off the membership fee.
· New members as well as organizations who did not join for MY2011, but who held membership in any of the previous MYs (1993-2010), will also be eligible for a 5% discount provided that they join/renew before March 1, 2012.
The following table provides exact pricing information.

MY2012 Fee
MY2012 Fee
with 5% Discount*
MY2012 Fee
with 10% Discount**
Not-for-Profit /US Government




Standard
US$2400
US$2280
US$2160

Subscription
US$3850
US$3658
US$3465
For-Profit




Standard
US$24000
US$22800
US$21600

Subscription
US$27500
US$26125
US$24750
Publications for MY2012 are still being planned; here are the working titles of data sets we intend to provide:
· ARRAU 1.2 (Anaphor Resolution and Underspecification)
· TORGO Dysarthic Speech
· Arabic Treebank BN (broadcast news)
· GALE data – all phases and tasks
· Digital Archive of Southern Speech
· Chinese Dependency Treebank
In addition to receiving new publications, current year members of the LDC also enjoy the benefit of licensing older data at reduced costs; current year for-profit members may use most data for commercial applications.

This past year, LDC members who joined early or kept their membership current saved almost US$70,000 on membership fees. Be sure to keep an eye on your mail as all previous and current LDC members will be sent an invitation to join letter and renewal invoice for MY2012. Renew early for MY2012 to save today!
Why Become an LDC Member?
LDC is offering early renewal discounts on membership fees for Membership Year 2012 making now a good time to consider joining or renewing membership. LDC membership has the following advantages:


· LDC membership provides cost-effective access to an extensive and growing catalog that spans 20 years and includes over 500 multilingual speech, text, and video resources. Even if your organization only needs a few datasets from a given membership year, membership is often the most economical way to obtain current corpora. Additionally, the generous discounts that member organizations receive on older corpora reduce the cost of acquiring such datasets.

· All members enjoy unlimited use of LDC data within their organizations. For universities, there is no difference in cost between a departmental membership and one that is university-wide. Departments can therefore combine resources and establish one LDC membership for use by the entire university community. Likewise, for-profit members with multiple branches can maintain one membership for use by their entire organization.
For-profit organizations are reminded that an LDC membership is a pre-requisite for obtaining a commercial license to almost all LDC databases. Non-member organizations, including non-member for-profit organizations, cannot use LDC data to develop or test products for commercialization, nor can they use LDC data in any commercial product or for any commercial purpose. LDC data users should consult corpus-specific license agreements for limitations, including commercial restrictions, on the use of certain corpora. In the case of a small group of corpora, commercial licenses must be obtained separately from the owners of the data.
LDC to Close for Thanksgiving Break
LDC would like to inform our customers that we will be closed on Thursday, November 24, 2011 and Friday, November 25, 2011 in observance of the US Thanksgiving Holiday. Our offices will reopen on Monday, November 28, 2011.
New Publications
(1) 2006 NIST Speaker Recognition Evaluation Training Set was developed by LDC and NIST (National Institute of Standards and Technology). It contains 595 hours of conversational telephone speech in English, Arabic, Bengali, Chinese, Hindi, Korean, Russian, Thai and Urdu and associated English transcripts used as training data in the NIST-sponsored 2006 Speaker Recognition Evaluation (SRE). The ongoing series of SRE yearly evaluations conducted by NIST are intended to be of interest to researchers working on the general problem of text independent speaker recognition.
The task of the 2006 SRE evaluation was speaker detection, that is, to determine whether a specified speaker is speaking during a given segment of conversational telephone speech. The task was divided into 15 distinct and separate tests involving one of five training conditions and one of four test conditions. Further information about the test conditions and additional documentation is available at the NIST web site for the 2006 SRE and within the 2006 SRE Evaluation Plan.
The speech data in this release was collected by LDC as part of the Mixer project, in particular Mixer Phases 1, 2 and 3. The Mixer project supports the development of robust speaker recognition technology by providing carefully collected and audited speech from a large pool of speakers recorded simultaneously across numerous microphones and in different communicative situations and/or in multiple languages. The data is mostly English speech, but includes some speech in the above languages.
The telephone speech segments are multi-channel data collected simultaneously from a number of auxiliary microphones. The files are organized into three types: two-channel excerpts of approximately 10 seconds, two-channel conversations of approximately 5 minutes and summed-channel conversations also of approximately 5 minutes.

English language transcripts in .ctm format were produced using an automatic speech recognition (ASR) system. 2006 NIST Speaker Recognition Evaluation Training Set is distributed on seven DVDs. 2011 Subscription Members will automatically receive two copies of this corpus. 2011 Standard Members may request a copy as part of their 16 free membership corpora. Non-members may license this data for US$2000.

*
(2) 2006 NIST/USF Evaluation Resources for the VACE Program - Meeting Data Test Set Part 2 was developed by researchers at the Department of Computer Science and Engineering, University of South Florida (USF), Tampa, Florida and the Multimodal Information Group at the National Institute of Standards and Technology (NIST). It contains approximately twenty hours of meeting room video data collected in 2005 and 2006 and annotated for the VACE (Video Analysis and Content Extraction) 2006 face and person tracking tasks.
The VACE program was established to develop novel algorithms for automatic video content extraction, multi-modal fusion, and event understanding. During VACE Phases I and II, the program made significant progress in the automated detection and tracking of moving objects including faces, hands, people, vehicles and text in four primary video domains: broadcast news, meetings, street surveillance, and unmanned aerial vehicle motion imagery. Initial results were also obtained on automatic analysis of human activities and understanding of video sequences.
Three performance evaluations were conducted under the auspices of the VACE program between 2004 and 2007. In 2006, the VACE program and the European Union's Computers in the Human Interaction Loop (CHIL) collaborated to hold the Classification of Events, Activities and Relationships (CLEAR) Evaluation. This was an international effort to evaluate systems designed to analyze people, their identities, activities, interactions and relationships in human-human interaction scenarios, as well as related scenarios. The VACE program contributed the evaluation infrastructure (e.g., data, scoring, tools) for a specific set of tasks, and the CHIL consortium, coordinated by the Karlsruhe Institute of Technology, contributed a separate set of evaluation infrastructure. To the extent possible, the VACE and CHIL programs harmonized their evaluation protocols and metrics.
The meeting room data used for the 2006 test set was collected by the following sites in 2005 and 2006: Carnegie Mellon University (USA), University of Edinburgh (Scotland), IDIAP Research Institute (Switzerland), NIST (USA), Netherlands Organization for Applied Scientific Research (Netherlands) and Virginia Polytechnic Institute and State University (USA).
2006 NIST/USF Evaluation Resources for the VACE Program - Meeting Data Test Set Part 2 is distributed on ten DVDs. 2011 Subscription Members will automatically receive two copies of this corpus. 2011 Standard Members may request a copy as part of their 16 free membership corpora. Non-members may license this data for US$2500.
*
(3) Chinese Gigaword Fifth Edition was produced by LDC. It is a comprehensive archive of newswire text data that has been acquired from Chinese news sources by LDC at the University of Pennsylvania. Chinese Gigaword Fifth Edition includes all of the content of the fourth edition of Chinese Gigaword (LDC2009T27) plus new data covering the period from January 2009 through December 2010.
Eight distinct sources of Chinese newswire are represented here:
  • Agence France Presse(afp_cmn)
  • Central News Agency, Taiwan(cna_cmn)
  • Central News Service(cns_cmn)
  • Guangming Daily(gmw_cmn)
  • People's Daily(pda_cmn)
  • People's Liberation Army Daily(pla_cmn)
  • Xinhua News Agency(xin_cmn)
  • Zaobao Newspaper(zbn_cmn)
The seven-letter codes in the parentheses above are used for the directory names and data files for each source. Articles covering the period from January 2009 through December 2010 have been added to the Agence France Presse, Central News Agency (CNA), Central News Service, Guangming Daily, People's Liberation Army Daily and Xinhua News Agency data sets. The data from People's Daily covers the period from late June 2009 through December 2010. No new data from Zaobao has been added.
Chinese Gigaword Fifth Edition is distributed on one DVD-ROM. 2011 Subscription Members will automatically receive two copies of this corpus. 2011 Standard Members may request a copy as part of their 16 free membership corpora. Non-members may license this data for US$6000.

Monday, October 31, 2011

LDC October 2011 Newsletter

New publications:


Fall 2011 LDC Data Scholarships recipients
LDC is pleased to announce the student recipients of the Fall 2011 LDC Data Scholarship program! The LDC Data Scholarship program provides university students with access to LDC data at no-cost. Data scholarships are offered twice a year to correspond to the Fall and Spring semesters. Students are asked to complete an application which consists of a data use proposal and letter of support from their academic adviser.

LDC received many strong applications from students attending universities across the globe. We've reviewed all the applications, and after careful consideration, we have selected four scholarship recipients! These students will receive no-cost copies of LDC data:
Haris B C - Indian Institute of Technology Guwahati (India), Electronics & Electrical Engineering. Haris has been awarded a copy of 2005 NIST Speaker Recognition Evaluation Training Data (LDC2011S01) and 2005 NIST Speaker Recognition Evaluation Test Data (LDC2011S04) to evaluate the performance of a sparse representation speaker verification system.

Friðjón Guðjohnsen - Reykjavik University (Iceland), Computer Science. Friðjón has been awarded a copy of Treebank-3 (LDC99T42) to be used in the development of tagging methods to improve the accuracy of tagging Icelandic texts.

Leili Javadpour - Louisiana State University (USA), Engineering Science. Leili has been awarded a copy of BBN Pronoun Coreference and Entity Type Corpus (LDC2005T33) and Message Understanding Conference (MUC) 7 (LDC2001T02) for her work in pronominal anaphora resolution.

Jad Makhlouta - American University of Beirut (Lebanon), Electrical and Computer Engineering. Jad has been awarded a copy of LDC Standard Arabic Morphological Analyzer (SAMA) Version 3.1 (LDC2010L01) for his work in Arabic text mining.


LDC data - now on Blu-ray!
LDC is pleased to announce that the Blu-ray revolution has come to linguistic data! We are now offering select databases on Blu-ray Disc (BD). With BD, we'll be able to distribute some of our larger databases using a smaller number of discs. BDs also have the potential to be read more quickly than DVDs, which means faster access to data. To introduce our Blu-ray option, we would like to announce that the following databases will now be distributed on BD in addition to DVD-ROM:

LDC2011T07 English Gigaword Fifth Edition, now available on 1 Blu-ray Disc

LDC2006T13 Web 1T 5-gram Version 1, now available on 2 Blu-ray Discs

Organizations with licenses to English Gigaword Fifth Edition will be given the opportunity to swap their DVDs for BDs. New licensees for Web 1T 5-gram have the option to select BD or DVD media.

We expect to extend the BD option over time to other corpora in the catalog and to new releases.

LDC at NWAV 2011
NWAV’s 40th Anniversary Conference will be hosted by Georgetown University from October 27-30 and LDC will be on-hand to celebrate! Please stop by the LDC exhibition at any point during the main conference and be sure to attend LDC’s pre-conference workshop on “Demographic Coding for Sociolinguistic Corpus Archive Preparation” from 4.00 – 6.00 pm on Thursday, October 27. This workshop will be hosted by LDC Executive Director Christopher Cieri and Malcah Yaeger-Dror of the University of Arizona. It has two stated goals:
1) to catalog the need for more detailed demographic categories based on field experience that other researchers can and should exploit both for their own immediate analyses, and to facilitate sharing among research groups, and
2) to encourage the use of a core set of demographic metadata coding options.
NWAV registration options can be found here. We hope to see you there! Please visit LDC’s Facebook page to follow our conference activities.
New publications
(1) 2008 NIST Speaker Recognition Evaluation Test Set contains 942 hours of multilingual telephone speech and English interview speech along with transcripts and other materials used as test data in the 2008 NIST Speaker Recognition Evaluation (SRE).
NIST SRE is part of an ongoing series of evaluations conducted by NIST. They are intended to be of interest to all researchers working on the general problem of text independent speaker recognition. The 2008 evaluation was distinguished from prior evaluations, in particular those in 2005 and 2006, by including not only conversational telephone speech data but also conversational speech data of comparable duration recorded over a microphone channel involving an interview scenario.
LDC previously released the 2008 NIST SRE Training Set in two parts as LDC2011S05 and LDC2011S07.
The speech data in this release was collected in 2007 by LDC at its Human Subjects Data Collection Laboratories in Philadelphia and by the International Computer Science Institute (ICSI) at the University of California, Berkeley. This collection was part of the Mixer 5 project, which was designed to support the development of robust speaker recognition technology by providing carefully collected and audited speech from a large pool of speakers recorded simultaneously across numerous microphones and in different communicative situations and/or in multiple languages. Mixer participants were native English and bilingual English speakers. The telephone speech in this corpus is predominantly English, but also includes the above languages. All interview segments are in English. Telephone speech represents approximately 368 hours of the data, whereas microphone speech represents the other 574 hours.
English language transcripts in .cfm format were produced using an automatic speech recognition (ASR) system.
2008 NIST Speaker Recognition Evaluation Test Set is distributed on 9 DVD-ROM.2011 Subscription Members will automatically receive two copies of this corpus. 2011 Standard Members may request a copy as part of their 16 free membership corpora. Non-members may license this data for US$2000.

*
(2) Arabic Gigaword Fifth Edition is a comprehensive archive of newswire text data that has been acquired from Arabic news sources over several years by LDC. Arabic Gigaword Fifth Edition includes all of the content of the fourth edition of Arabic Gigaword (LDC2009T30) plus new data covering the period from January 1, 2009 through December 31, 2010.
Nine distinct sources of Arabic newswire are represented in this distribution:
Asharq Al-Awsat (aaw_arb)
Agence France Presse (afp_arb)
Al-Ahram (ahr_arb)
Assabah (asb_arb)
Al Hayat (hyt_arb)
An Nahar (nhr_arb)
Al-Quds Al-Arabi (qds_arb)
Ummah Press (umh_arb)
Xinhua News Agency (xin_arb)
The seven-character codes shown above represent both the directory names where the data files are found, and the 7-letter prefix that appears at the beginning of every file name. The 7-letter codes consist of the three-character source name IDs and the three-character language code ("arb") separated by an underscore ("_") character. The three-character language code conforms to the ISO 639-3 standard.
In addition to adding new data, the following updates were made:
Repeated documents in Asharq Al-Awsat data from 2008 were removed.
Document formatting and docid duplication problems were corrected in Agence France Presse data.
Significant duplication of content in 2007-2008 An Nahar data was detected, and the duplicated documents were removed.
Arabic Gigaword Fifth Edition is distributed on 1 DVD-ROM. 2011 Subscription Members will automatically receive two copies of this corpus. 2011 Standard Members may request a copy as part of their 16 free membership corpora. Non-members may license this data for US$6000.

*
(3) Spanish Gigaword Third Edition was produced by LDC. It is a comprehensive archive of Spanish newswire text data that has been acquired over several years by LDC. Spanish Gigaword Third Edition includes all of the content of the second edition (LDC2009T21) and adds data collected from January 1, 2009 through December 31, 2010.
The three distinct international sources of Spanish newswire in this edition, and the time spans of collection covered for each, are as follows:
Agence France-Presse, Spanish (afp_spa) May 1994 - Dec 2010
Associated Press, Spanish (apw_spa) Nov 1993 - Dec 2010
Xinhua News Agency, Spanish (xin_spa) Sep 2001 - Dec 2010
The seven-letter codes in the parentheses above include the three-character source name abbreviations and the three-character language code ("spa") separated by an underscore ("_") character. The three-letter language code conforms to LDC's internal convention based on the ISO 639-3 standard.
All text data are presented in SGML/XML form, using a very simple, minimal markup structure; all text consists of printable ASCII, whitespace, and printable code points in the "Latin1 Supplement" character table, as defined by both ISO-8859-1 and the Unicode Standard (ISO 10646) for the "accented" characters used in Spanish. The Supplement/accented characters are rendered using UTF-8 encoding.
Spanish Gigaword Third Edition is distributed on 1 DVD-ROM. 2011 Subscription Members will automatically receive two copies of this corpus. 2011 Standard Members may request a copy as part of their 16 free membership corpora. Non-members may license this data for US$4500.

Thursday, September 15, 2011

LDC September 2011 Newsletter

Cataloging the communication of Asian Elephants

New publications:

LDC2011V05



Cataloging the communication of Asian Elephants

LDC distributes a broad selection of databases, the majority of which are used for human language research and technology development. Our corpus catalog also includes the vocalizations of other animal species. We'd like to highlight the intriguing work behind one such animal communication corpus, Asian Elephant Vocalizations, LDC2010S05.

Asian Elephant Vocalizations contains audio recordings of vocalizations by Asian Elephants (Elephas maximus) in Uda Walawe National Park, Sri Lanka. The data was collected by Shermin de Silva as part of her doctoral thesis at the University of Pennsylvania. Recordings were made using a Fostex field recorder with a Sennheiser 'shot-gun' microphone. In addition, de Silva utilized a second dictation microphone that allows observers to narrate what's happening without talking over the elephant recording. The digital files were then downloaded and visualized using the Praat TextGrid Editor, a tool originally developed for studying human speech which has since been adopted by elephant researchers. With Praat, trained annotators are able to characterize call types and extract particular segments for later analysis.

Until recently, the majority of research on the behavior of wild elephants focused on one species - the African savannah elephant.
There has been comparatively less study of communication in Asian elephants, primarily because the habitat in which Asian elephants typically live makes them more difficult to study than African forest elephants. Asian and African elephants diverged from one another approximately six million years ago and evolved separately in very distinct environments. de Silva's work has shown that Asian elephants have highly dynamic social lives, that are markedly different from that of African elephants. Asian elephants tend to form smaller, fragmented groups on a day-to-day basis but maintain long-term pools of companions over many years. Because communication in elephants appears to be largely socially-motivated, differences in social behavior and ecology may also be a source of differences in their vocal behavior and repertoire.

de Silva and her colleagues study elephant communication as an opportunity to understand the evolution of social behavior and communication in a system that is very different from our own primate experience. Human language is only one manifestation of communication in the natural world. Perhaps this is why it is fitting to place animal vocalizations side-by-side with human speech in LDC's catalog. In this way, we can better understand how human language relates to the communicative capabilities of other species.


For further information on Shermin de Silva's current research at the
Elephant Forest and Environment Conservation Trust visit:

Web:
http://elephantresearch.net
Blog: http://elephantresearch.net/fieldnotes/

New Publications

(1) 2006 NIST/USF Evaluation Resources for the VACE Program - Meeting Data Test Set Part 1 was developed by researchers at the Department of Computer Science and Engineering, University of South Florida (USF), Tampa, Florida and the Multimodal Information Group at the National Institute of Standards and Technology (NIST). It contains approximately fifteen hours of meeting room video data collected in 2005 and 2006 and annotated for the VACE (Video Analysis and Content Extraction) 2006 face and person tracking tasks.

The VACE program was established to develop novel algorithms for automatic video content extraction, multi-modal fusion, and event understanding. During VACE Phases I and II, the program made significant progress in the automated detection and tracking of moving objects including faces, hands, people, vehicles and text in four primary video domains: broadcast news, meetings, street surveillance, and unmanned aerial vehicle motion imagery. Initial results were also obtained on automatic analysis of human activities and understanding of video sequences.

Three performance evaluations were conducted under the auspices of the VACE program between 2004 and 2007. In 2006, the VACE program and the European Union's Computers in the Human Interaction Loop (CHIL)CLassification of Events, Activities and Relationships (CLEAR) Evaluation. This was an international effort to evaluate systems designed to analyze people, their identities, activities, interactions and relationships in human-human interaction scenarios, as well as related scenarios. The VACE program contributed the evaluation infrastructure (e.g., data, scoring, tools) for a specific set of tasks, and the CHIL consortium, coordinated by the Karlsruhe Institute of Technology, contributed a separate set of evaluation infrastructure. collaborated to hold the

The meeting room data used for the 2006 test set was collected by the following sites in 2005 and 2006: Carnegie Mellon University (USA), University of Edinburgh (Scotland), IDIAP Research Institute (Switzerland), NIST (USA), Netherlands Organization for Applied Scientific Research (Netherlands) and Virginia Polytechnic Institute and State University (USA). Each site had its own independent camera setup, illuminations, viewpoints, people and topics. Most of the datasets included High-Definition (HD) recordings, but those were subsequently formatted to MPEG-2 for the evaluation.

2006 NIST/USF Evaluation Resources for the VACE Program - Meeting Data Test Set Part 1 is distributed on 9 DVD-ROM. 2011 Subscription Members will automatically receive two copies of this corpus. 2011 Standard Members may request a copy as part of their 16 free membership corpora. Non-members may license this data for $2500.

*

(2) 2008 NIST Speaker Recognition Evaluation Training Set Part 2 was developed by LDC and NIST (National Institute of Standards and Technology). It contains 950 hours of multilingual telephone speech and English interview speech along with transcripts and other materials used as training data in the 2008 NIST Speaker Recognition Evaluation (SRE). SRE is part of an ongoing series of evaluations conducted by NIST. These evaluations are an important contribution to the direction of research efforts and the calibration of technical capabilities. They are intended to be of interest to all researchers working on the general problem of text independent speaker recognition. To this end the evaluation is designed to be simple, to focus on core technology issues, to be fully supported, and to be accessible to those wishing to participate.

The 2008 evaluation was distinguished from prior evaluations, in particular those in 2005 and 2006, by including not only conversational telephone speech data but also conversational speech data of comparable duration recorded over a microphone channel involving an interview scenario.

The speech data in this release was collected in 2007 by LDC at its Human Subjects Data Collection Laboratories in Philadelphia and by the International Computer Science Institute (ICSI) at the University of California, Berkeley. This collection was part of the Mixer 5 project, which was designed to support the development of robust speaker recognition technology by providing carefully collected and audited speech from a large pool of speakers recorded simultaneously across numerous microphones and in different communicative situations and/or in multiple languages. Mixer participants were native English speakers and bilingual English speakers. The telephone speech in this corpus is predominately English; all interview segments are in English. Telephone speech represents approximately 523 hours of the data, and microphone speech represents the other 427 hours.

The telephone speech segments include summed-channel excerpts in the range of 5 minutes from longer original conversations. The interview material includes single channel conversation interview segments of at least 8 minutes from a longer interview session. English language transcripts were produced using an automatic speech recognition (ASR) system.

2008 NIST Speaker Recognition Evaluation Training Set Part 2 is distributed on 7 DVD-ROM. 2011 Subscription Members will automatically receive two copies of this corpus. 2011 Standard Members may request a copy as part of their 16 free membership corpora. Non-members may license this data for $2000.

*

(3) French Gigaword Third Edition is a comprehensive archive of newswire text data that has been acquired over several years by LDC. This third edition updates French Gigaword Second Edition (LDC2009T28) and adds material collected from January 1, 2009 through December 31, 2010.

The two distinct international sources of French newswire in this edition, and the time spans of collection covered for each, are as follows:

  • Agence France-Presse (afp_fre) May 1994 - Dec. 2010
  • Associated Press French Service (apw_fre) Nov. 1994 - Dec. 2010

All text data are presented in SGML form, using a very simple, minimal markup structure; all text consists of printable ASCII, white space, and printable code points in the "Latin1 Supplement" character table, as defined by the Unicode Standard (ISO 10646) for the "accented" characters used in French. The Supplement/accented characters are presented in UTF-8 encoding.

The overall totals for each source are summarized below. Note that the "Totl-MB" numbers show the amount of data when the files are uncompressed (i.e. approximately 15 gigabytes, total); the "Gzip-MB" column shows totals for compressed file sizes as stored on the DVD-ROM; the "K-wrds" numbers are simply the number of white space-separated tokens (of all types) after all SGML tags are eliminated.

Source

#Files

Gzip-MB

Totl-MB

K-wrds

#DOCs

afp_fre

195

1503

4255

641381

2356888

apw_fre

194

489

1446

221470

801075

TOTAL

389

1992

5701

862851

3157963

French Gigaword Third Edition is distributed on 1 DVD-ROM. 2011 Subscription Members will automatically receive two copies of this corpus. 2011 Standard Members may request a copy as part of their 16 free membership corpora. Non-members may license this data for US$4500.

Monday, August 15, 2011

LDC August 2011 Newsletter

Fall 2011 LDC Data Scholarship Program

Checking in with previous LDC Data Scholarship recipients

Weizmann Institute students are introduced to LDC data

LDC Exhibiting at Interspeech 2011, Florence Italy

New publications:

LDC2011S06

- 2005 Spring NIST Rich Transcription (RT-05S) Evaluation Set -

LDC2011S05

- 2008 NIST Speaker Recognition Evaluation Training Set Part 1 -

LDC2011T09

- Arabic Treebank: Part 2 v 3.1 -


Fall 2011 LDC Data Scholarship Program


Applications are now being accepted through September 15, 2011 for the Fall 2011 LDC Data Scholarship program! The LDC Data Scholarship program provides university students with access to LDC data at no-cost. During the previous two cycles of the program, LDC has awarded no-cost copies of LDC data valued at over US$25,000.

This program is open to students pursuing both undergraduate and graduate studies in an accredited college or university. LDC Data Scholarships are not restricted to any particular field of study; however, students must demonstrate a well-developed research agenda and a bona fide inability to pay. The selection process is highly competitive.


The application consists of two parts:


(1)
Data Use Proposal. Applicants must submit a proposal describing their intended use of the data. The proposal must contain the applicant's name, university, and field of study. The proposal should state which data the student plans to use and contain a description of their research project.

Applicants should consult the
LDC Corpus Catalog for a complete list of data distributed by LDC. Due to certain restrictions, a handful of LDC corpora are restricted to members of the Consortium. Applicants are advised to select a maximum of one to two datasets; students may apply for additional datasets during the following cycle once they have completed processing of the initial datasets and publish or present work in some juried venue.

(2)
Letter of Support. Applicants must submit one letter of support from their thesis adviser or department chair. The letter must confirm that the department or university lacks the funding to pay the full Non-member Fee for the data and verify the student's need for data.

For further information on application materials and program rules, please visit the
LDC Data Scholarship page.

Students can email their applications to the
LDC Data Scholarship program. Decisions will be sent by email from the same address.

The deadline for the Fall 2011 program cycle is September 15, 2011.

Checking in with previous LDC Data Scholarship recipients


LDC introduced the Data Scholarship program during the Fall 2010 semester. Since that time, more than fifteen individual students and student research groups have been awarded no-cost copies of LDC data for their research endeavors. Here is an update on the work of a few of our student recipients:

  • Zachary Brooks - University of Arizona (USA), PhD Candidate, Second Language Acquisition and Teaching. Zachary and his research group were awarded a copy of ECI Multilingual Text (LDC94T5) for research in eye movement tracking by native and non-natives readers. Zachary used the ECI Multilingual Text data to test how second language readers process high and low frequency words in German. The results thus far show that processing a low frequency word can make it harder to process words that come next. The group's bilingual reading processes research is ongoing and Zachary anticipates the need to utilize additional speech and text corpora for future work.
  • Benjamin Martinez Elizalde - Monterrey Institute of Technology and Superior Studies, ITESM (Mexico), graduate student, Computer Science. Benjamin was awarded a copy of Switchboard-1 Release 2 (LDC97S62) and 2002 NIST Speaker Recognition Evaluation (SRE) (LDC2004S04) to support his research in speaker verification modeling. Benjamin's group has prepared a robust Universal Background Model (UBM) and will use the Switchboard and 2002 NIST SRE data to run enrollment and test experiments once a lower baseline is achieved. The Switchboard and SRE data will also be used to prepare the system for the 2012 NIST SRE.
  • Xiaohui Huang - Harbin Institute of Technology (China), Shenzhen Graduate School. Xiaohui and his research group were awarded a copy of TDT5 Topics and Annotations (LDC2006T19) for his work in topic detection and tracking for large-scale web data. Xiaohui extracted 607 documents from TDT5 Multilingual Text (LDC2006T18) and designed a new clustering approach for this data set. TDT5 Topics and Annotations (LDC2006T19 ) was used to label for measuring the precision of clustering. Xiaohui next compared his clustering approach with other text clustering approaches such as k-means and agglomerative hierarchical clustering and was able to achieve good performance. Since his group's method has been validated on small test data sets, next they will look to validate the system using larger text databases and time-series databases.
  • Muhua Zhu - Northeastern University (China), graduate student, Natural Language Processing. Muhua was awarded a copy of Chinese Treebank (CTB) 7.0 (LDC2010T07) to support the development of a high-accuracy Chinese parser. Currently, Muhua is writing a survey paper on Chinese syntactic parsing which studies the performance of different parsing models on the versions of LDC's CTBs. Muhua had expected that parsing accuracy would increase with the additional data from CTB7.0, but accuracy decreased in some instances perhaps because of the inclusion of web text in CTB 7.0. Muhua next plans to use re-ranking methods for syntactic parsing and to extract a Combinatory Categorial Grammar bank (CCG bank) from CTB7.0.


We would like to thanks these students for providing an update on their research. Stay tuned for further reports from other data scholarship recipients.

Weizmann Institute students are introduced to LDC data


LDC data was featured in an introductory speech recognition course at the Weizmann Institute of Science in Rehovot, Israel. Visiting professor, Karen Livescu, of Toyota Technological Institute at Chicago and University of Chicago, Department of Computer Science used several LDC corpora, including CSR-I (WSJ0) Complete (LDC93S6A), Switchboard-1 Release 2 (LDC97S62), TIDIGITS (LDC93S10), and TIMIT Acoustic-Phonetic Continuous Speech Corpus (LDC93S1) for homework and term projects, with a few examples shown during in-class demonstration.

The students enrolled in the course were computer science and mathematics graduate students and all were new to automatic speech recognition (ASR). They had backgrounds in probability, but no significant experience with the probabilistic models used in ASR, such as hidden Markov models and Gaussian mixtures. Livescu provided baseline recognizers that the students could modify, so that even beginning students could focus on specific components, while using real data with results in the literature to compare against.


Since the students were provided with real data that the research community actively uses, students were motivated by the potential for 'real' results if their projects went as planned. As Livescu noted, 'while starting out in ASR from scratch is very difficult, the availability of toolkits and LDC data makes it possible for students in an introductory class to do productive research quite quickly'.


Many thanks to Karen Livescu for sharing an example of how LDC data can be used for teaching purposes.

LDC Exhibiting at Interspeech 2011, Florence Italy

LDC is returning to Europe to participate in Interspeech 2011. The conference will be held from August 28-31 at the Firenze Fiera, conveniently located near the Stazione di Santa Maria Novella. Please stop by LDC’s exhibition booth to say hello and learn more about current happenings at the Consortium.

Interspeech 2011’s theme is ‘Speech Science and Technology for Real Life’. You may learn more about the conference here.

The main conference will feature keynotes on the following topics:

Speaking More Like You: Entrainment in Conversational Speech, Prof. Julia Hirschberg

Neural Representations of Word Meanings, Prof. Tom Mitchell

Honest Signals, Prof. Sandy Pentland

Conference organizers have also scheduled a roundtable discussion for August 31st on ‘Future and Applications of Speech and Language Technologies for the Good Health of Society’ which will be led by Profs. Gabriele Miceli, Björn Granström and Hiroshi Ishiguro.

You are encouraged to keep track of LDC’s Interspeech preparations on our Facebook page. We hope to see you there!

New Publications

(1) 2005 Spring NIST Rich Transcription (RT-05S) Conference Meeting Evaluation Set was developed by LDC and the National Institute of Standards and Technology (NIST). It contains approximately 78 hours of English meeting speech, reference transcripts and other material used in the RT Spring 2005 evaluation. Rich Transcription (RT) is broadly defined as a fusion of speech-to-text (STT) technology and metadata extraction technologies providing the bases for the generation of more usable transcriptions of human-human speech in meetings.

RT-05S included the following tasks in the meeting domain:

Speech-To-Text (STT) - convert spoken words into streams of text

Speaker Diarization (SPKR) - find the segments of time within a meeting in which each meeting participant is talking

Speech Activity Detection (SAD) - detect when someone in a meeting space is talking

Further information about the evaluation is available on the RT-05 Spring Evaluation Website.

The data in this release consists of portions of meeting speech collected between 2001 and 2005 by the IDIAP Research Institute's Augmented Multi-Party Interaction project (AMI), Martigny, Switzerland; International Computer Science Institute (ICSI) at University of California, Berkeley; Interactive Systems Laboratories (ISL) at Carnegie Mellon University (CMU), Pittsburgh, PA; NIST; and Virginia Polytechnic Institute and State University (VT), Blacksburg, VA. Each meeting excerpt contains a head-mic recording for each subject and one or more distant microphone recordings.

Reference transcripts for the evaluation excerpts were prepared by LDC according to its Meeting Recording Careful Transcription Guidelines. Those specifications are designed to provide an accurate, verbatim (word-for-word) transcription, time-aligned with the audio file and including the identification of additional audio and speech signals with special mark-up.

2005 Spring NIST Rich Transcription (RT-05S) Conference Meeting Evaluation Set is distributed on 3 DVD-ROM. 2011 Subscription Members will automatically receive two copies of this corpus. 2011 Standard Members may request a copy as part of their 16 free membership corpora. Non-members may license this data for $2250.

*

(2) 2008 NIST Speaker Recognition Evaluation Training Set Part 1 was developed by LDC and the National Institute of Standards and Technology (NIST). It contains 640 hours of multilingual telephone speech and English interview speech along with transcripts and other materials used as training data in the 2008 NIST Speaker Recognition Evaluation (SRE).

SRE is part of an ongoing series of evaluations conducted by NIST. These evaluations are an important contribution to the direction of research efforts and the calibration of technical capabilities. They are intended to be of interest to all researchers working on the general problem of text independent speaker recognition.

The 2008 evaluation was distinguished from prior evaluations, in particular those in 2005 and 2006, by including not only conversational telephone speech data but also conversational speech data of comparable duration recorded over a microphone channel involving an interview scenario.

The speech data in this release was collected in 2007 by LDC at its Human Subjects Data Collection Laboratories in Philadelphia and by the International Computer Science InstituteMixer 5 project, which was designed to support the development of robust speaker recognition technology by providing carefully collected and audited speech from a large pool of speakers recorded simultaneously across numerous microphones and in different communicative situations and/or in multiple languages. Mixer participants were native English and bilingual English speakers. The telephone speech in this corpus is predominately English; all interview segments are in English. Telephone speech represents approximately 565 hours of the data, where as microphone speech represents the other 75 hours. (ICSI) at the University of California, Berkley. This collection was part of the

The telephone speech segments include excerpts in the range of 8-12 seconds and 5 minutes from longer original conversations. The interview material includes short conversation interview segments of approximately 3 minutes from a longer interview session. English language transcripts in .cfm format were produced using an automatic speech recognition (ASR) system.

2008 NIST Speaker Recognition Evaluation Training Set Part 1 is distributed on 9 DVD-ROM.
2011 Subscription Members will automatically receive two copies of this corpus. 2011 Standard Members may request a copy as part of their 16 free membership corpora. Non-members may license this data for $2000.

*

(3) Arabic Treebank: Part 2 (ATB2) v 3.1 was developed at LDC. It consists of 501 newswire stories from Ummah Press with part-of-speech (POS), morphology, gloss and syntactic treebank annotation in accordance with the Penn Arabic Treebank (PATB) Guidelines developed in 2008 and 2009. This release represents a significant revision of LDC's previous ATB2 publication: Arabic Treebank: Part 2 v 2.0 LDC2004T02.

The ongoing PATB project supports research in Arabic-language natural language processing and human language technology development. The methodology and work leading to the release of this publication are described in detail in the documentation accompanying this corpus and in two research papers: Enhancing the Arabic Treebank: A Collaborative Effort toward New Annotation Guidelines and Consistent and Flexible Integration of Morphological Annotation in the Arabic Treebank.

ATB2 v 3.1 contains a total of 144,199 source tokens before clitics are split, and 169,319 tree tokens after clitics are separated for the treebank annotation. Source texts were selected from Ummah Press news archives covering the period from July 2001 through September 2002.

Arabic Treebank: Part 2 v 3.1 is distributed via web download. 2011 Subscription Members will automatically receive two copies of this corpus on disc. 2011 Standard Members may request a copy as part of their 16 free membership corpora. Non-members may license this data for $4500.