LDC August 2017 Newsletter

Fall 2017 LDC Data Scholarship program

LDC at Interspeech 2017

New Publications:

Multi-Language Conversational Telephone Speech 2011 -- South Asian
GALE Phase 4 Arabic Broadcast Conversation Speech
GALE Phase 4 Arabic Broadcast Conversation Transcripts

________________________________________________________________

Fall 2017 LDC Data Scholarship program - September 15 deadline approaching

There is still time to apply to the Fall 2017 LDC Data Scholarship program. Applications will be accepted through Friday September 15, 2017. The LDC Data Scholarship program provides university students with access to LDC data at no cost. Students must complete an application which consists of a data use proposal and letter of support from their advisor.

For more information on application requirements and program rules, please visit the LDC Data Scholarship page. 

Applicants can email their materials to the LDC Data Scholarship program. 

LDC at Interspeech 2017

LDC will once again be exhibiting at Interspeech, held this year August 20-24 in Stockholm, Sweden. Stop by booth 17 to learn more about recent developments at the Consortium and new publications.

Also, be on the lookout for the following presentations featuring LDC work:

Speaker Comparison for Forensic and Investigative Applications III

LDC Executive Director, Chris Cieri, panelist for Topic A: “Process Map and Standardization”

Special Event Session, Wednesday August 23, 13:30-15:30, Hall B3

Call My Net Corpus: A Multilingual Corpus for Evaluation of Speaker Recognition Technology 

Karen Jones, Stephanie Strassel, Kevin Walker, David Graff, Jonathan Wright

Wednesday, August 23, 17:40-18:00 in the Agula Magna room 

LDC will post conference updates via our Twitter feed and Facebook page. We hope to see you there!   

New publications:

(1) Multi-Language Conversational Telephone Speech 2011 -- South Asian was developed by LDC and is comprised of approximately 118 hours of telephone speech in five distinct language varieties of South Asia (i.e. the Indian sub-continent): Bengali, Hindi, Punjabi, Tamil and Urdu. The data were collected primarily to support research and technology evaluation in automatic language identification, and portions of these telephone calls were used in the NIST 2011 Language Recognition Evaluation (LRE). LRE 2011 focused on language pair discrimination for 24 languages/dialects, some which could be considered mutually intelligible or closely related.

Participants were recruited by native speakers who contacted acquaintances in their social network. Those native speakers made one call, up to 15 minutes, to each acquaintance. The data was collected using LDC's telephone collection infrastructure, comprised of three computer telephony systems. Human auditors labeled calls for callee gender, dialect type, and noise. Demographic information about the participants was not collected.

LDC has also released the following as part of the Multi-Language Conversation Telephone Speech 2011 series: Slavic Group (LDC2016S11)  and Turkish (LDC2017S09).

Multi-Language Conversational Telephone Speech 2011 -- South Asian is distributed via web download.

2017 Subscription Members will receive copies of this corpus. 2017 Standard Members may request a copy as part of their 16 free membership corpora. Non-members may license this data for a fee.

*

(2) GALE Phase 4 Arabic Broadcast Conversation Speech was developed by LDC and is comprised of approximately 75 hours of Arabic broadcast conversation speech collected in 2008 and 2009 by LDC, MediaNet, Tunis, Tunisia and MTC, Rabat, Morocco during Phase 4 of the DARPA GALE (Global Autonomous Language Exploitation) Program.

Corresponding transcripts are released as GALE Phase 4 Arabic Broadcast Conversation Transcripts (LDC2017T12).

This release contains 83 audio files presented in FLAC-compressed Waveform Audio File format (.flac), 16000 Hz single-channel 16-bit PCM. Each file was audited by a native Arabic speaker following Audit Procedure Specification Version 2.0 which is included in this release. 

The broadcast conversation recordings in this release feature interviews, call-in programs and roundtable discussions focusing principally on current events from the following sources: Al Alam News Channel, based in Iran; Al Fayhaa, an Iraqi television channel; Al Hiwar, a regional broadcast station based in the United Kingdom; Alnurra, a U.S. government-funded regional broadcaster; Aljazeera, a regional broadcaster located in Doha, Qatar; Al Ordiniyah, a national broadcast station in Jordan; Dubai TV, a broadcast station in the United Arab Emirates; Lebanese Broadcasting Corporation, a Lebanese television station; Saudi TV, a national television station based in Saudi Arabia; Syria TV, the national television station in Syria; and Tunisian National TV, a national television station in Tunisia.

GALE Phase 4 Arabic Broadcast Conversation Speech is distributed via web download.

2017 Subscription Members will receive copies of this corpus. 2017 Standard Members may request a copy as part of their 16 free membership corpora. Non-members may license this data for a fee.

*

(3) GALE Phase 4 Arabic Broadcast Conversation Transcripts was developed by LDC and contains transcriptions of approximately 75 hours of Arabic broadcast conversation speech collected in 2008 and 2009 by LDC, MediaNet, Tunis, Tunisia and MTC, Rabat, Morocco during Phase 4 of the DARPA GALE (Global Autonomous Language Exploitation) program.

Corresponding audio data is released as GALE Phase 4 Arabic Broadcast Conversation Speech (LDC2017S15).

The transcript files are in plain-text, tab-delimited format (TDF) with UTF-8 encoding, and the transcribed data totals 475,211 tokens. The files in this corpus were transcribed by LDC staff and/or by transcription vendors under contract to LDC. Transcribers followed LDC's quick transcription guidelines (QTR) and quick rich transcription specification (QRTR) both of which are included in the documentation with this release.

GALE Phase 4 Arabic Broadcast Conversation Transcripts is distributed via web download.

2017 Subscription Members will receive copies of this corpus. 2017 Standard Members may request a copy as part of their 16 free membership corpora. Non-members may license this data for a fee.

LDC 2016 February Newsletter

­­­­­­­­­­­­­Only two weeks left to enjoy 2016 membership savings

Spring 2016 LDC Data Scholarship recipients

How to Share Data through LDC webinar on YouTube

New publications:

BOLT Chinese Discussion Forums

GALE Phase 3 Arabic Broadcast Conversation Speech Part 2

GALE Phase 3 Arabic Broadcast Conversation Transcripts Part 2

_______________________________________________________________________

Only two weeks left to enjoy 2016 membership savings

There’s still time to save on 2016 membership fees. Now through March 1, all organizations receive a 5% discount when they join for MY2016. MY2015 members are eligible for an additional 5% off the fee (10% total savings) when they renew before March 1.  

To join, create or sign into your LDC user account, select your preferred membership type from the Catalog, add the item to your bin and follow the check-out process. The Membership Office will apply any discounts. Alternatively, if you have already received a renewal invoice from LDC, you can simply pay against that.

For more information on the benefits of membership, visit Join LDC.

Spring 2016 LDC Data Scholarship recipients
Congratulations to the recipients of LDC's Spring 2016 data scholarships:

Shefali Waldekar: Indian Institute of Technology Kharagpur (India), PhD Candidate, Electronics and Electrical Communications Engineering. Shefali is awarded copies of 2002 Rich Transcription Broadcast News and Conversational Telephone Speech and 2005 Spring NIST Rich Transcription (RT05-S) Evaluation Set for her research in audio diarization.

Nikola Invanov Nikolov: University of Zurich and ETH Zurich (Switzerland), MSc candidate in Informatics. Nikola is awarded a copy of Annotated English Gigaword for his research in text summarization.

Om Prakash Singh: Indian Institute of Technology, Guwahati (India), Research scholar in spoken language identification. Om is awarded a copy of NIST Language Recognition Evaluation Test Set for his work in language identification.

Moshen Mohammadi: Iranian Research Institute for Electrical Engineering (Iran), PhD Candidate in Communications. Moshen is awarded copies of the 2008 NIST Speaker Recognition Evaluation Training Sets 1 and 2, the Evaluation Test Set and the Supplemental Set for his work in speaker recognition in noisy environments.

For program information visit the Data Scholarship page.

How to Share Data through LDC webinar on YouTube

LDC’s first webinar, How to Share Data through LDC, is now available for viewing on our YouTube page. Presented live on January 22, 2016, the webinar outlined in easy steps the process for submitting language resources to LDC for publication in the Catalog. In addition, discussion topics included the benefits of sharing data through LDC, the corpus life cycle, data delivery, quality control and more.

New Corpora

(1) BOLT Chinese Discussion Forums was developed by LDC and consists of 1,597,500 discussion forum threads in Chinese harvested from the Internet using a combination of manual and automatic processes.

The DARPA BOLT (Broad Operational Language Translation) program developed machine translation and information retrieval for less formal genres, focusing particularly on user-generated content. LDC supported the BOLT program by collecting informal data sources -- discussion forums, text messaging and chat -- in Chinese, Egyptian Arabic and English. The collected data was translated and annotated for various tasks including word alignment, treebanking, propbanking and co-reference. The material in this release represents the Chinese source data in the discussion forum genre.

Collection was seeded based on the results of manual data scouting by native speaker annotators. When multiple threads from a forum were submitted, the entire forum was automatically harvested and added to the collection. The scale of the collection precluded manual review of all data. Only a small portion of the threads included in this release were manually reviewed, and it is expected that there may be some offensive or otherwise undesired content as well as some threads that contain a large amount of non-Chinese content. Language identification was performed on all threads in this corpus (using CLD2), and threads for which the results indicated a high probability of largely non-Chinese content are identified in this release.

BOLT Chinese Discussion Forums is distributed via web download as a multi-part zip file. Consult the Using LDC Data page (https://www.ldc.upenn.edu/data-management/using) for more information about this format.

2016 Subscription Members will automatically receive two copies of this corpus.  2016 Standard Members may request a copy as part of their 16 free membership corpora.  Non-members may license this data for a fee.

*

(2) GALE Phase 3 Arabic Broadcast Conversation Speech Part 2 was developed by LDC and is comprised of approximately 129 hours of Arabic broadcast conversation speech collected in 2007 and 2008 by LDC, MediaNet, Tunis, Tunisia and MTC, Rabat, Morocco during Phase 3 of the DARPA GALE (Global Autonomous Language Exploitation) program.

Corresponding transcripts are released as GALE Phase 3 Arabic Broadcast Conversation Transcripts Part 2 (LDC2016T06).

These broadcast conversation recordings feature interviews, call-in programs and roundtable discussions focusing principally on current events and are contained in 142 audio files presented in FLAC-compressed Waveform Audio File format (.flac), 16000 Hz single-channel 16-bit PCM. Each file was audited by a native Arabic speaker following Audit Procedure Specification Version 2.0 which is included in this release.

GALE Phase 3 Arabic Broadcast Conversation Speech Part 2 is distributed via web download.

2016 Subscription Members will automatically receive two copies of this corpus. 2016 Standard Members may request a copy as part of their 16 free membership corpora.  Non-members may license this data for a fee.

*

(3) GALE Phase 3 Arabic Broadcast Conversation Transcripts Part 2 was developed by LDC and contains transcriptions of approximately 129 hours of Arabic broadcast conversation speech collected in 2007 and 2008 by LDC, MediaNet, Tunis, Tunisia and MTC, Rabat, Morocco during Phase 3 of the DARPA GALE (Global Autonomous Language Exploitation) program.

Corresponding audio data is released as GALE Phase 3 Arabic Broadcast Conversation Speech Part 2 (LDC2016S01).

The transcript files are in plain-text, tab-delimited format (TDF) with UTF-8 encoding, and the transcribed data totals 845,791 tokens. The transcripts were created with the LDC tool, XTrans, which supports manual transcription and annotation of audio recordings.

The files in this corpus were transcribed by LDC staff and/or by transcription vendors under contract to LDC. Transcribers followed LDC's quick transcription guidelines (QTR) and quick rich transcription specification (QRTR) both of which are included in the documentation with this release.

GALE Phase 3 Arabic Broadcast Conversation Transcripts Part 2 is distributed via web download.

2016 Subscription Members will automatically receive two copies of this corpus.  2016 Standard Members may request a copy as part of their 16 free membership corpora.  Non-members may license this data for a fee.

LDC 2015 August Newsletter

Fall 2015 LDC Data Scholarship program - September 15 deadline approaching 

LDC at Interspeech 2015

2013 Data Pack deadline is September 15

LDC co-organizes LSA2016 Pre-conference Workshop

New publications:

Arabic Learner Corpus

GALE Phase 3 Arabic Broadcast Conversation Speech Part 1

GALE Phase 3 Arabic Broadcast Conversation Transcripts Part 1

---

Fall 2015 LDC Data Scholarship program - September 15 deadline approaching

Student applications for the Fall 2015 LDC Data Scholarship program are being accepted now through Tuesday, September 15, 2015, 11:59PM EST.  The LDC Data Scholarship program provides university students with access to LDC data at no cost.  This program is open to students pursuing both undergraduate and graduate studies in an accredited college or university. LDC Data Scholarships are not restricted to any particular field of study; however, students must demonstrate a well-developed research agenda and a bona fide inability to pay. 

Students will need to complete an application which consists of a data use proposal and letter of support from their adviser.  For further information on application materials and program rules, please visit the LDC Data Scholarship page.  

Applicants can email their materials to the LDC Data Scholarship program. 

LDC at Interspeech 2015

LDC will once again be exhibiting at Interspeech, held this year September 7-10 in Dresden, Germany.  Stop by booth 20 to learn more about recent developments at the Consortium and new publications.

Also, be on the lookout for the following presentations featuring LDC work:

Monday 7 September, Poster Session 3-9, 11:00–13:00
Investigating Consonant Reduction in Mandarin Chinese with Improved Forced Alignment: Jiahong Yuan and Mark Liberman (both LDC) 

Wednesday 9 September, Oral Session 36-5, 17:50-18:10
The Effect of Spectral Slope on Pitch Perception: Jianjing Kuang (UPenn) and Mark Liberman (LDC)

LDC will post conference updates via our Twitter feed and Facebook page. We hope to see you there!  

2013 Data Pack deadline is September 15

One month remains for not-for-profit and government organizations to create a custom data collection of eight corpora from among LDC’s 2013 releases. Selection options include: 1993-2007 United Nations Parallel Text, Chinese Treebank 8.0, CSC Deceptive Speech, GALE Arabic and Chinese speech and text releases, Greybeard, MADCAT training data, NIST 2012 Open Machine Translation (OpenMT) evaluation and progress sets, and more. The 2013 Data Pack is available for a flat rate of $3500 through September 15, 2015.

To license the Data Pack and select eight corpora, login or register for an LDC user account and add the 2013 Data Pack and each of the eight data sets to your bin. Follow the check-out procedure, sign all applicable user agreements and select payment via wire transfer, purchase order or check. LDC will adjust the invoice total to reflect the data pack fee.

To pay via credit card, add the 2013 Data Pack to your bin and check out using the system prompts. At the completion of the transaction, send an email to LDC indicating the eight data sets to include in your order. 

LDC co-organizes LSA2016 Pre-conference Workshop

University of Arizona’s Malcah Yeager-Dror and LDC’s Chris Cieri are organizing the upcoming LSA 2016 workshop “Preparing your Corpus for Archival Storage”. The session is sponsored by the National Science Foundation (BCS #1549994) and will be held on Thursday, January 7, 2016 in Washington, DC before the start of the 90th Annual Meeting of the Linguistic Society of America (LSA 2016).

The workshop will examine critical factors which must be considered when preparing data for comparison and sharing following on the topics discussed in the LSA 2012 workshop, "Coding for Sociolinguistic Archive Preparation". Invited speakers will discuss specific coding conventions for such factors as socioeconomic and educational speaker demographics, language choice, stance and footing.

There will be no additional registration fees to attend the session for those already taking part in the annual meeting. Students who are about to carry out their own fieldwork, or who have begun doing so, are eligible to apply for funding by November 2, 2015 to help defray the extra costs for attending the workshop. For more information about the speakers and topics, visit LDC’s workshop page.

New publications

(1) Arabic Learner Corpus was developed at the University of Leeds and consists of written essays and spoken recordings by Arabic learners collected in Saudi Arabia in 2012 and 2013. The corpus includes 282,732 words in 1,585 materials, produced by 942 students from 67 nationalities studying at pre-university and university levels. The average length of an essay is 178 words.

Two tasks were used to collect the written data, and participants had the choice to do one or both of them. In each of those tasks, learners were asked to write a narrative about a vacation trip and a discussion about the participant's study interest. Those choosing the first task generated a 40 minute timed essay without the use of any language reference materials. In the second task, participants completed the writing as a take-home assignment over two days and were permitted to use language reference materials.

The audio recordings were developed by allowing students a limited amount of time to talk about the topics above without using language reference materials.

The original handwritten essays were transcribed into an electronic text format. The corpus data consists of three types: (1) handwritten sheets scanned in PDF format; (2) audio recordings in MP3 format; and (3) textual unicode data in plain text and XML formats (including the transcribed audio and transcripts of the handwritten essays). The audio files are either 44100Hz 2-channel or 16000Hz 1-channel mp3 files.

Arabic Learner Corpus is distributed via web download.

2015 Subscription Members will automatically receive two copies of this corpus provided that they have completed the license agreement.  2015 Standard Members may request a copy as part of their 16 free membership corpora.  Non-members may license this data for a fee.

*

(2) GALE Phase 3 Arabic Broadcast Conversation Speech Part 1 was developed by LDC and is comprised of approximately 123 hours of Arabic broadcast conversation speech collected in 2007 by LDC, MediaNet, Tunis, Tunisia and MTC, Rabat, Morocco during Phase 3 of the DARPA GALE (Global Autonomous Language Exploitation) program.

Corresponding transcripts are released as GALE Phase 3 Arabic Broadcast Conversation Transcripts Part 1 (LDC2015T16).

Broadcast audio for the GALE program was collected at LDC’s Philadelphia, PA USA facilities and at three remote collection sites. The combined local and outsourced broadcast collection supported GALE at a rate of approximately 300 hours per week of programming from more than 50 broadcast sources for a total of over 30,000 hours of collected broadcast audio over the life of the program.

The broadcast conversation recordings in this release feature interviews, call-in programs and roundtable discussions focusing principally on current events from the following sources: Abu Dhabi TV, Al Alam News Channel,  Al Arabiya,  Aljazeera,  Al Ordiniyah,  Dubai TV,  Lebanese Broadcasting Corporation, Oman TV,  Saudi TV,  and Syria TV.

This release contains 149 audio files presented in FLAC-compressed Waveform Audio File format (.flac), 16000 Hz single-channel 16-bit PCM. 

GALE Phase 3 Arabic Broadcast Conversation Speech Part 1 is distributed via web download.

2015 Subscription Members will automatically receive two copies of this corpus.  2015 Standard Members may request a copy as part of their 16 free membership corpora.  Non-members may license this data for a fee.

*

(3) GALE Phase 3 Arabic Broadcast Conversation Transcripts Part 1 was developed by LDC and contains transcriptions of approximately 123 hours of Arabic broadcast conversation speech collected in 2007 by LDC, MediaNet, Tunis, Tunisia and MTC, Rabat, Morocco during Phase 3 of the DARPA GALE (Global Autonomous Language Exploitation) program.

Corresponding audio data is released as GALE Phase 3 Arabic Broadcast Conversation Speech Part 1 (LDC2015S11).

The transcript files are in plain-text, tab-delimited format (TDF) with UTF-8 encoding, and the transcribed data totals 733,233 tokens. The transcripts were created with the LDC-developed transcription tool, XTrans, a multi-platform, multilingual, multi-channel transcription tool that supports manual transcription and annotation of audio recordings. 

The files in this corpus were transcribed by LDC staff and/or by transcription vendors under contract to LDC. Transcribers followed LDC's quick transcription guidelines (QTR) and quick rich transcription specification (QRTR) both of which are included in the documentation with this release.

GALE Phase 3 Arabic Broadcast Conversation Transcripts Part 1 is distributed via web download.

2015 Subscription Members will automatically receive two copies of this corpus.  2015 Standard Members may request a copy as part of their 16 free membership corpora.  Non-members may license this data for a fee.