LDC 2026 October Newsletter

Membership year 2027 publication preview  

Fall 2026 data scholarship recipients

New publications:

Global TIMIT L1 Tianjin – L2 English

LoReHLT Turkish Representative Language Pack

Multi-Language Conversational Telephone Speech 2014 – French Group

_______________________________________________________________________

Membership year 2027 publication preview 
The 2027 membership year is approaching and plans for next year’s publications are in progress. Among the expected releases are:  

• Fisher English Training Part 3 Speech: 1300+ hours of previously unreleased English conversational telephone speech from the Fisher collection with speaker metadata  
• Computational Cultural Understanding: multimodal data and annotations, assessments, results and supplemental resources to support system development for analyzing and interpreting sociocultural norms, recognizing emotion across cultural contexts, and detecting changes that affect cultural understanding; includes Chinese, Japanese, Russian, Spanish and Turkish releases 
• Urdu Conversational Telephone Speech and Transcripts: 74 Urdu telephone conversations (14 hours) with transcripts, developed to support language identification, spoken language recognition and transcription of the Urdu language 
• Speech Biomarkers: 23 hours of spontaneous and elicited speech and corresponding transcripts from American English speakers designed to establish a baseline of language and speech patterns from healthy speakers for comparison with clinical populations such as persons with dementia
• Global TIMIT: a continuing series of TIMIT-like read speech corpora, focusing on Mandarin Chinese, Mandarin Chinese dialects and English 

Check your inbox for more information about membership renewal.

Fall 2026 data scholarship recipients
Congratulations to the recipients of LDC's Fall 2026 data scholarships:

Muhammad Azeem: Comenius University Bratislava; Mathematical Institute, Slovak Academy of Sciences (Slovakia): PhD candidate. Muhammad is awarded copies of 2008 NIST Speaker Recognition Evaluation Training Set Part 1 LDC2011S05 and UCLA Speaker Variability Database LDC2021S09 for his work in speaker recognition and pairwise classification. 

Cole Baisch: Claremont Graduate University (USA): PhD candidate, Positive Development Psychology, Division of Behavioral and Organizational Sciences. Cole is awarded copies of Switchboard-1 Release 2 LDC97S62 and NXT Switchboard Annotations LDC2009T26 for his work in conversational quality.

Chungche Huang: National Taiwan University (Taiwan): PhD candidate, Graduate Institute of Linguistics. Chungche is awarded a copy of ETS Corpus of Non-Native Written English LDC2014T06 for his work in LLM-based automated essay scoring. 

Zhengyang Li, PhD candidate, and Miguel von Aspern, M. Sc.: Technical University of Braunschweig (Germany): Institute of Communications Technology. Zhengyang and Miguel are awarded copies of Switchboard-1 Release 2 LDC97S62, Fisher English Training Speech, Part 1 Speech LDC2004S13, Fisher English Training Speech, Part 1 Transcripts LDC2004T19, Fisher English Training Part 2, Speech LDC2005S13 and Fisher English Training, Part 2 Transcripts LDC2005T19 for their work in low-latency and full-duplex spoken language modeling.

Garrett Oliver: Presbyterian College (USA): B.A. English candidate, English Department. Garrett is awarded a copy of Penn Parsed Corpora of Historical English Second Edition LDC2025T09 for his work in Celtic influence in English.

New publications:

Global TIMIT L1 Tianjin – L2 English was developed by LDC and Tianjin University and contains 43 hours of read speech and transcripts in the Tianjin dialect of Mandarin Chinese and English. Fifty speakers read sentences from an everyday Chinese sentence list, Chinese Gigaword Fifth Edition (LDC2011T13), TIMIT Acoustic-Phonetic Continuous Speech Corpus (LDC93S1) and Treebank-3 (LDC99T42). Some sentences were read by all speakers, others were read by a few speakers, and others were read by just one speaker.

The corpus was collected at Tianjin University, China. Speakers (25 female, 25 male) were born and raised in one of the six urban districts in Tianjin, China and spoke Tianjin. They were also fluent in English based on passing scores on a standard English assessment test.

All recordings were aligned into word, phone and tone segments. Each recording file has corresponding phone, tone, and word segmentation files and Praat TextGrid files.

The Global TIMIT project aimed to create a series of corpora in a variety of languages with a similar set of key features as in the original TIMIT Acoustic-Phonetic Continuous Speech Corpus (LDC93S1) which was designed for acoustic-phonetic studies and for the development and evaluation of automatic speech recognition systems. 

2026 members can access this corpus through their LDC accounts. Non-members may license this data for a fee. 

*

LoReHLT Turkish Representative Language Pack was developed by LDC and contains 422 million words of Turkish monolingual text, about 830,000 words of which were translated into English, 84,000 Turkish words translated from English data, and 25 hours of Turkish broadcast news and amateur web audio recordings. Approximately 88,000 words were annotated for named entities and over 27,000 words were annotated for full entity including nominals and pronouns. Noun-phrase chunking was applied to more than 9,300 words. Over 6,200 words were morphologically aligned, and more than 25,400 words were labeled with simple semantic annotation. Topic annotation was applied to the audio recordings. Data was collected from discussion forum, news, reference, social network, broadcast news, web audio recordings, and weblogs.

LoReHLT was a companion project of the DARPA LORELEI program. The LORELEI (Low Resource Languages for Emergent Incidents) program was concerned with building human language technology for low resource languages in the context of emergent situations. Representative languages were selected to provide broad typological coverage.

2026 members can access this corpus through their LDC accounts. Non-members may license this data for a fee.

*

Multi-Language Conversational Telephone Speech 2014 – French Group was developed by LDC and contains 156 hours of Haitian Creole and West African French telephone speech. The data was collected to support research and technology evaluation in automatic language identification; portions of these recordings were used in the NIST 2015 and 2017 language recognition evaluations. The collection focused on language pair discrimination for 20 languages/dialects, some of which could be considered mutually intelligible or closely related. 

This corpus contains 722 recordings: 412 Haitian Creole recordings and 310 West African French recordings. Participants were recruited by native speakers who contacted acquaintances in their social network. Those native speakers made one call, up to 8 minutes, to each acquaintance. Human auditors labeled the calls for language, callee gender, native speaker status and speech clarity.

2026 members can access this corpus through their LDC accounts. Non-members may license this data for a fee.

LDC August 2026 Newsletter

Fall 2026 LDC data scholarship program 

New publications:

Parsed Early English Books Online -Text Creation Partnership

Multi-Language Conversational Telephone Speech 2014 – Slavic Group

LORELEI Arabic Representative Language Pack

___________________________________________________________________

Fall 2026 LDC data scholarship program 
Student applications for the Fall 2026 LDC data scholarship program are being accepted now through September 15, 2026. This program provides eligible students with no-cost access to LDC data. Students must complete an application consisting of a data use proposal and letter of support from their advisor. For application requirements and program rules, visit the LDC Data Scholarships page. 

New publications:

Parsed Early English Books Online - Text Creation Partnership (EEBO-TCP), developed by LDC, is a part-of-speech tagged and syntactically parsed version of the EEBO-TCP collection of Early Modern English texts. The corpus consists of 59,433 texts dating primarily from 1600-1700, with a smaller number of texts from earlier and later periods. It contains 48.6 million parsed sentences (trees) comprising more than 1.5 billion tokens. The parses were produced automatically using a parser trained on the Penn Helsinki Parsed Corpus of Early Modern English, part of the Penn Parsed Corpora of Historical English (LDC2020T16). The automatically generated parses were not manually reviewed.

Parsed EEBO-TCP includes the CorpusSearch 2 program and associated documentation. This tool allows users to search the data for syntactic structure, word sequences and words. An alternative version of CorpusSearch 2 for use on very large corpora is also included in this release.

2026 members can access this corpus through their LDC accounts. Non-members may license this data for a fee.

*

Multi-Language Conversational Telephone Speech 2014 – Slavic Group was developed by LDC and is comprised of 20 hours of Polish and Russian telephone speech. The data was collected to support research and technology evaluation in automatic language identification; portions of these recordings were used in the NIST 2015 and 2017 language recognition evaluations. The collection focused on language pair discrimination for 20 languages/dialects, some of which could be considered mutually intelligible or closely related. 

This corpus contains 96 recordings: 61 Polish recordings and 35 Russian recordings. Participants were recruited by native speakers who contacted acquaintances in their social network. Those native speakers made one call, up to 8 minutes, to each acquaintance. Human auditors labeled the calls for language, callee gender, native speaker status and speech clarity.

2026 members can access this corpus through their LDC accounts. Non-members may license this data for a fee.

*

LORELEI Arabic Representative Language Pack contains 2.4 million words of Arabic monolingual text, 930,00 words of which were translated into English, and 225,000 Arabic words translated from English data. Nearly 88,000 words were annotated for simple named entities, and approximately 29,000 words were annotated for full entities (including nominals and pronouns). In addition, over 10,000 words were annotated for noun phrase chunking, 18,000 words were labeled with semantic annotation, and situation frame annotation was applied to more than 24,000 words.

The LORELEI (Low Resource Languages for Emergent Incidents) program was concerned with building human language technology for low resource languages in the context of emergent situations. Representative languages were selected to provide broad typological coverage.

The knowledge base for entity linking annotation is available separately as LORELEI Entity Detection and Linking Knowledge Base (LDC2020T10).

2026 members can access this corpus through their LDC accounts. Non-members may license this data for a fee.

LDC April 2026 Newsletter

New publications:

DEFT Chinese and English Light and Rich ERE Parallel Annotation

MATERIAL Tagalog-English Language Pack

LORELEI Somali Representative Language Pack

____________________________________________________________________

New publications:

DEFT Chinese and English Light and Rich ERE Parallel Annotation was developed by LDC and consists of 179 Chinese discussion forum documents and their English translations annotated for entities, relations, and events (ERE). Light ERE annotation labels entity mentions for the target set of entity, relation and event types between and among those entities including coreference. Rich ERE annotation expands types and tagging in the entities, relations, and events annotation tasks and replaces strict event coreference with a more loosely defined event hopper annotation. 179 Chinese-English document pairs were annotated following Light ERE annotation guidelines; a subset of 171 Chinese-English document pairs were also labeled with Rich ERE annotation. The source data and English translations were drawn from BOLT Chinese Discussion Forum Parallel Training Data (LDC2017T05) originally collected and translated by LDC under the DARPA BOLT program.

DARPA's Deep Exploration and Filtering of Text (DEFT) program aimed to address remaining capability gaps in state-of-the-art natural language processing technologies related to inference, causal relationships and anomaly detection. LDC supported the DEFT program by collecting, creating and annotating a variety of data sources.

2026 members can access this corpus through their LDC accounts. Non-members may license this data for a fee.

*

MATERIAL Tagalog-English Language Pack was developed by Appen for the IARPA MATERIAL program and contains 100 hours of Tagalog conversational telephone speech, transcripts, English translations, annotations and queries. Calls were made using different telephones (e.g., mobile, landline) from a variety of environments. Transcripts cover approximately 30% of the speech files, 2% of which were translated into English. This release also includes domain annotations, English queries and their relevance annotations. 

The MATERIAL program focused on underserved languages with the ultimate goal to build cross language information retrieval systems to find speech and text content using English search queries.

2026 members can access this corpus through their LDC accounts provided they have submitted a completed copy of the special license agreement. Non-members may license this data for a fee.

*

LORELEI Somali Representative Language Pack contains over 13 million words of Somali monolingual text, 800,00 words of which were translated into English, and 106,000 Somali words translated from English data. Approximately 73,000 words were annotated for simple named entities, around 23,000 words were annotated for full entity (including nominals and pronouns), and over 10,000 words were covered by noun phrase chunking annotation. Data was collected from discussion forum, news, reference, social network, and weblogs.

The LORELEI (Low Resource Languages for Emergent Incidents) program was concerned with building human language technology for low resource languages in the context of emergent situations. Representative languages were selected to provide broad typological coverage.

The knowledge base for entity linking annotation is available separately as LORELEI Entity Detection and Linking Knowledge Base (LDC2020T10).

2026 members can access this corpus through their LDC accounts. Non-members may license this data for a fee.

LDC December 2025 Newsletter

LDC 2026 membership discounts now available

LDC’s 1000th corpus

Approaching deadline for Spring 2026 data scholarship applications 

LDC closed for Winter Break December 25 – January 2  

New publications:

2021 NIST Speaker Recognition Evaluation Development and Test Set

LORELEI Sinhala Incident Language Pack

_______________________________________________________________________

LDC 2026 membership discounts now available 
Now through March 2, 2026, any organization that joins the Consortium or renews their membership will receive a 10% discount off the 2026 membership fee. Membership remains the most economical way to access current and past LDC releases. Consult Join LDC for details on membership options and benefits. 

LDC’s 1,000th corpus
LDC is delighted to announce the release of the 1,000th corpus into the Catalog! This milestone represents the commitment we made over thirty years ago to provide large quantities of diverse data, robust research program support and exceptional member services. We are grateful for the continued support and collaboration of our members, friends and the community.  

Approaching deadline for Spring 2026 data scholarship applications
Attention students: don’t miss out on the chance to receive no-cost access to LDC data for your research. Applications for Spring 2026 data scholarships are due January 15, 2026. For more information on requirements and program rules, see LDC Data Scholarships. 

LDC closed for Winter Break December 25-January 2 
LDC will be closed from Thursday, December 25, 2025 through Friday, January 2, 2026 in accordance with the University of Pennsylvania Winter Break Policy. Our offices will reopen on Monday, January 5, 2026. Requests received by the Membership Office during Winter Break will be processed when the office reopens. 
 
New publications:
2021 NIST Speaker Recognition Evaluation Test Set was developed by LDC and NIST (National Institute of Standards and Technology). It contains approximately 447 hours of Cantonese, Mandarin, and English conversational telephone speech, audio from video, and selfie image data for development and test, along with answer keys, enrollment, trial files and documentation from the NIST-sponsored 2021 Speaker Recognition Evaluation (SRE).

The SRE task is speaker detection, that is, to determine whether a specified target speaker was speaking during a segment of speech. SRE21 focused on telephone speech and audio from video and included close-up images of participants. The evaluation also featured cross-lingual trials, that is, enrollment and test segments spoken in different languages.

The data was drawn from the WeCanTalk corpus collected by LDC in which speakers called friends or relatives who agreed to record their telephone conversations lasting between 8-10 minutes. Subjects contributed multiple conversational telephone speech recordings and audio recordings in which they were talking, plus a single selfie image. Recordings were manually audited to verify speaker, language, and quality.

2025 members can access this corpus through their LDC accounts. Non-members may license this data for a fee.

*

LORELEI Sinhala Incident Language Pack was developed by LDC and is comprised of 8.1 million words of Sinhala monolingual text, 700,00 words of English monolingual text, 6.4 million words of parallel Sinhala- English text, and 50,000 words annotated for entity discovery and linking and situation frames. It constitutes all of the text data, annotations, supplemental resources and related software tools for the Sinhala language used in the DARPA LORELEI / LoReHLT 2018 Evaluation.

The LORELEI (Low Resource Languages for Emergent Incidents) program was concerned with building human language technology for low resource languages in the context of emergent situations. In the evaluation scenario, an unforeseen event triggered a need for humanitarian and logistical support in a region where the incident language had received little or no attention in NLP research. Evaluation participants provided NLP solutions, including information extraction and machine translation, with limited resources and limited development time.

Data was collected from news, social network, weblog, newsgroup, discussion forum, and reference material. Entity discovery and linking annotation identified entities to be detected by systems for scoring purposes. Situation frame analysis was designed to extract basic information about needs and relevant issues for planning a disaster response effort.

2025 members can access this corpus through their LDC accounts. Non-members may license this data for a fee. 

LDC August 2024 Newsletter

Fall 2024 LDC Data Scholarship Program  

New publications:

LORELEI Uyghur Incident Language Pack

Ravnursson Faroese Speech and Transcripts

_______________________________________________________________

Fall 2024 LDC Data Scholarship Program  

Student applications for the Fall 2024 LDC Data Scholarship program are being accepted now through September 15, 2024. This program provides eligible students with no-cost access to LDC data. Students must complete an application consisting of a data use proposal and letter of support from their advisor. For application requirements and program rules, visit the LDC Data Scholarships page. 

New publications: 

LORELEI Uyghur Incident Language Pack was developed by LDC and is comprised of 28 million words of Uyghur monolingual text, 500,000 words of English monolingual text, 3.3 million words of parallel and comparable Uyghur-English text, and 200,000 words annotated for simple named entities and situation frames. It constitutes all of the text data, annotations, supplemental resources and related software tools for the Uyghur language that were used in the DARPA LORELEI / LoReHLT 2016 Evaluation.

The LORELEI (Low Resource Languages for Emergent Incidents) program was concerned with building human language technology for low resource languages in the context of emergent situations. In the evaluation scenario, an unforeseen event triggered a need for humanitarian and logistical support in a region where the incident language had received little or no attention in NLP research. Evaluation participants provided NLP solutions, including information extraction and machine translation, with limited resources and limited development time.

Data was collected from news, social network, weblog, newsgroup, discussion forum, and reference material. Named entity annotation identified entities to be detected by systems for scoring purposes. Situation frame analysis was designed to extract basic information about needs and relevant issues for planning a disaster response effort.

2024 members can access this corpus through their LDC accounts. Non-members may license this data for a fee.

*

Ravnursson Faroese Speech and Transcripts contains 109 hours of Faroese prompted speech from 433 speakers (249 female, 184 male), corresponding transcripts and speaker metadata. It is an extract from the Basic Language Resource Kit 1.0 (BLARK 1.0) developed by the Faroe Islands' Ravnur Project.
 
Speech data was collected in 2022. Speakers from all major dialect areas in the Faroe Islands in three age groups -- 15-35, 36-60, and 61+ years -- read texts that included a word list, a phrase list, closed vocabulary readings, and short texts. Recordings also contain spontaneous speech. Orthographic transcripts are included. 

2024 members can access this corpus through their LDC accounts provided they have submitted a completed copy of the special license agreement. Non-members may license this data at no cost.
 

 

LDC June 2023 Newsletter

LDC at ACL 2023

LDC data and commercial technology development

New publications:

Moroccan Arabic – English Lexical Database

LORELEI Indonesian Representative Language Pack

_________________________________________________________________

LDC at ACL 2023

LDC will be exhibiting at ACL 2023, held this year July 9-14 in Toronto, Canada. Stop by our booth to learn more about recent developments at the Consortium and the latest publications. LDC will post conference updates via Twitter and Facebook. We look forward to seeing you there!

LDC data and commercial technology development

For-profit organizations are reminded that an LDC membership is a pre-requisite for obtaining a commercial license to almost all LDC databases. Non-member organizations, including non-member for-profit organizations, cannot use LDC data to develop or test products for commercialization, nor can they use LDC data in any commercial product or for any commercial purpose. LDC data users should consult corpus-specific license agreements for limitations on the use of certain corpora. Visit the Licensing page for further information.

New publications:

Moroccan Arabic - English Lexical Database was developed by LDC. It contains a set of five interrelated tables presenting each Moroccan Arabic word as an orthographic form in Arabic script and a pronunciation form in International Phonetic Alphabet (IPA) format. This release contains over 21,000 Moroccan Arabic words in Arabic script and IPA notation and more than 33,000 English tokens.

This lexical database is the result of a collaboration with Georgetown University Press (GUP) to enhance and update three dialectal Arabic dictionaries -- Iraqi, Moroccan and Syrian -- originally published in paper form in the 1960s by GUP.  LDC also undertook to develop a lexical database for each dialect. The Georgetown Dictionary of Moroccan Arabic was published in 2019; this work was based on, and expanded, A Dictionary of Moroccan Arabic.

The several enhancements developed by LDC included facilitating comparisons across Arabic dialects and Modern Standard Arabic by providing Arabic script spellings and IPA pronunciations to Moroccan words and phrases; promoting ease of use by language learners and researchers by developing reasonable orthographic conventions for applying the Arabic alphabet to the dialect; and facilitating a user's understanding of morphological and lexical relations by adding information on the linguistic structures of Moroccan Arabic.

2023 members can access this corpus through their LDC accounts provided they have submitted a signed copy of the special license agreement. Non-members may license this data for a fee.

*

LORELEI Indonesian Representative Language Pack is comprised of over 17 million words of Indonesian monolingual text, 950,000 million words of found Indonesian-English parallel text, and 92,000 Indonesian words translated from English data. Over 113,000 words were annotated for named entities and more than 24,000 words were annotated for entity discovery and linking and situation frames (identifying entities, needs and issues). Data was collected from discussion forum, news, reference, social network, and weblogs.

The LORELEI (Low Resource Languages for Emergent Incidents) program was concerned with building human language technology for low resource languages in the context of emergent situations. Representative languages were selected to provide broad typological coverage.

The knowledge base for entity linking annotation is available separately as LORELEI Entity Detection and Linking Knowledge Base (LDC2020T10).

2023 members can access this corpus through their LDC accounts. Non-members may license this data for a fee.

LDC May 2023 Newsletter

LDC at ICASSP 2023

New publications:

2019 NIST Speaker Recognition Evaluation Test Set – CTS Challenge

LORELEI Zulu Representative Language Pack
_____________________________________________________________
LDC at ICASSP 2023

LDC will be exhibiting at ICASSP 2023, held this year June 4-10 in Rhodes, Greece. Stop by booth 15 to learn more about recent developments at the Consortium and the latest publications.

LDC will post conference updates via Twitter and Facebook. We look forward to seeing you there!

New publications:

2019 NIST Speaker Recognition Evaluation Test Set – CTS Challenge, developed by LDC and NIST, contains 635 hours of Tunisian Arabic telephone recordings for development and test, answer keys, enrollment, trial files and documentation from the CTS Challenge portion of the NIST-sponsored 2019 Speaker Recognition Evaluation. The 2019 evaluation was conducted in two parts: (1) a leaderboard-style challenge based on conversational telephone speech from LDC's Call My Net 2 (CMN2) corpus; and (2) a separate evaluation using audio-visual material collected by LDC for the VAST (Video Annotation for Speech Technology) project (released as LDC2023V01).
 
The telephone speech data for the CTS Challenge was drawn from the CMN2 collection conducted by LDC in Tunisia in which Tunisian Arabic speakers called friends or relatives who agreed to record their telephone conversations lasting between 8-10 minutes. The speech segments include PSTN (public switched telephone network) and VOIP (voice over IP) data.
2023 members can access this corpus through their LDC accounts. Non-members may license this data for a fee.

*

LORELEI Zulu Representative Language Pack is comprised of over 5 million words of Zulu monolingual text, 2.7 million words of found Zulu-English parallel text, and 71,000 Zulu words translated from English data. Approximately 100,000 words were annotated for named entities and over 23,000 words were annotated for entity discovery and linking and situation frames (identifying entities, needs and issues). Data was collected from discussion forum, news, reference, social network, and weblogs.

The LORELEI (Low Resource Languages for Emergent Incidents) program was concerned with building human language technology for low resource languages in the context of emergent situations. Representative languages were selected to provide broad typological coverage.

The knowledge base for entity linking annotation is available separately as LORELEI Entity Detection and Linking Knowledge Base (LDC2020T10).

2023 members can access this corpus through their LDC accounts. Non-members may license this data for a fee.

LDC April 2022 Newsletter

LDC Celebrates 30 Years

LDC Releases Ukrainian Data for Disaster and Refugee Relief Research

New publication:
LORELEI Wolof Representative Language Pack
_______________________________________________________________

LDC Celebrates 30 Years
April 2022 marks the beginning of LDC’s 30th year as the leader in language resource development and distribution. Founded in 1992, the Consortium has grown from a data repository to a vibrant data center that creates, shares and preserves language resources for research, education and technology development. The Catalog continues to grow, housing over 900 titles in more than 90 languages. With the support of members, licensees, sponsors and collaborators, LDC has distributed over 200,000 copies of data to more than 6,000 organizations worldwide. We are sincerely grateful to the community, and we pledge to continue the mission to provide diverse data, high-quality member services and research program support. 

Stay tuned for upcoming newsletter highlights from the last three decades! 

LDC Releases Ukrainian Data for Disaster and Refugee Relief Research
LDC is releasing Ukrainian data it developed in the DARPA AIDA program, the NIST Language Recognition Evaluation series and the DARPA LORELEI program under a special no-cost, limited license for disaster and refugee relief research. 

These resources are available in three corpora:

LDC2022E06     AIDA Ukrainian Broadcast and Telephone Speech Audio and Transcripts
LDC2020T24     LORELEI Ukrainian Representative Language Pack
LDC2020T10     LORELEI Entity Detection and Linking Knowledge Base

For further information about these data sets and licensing terms, see Disaster and Refugee Relief Research.

New publication:

LORELEI Wolof Representative Language Pack was developed by LDC and is comprised of approximately 225,000 words of Wolof monolingual text, 115,000 Wolof words translated from English data, 15,000 words annotated for named entities and 5,000-8,000 words annotated for entity discovery and linking and situation frames. 

The LORELEI (Low Resource Languages for Emergent Incidents) program was concerned with building human language technology for low resource languages in the context of emergent situations. Representative languages were selected to provide broad typological coverage.

Data was collected from news, social network, weblog, discussion forum, and reference material. Entity detection and linking annotation identified entities to be detected by systems for scoring purposes. Situation frame analysis was designed to extract basic information about needs and relevant issues for planning a disaster response effort.

The knowledge base for entity linking annotation is available separately as LORELEI Entity Detection and Linking Knowledge Base (LDC2020T10).

LORELEI Wolof Representative Language Pack is distributed via web download.  

2022 Subscription Members will automatically receive copies of this corpus. 2022 Standard Members may request a copy as part of their 16 free membership corpora. Non-members may license this data for a fee.

LDC 2020 November Newsletter

Join LDC for Membership Year 2021
Spring 2021 Data Scholarship Application Deadline

New Publications:
Global TIMIT Learner Simple English
LORELEI Ukrainian Representative Language Pack 

TAC KBP Event Argument – Comprehensive Training and Evaluation Data 2016-2017 

________________________________________________________________________

Join LDC for Membership Year 2021 

Membership Year 2021 (MY2021) is open and discounts are available for those who keep their membership current and join early. Current MY2020 members who renew their LDC membership before March 1, 2021 will receive a 10% discount off the membership fee. New or returning organizations will receive a 5% discount when joining by March 1.

In addition to receiving new publications, current LDC members also enjoy the benefit of licensing older data at reduced costs from our Catalog of over 850 holdings. Current-year for-profit members may use most data for commercial applications.

Plans for MY2021 publications are in progress. Among the expected releases are:

• Global TIMIT Mandarin Chinese: 6,000 linguistically rich utterances featuring time-aligned lexical and phonetic transcription
• Columbia Games Corpus: 12 spontaneous task-oriented dyadic conversations elicited from native Standard American English speakers playing computer games, transcribed and annotated for discourse/pragmatic phenomena
• My Science Tutor Children’s Conversational Speech: 400+ hours of speech from 1,371 US third, fourth, and fifth grade students participating in sessions with a virtual science tutor, transcripts included
• The SSNCE Database of Tamil Dysarthric Speech: Tamil speech from 20 dysarthric speakers aged 12-40 years and a control group (10 speakers) with time-aligned phonetic transcripts
• Icelandic Parliamentary Speech: 6,493 Icelandic Parliament recordings from 2005-2016 with 196 speakers, aligned and segmented and divided into training, development, and evaluation sets for ASR development
• LORELEI: representative and incident language packs containing monolingual text, bi-text, translations, annotations, supplemental resources, and related tools (Akan, Kinyarwanda, and Wolof)
• BOLT: co-reference, treebank, propbank, and translation resources for discussion forum, SMS/Chat, and conversational telephone speech data in all languages (Chinese, Egyptian Arabic, and English)
• TAC KBP: training and evaluation data for English surprise slot filling (2010) and English sentiment slot filling (2013-2014) tasks 

It’s also not too late to join for MY2019 (through December 31, 2020) and MY2020 (through December 31, 2021). Data sets from those years include Penn Discourse Treebank Version 3.0, DEFT Committed Belief Annotation (Chinese, English, Spanish), 2018 NIST Speaker Recognition Evaluation Test Set, Mixer 4 and 5 Speech, AMR Annotation Release 3.0, and Penn Parsed Corpora of Historical English.

For full descriptions of all LDC data sets, browse our Catalog.  

Visit Join LDC for details on membership, user accounts and payment.

Spring 2021 Data Scholarship Application Deadline
Applications are now being accepted through January 15, 2021 for the Spring 2021 LDC Data Scholarship program which provides university students with no-cost access to LDC data. Consult the LDC Data Scholarship page for more information about program rules and submission requirements.

---

New publications:
(1) Global TIMIT Learner Simple English was developed by LDC and Shanghai Jiao Tong University and consists of approximately 12 hours of L1 and L2 English read speech and transcripts. It is comprised of two separate data sets of 50 speakers reading 120 sentences from TIMIT Acoustic-Phonetic Continuous Speech Corpus (LDC93S1) deemed “simple” to read by Chinese learners of English. Among the 120 sentences, 20 sentences were read by all speakers, 40 sentences were read by 10 speakers, and 60 sentences were read by one speaker, for a total of 820 sentence types. 

L1 Simple English was recorded at the University of Pennsylvania, USA; participants were 25 female and 25 male native American English speakers. L2 Simple English was recorded at Shanghai Jiao Tong University, China. L2 speakers (25 female, 25 male) were Chinese learners of English considered fluent.

The Global TIMIT project aimed to create a series of corpora in a variety of languages with a similar set of key features as in the original TIMIT data set, which was designed for acoustic-phonetic studies and for the development and evaluation of automatic speech recognition systems. Specifically, those features include:

• A large number of fluently read sentences, containing a representative sample of phonetic, lexical, syntactic, semantic, and pragmatic patterns
• A relatively large number of speakers
• Time-aligned lexical and phonetic transcription of all utterances
• Some sentences read by all speakers, others read by a few speakers, and others read by just one speaker
  
  

Global TIMIT Learner Simple English is distributed via web download.  

2020 Subscription Members will automatically receive copies of this corpus. 2020 Standard Members may request a copy as part of their 16 free membership corpora. Non-members may license this data for a fee.

*

(2) LORELEI Ukrainian Representative Language Pack consists of Ukrainian monolingual text, Ukrainian-English parallel and comparable text, annotations, supplemental resources, and related software tools developed by LDC for the DARPA LORELEI program.

The LORELEI (Low Resource Languages for Emergent Incidents) program was concerned with building human language technology for low resource languages in the context of emergent situations like natural disasters or disease outbreaks. Linguistic resources for LORELEI include Representative Language Packs and Incident Language Packs for over two dozen low resource languages, comprising data, annotations, basic natural language processing tools, lexicons, and grammatical resources. Representative languages were selected to provide broad typological coverage, while incident languages were selected to evaluate system performance on a language whose identity was disclosed at the start of the evaluation.

Data was collected from discussion forum, news, reference, social network, and weblog. Data volumes are as follows:

• 111 million words of Ukrainian monolingual text, approximately 700,000 words of which were translated into English
• 86,000 Ukrainian words translated from English data
• 174,000 words of found parallel text
• over 2,000,000 words of comparable text
  
  

Approximately 75,000 words were annotated for named entities and up to 50,000 words contain additional annotation, including situation frames (identifying entities, needs and issues) and entity linking and detection.

LORELEI Ukrainian Representative Language Pack is distributed via web download.

2020 Subscription Members will automatically receive copies of this corpus. 2020 Standard Members may request a copy as part of their 16 free membership corpora. Non-members may license this data for a fee.

*

(3) TAC KBP Event Argument – Comprehensive Training and Evaluation Data 2016-2017 was developed by LDC and contains training and evaluation data produced in support of the 2016 TAC KBP Event Argument Linking Pilot and Evaluation tasks and the 2017 Event Argument Linking Training Evaluation task. 

The Event Argument Extraction and Linking task required systems to extract event arguments (entities or attributes playing a role in an event) from unstructured text, indicate the role they play in an event, and link the arguments appearing in the same event to each other. Since the extracted information must be suitable as input to a knowledge base, systems constructed tuples indicating the event type, the role played by the entity in the event, and the most canonical mention of the entity from the source document. The event types and roles were drawn from an externally specified ontology of 31 event types, which included financial transactions, communication events, and attacks.

This corpus includes source documents, manual runs, assessments and event hoppers, a form of identity coreference for events. Source data is Chinese, English, and Spanish newswire and discussion forum text collected by LDC.

TAC KBP Event Argument – Comprehensive Training and Evaluation Data 2016-2017 is distributed via web download. 

2020 Subscription Members will automatically receive copies of this corpus. 2020 Standard Members may request a copy as part of their 16 free membership corpora. Non-members may license this data for a fee. 

LDC 2020 October Newsletter

Fall 2020 Data Scholarship Recipients 

Membership Year 2021 Publication Preview 

LDC data and commercial technology development  
 
New Publications: 
Global TIMIT Learner Treebank English 
Corpus of Law, Academic, and News 
IARPA Babel Mongolian Language Pack IARPA-babel401b-v2.0b 

__________________________________________________________________

 
Fall 2020 data scholarship recipients 
Congratulations to the recipients of LDC's Fall 2020 data scholarships: 
 
Nicole Dodd: University of California, Davis (USA); MA, Linguistics. Nicole is awarded a copy of Arabic Treebank Part 3 v. 3.2 LDC2010T08 for her research in relative clause processing in Standard Arabic. 
 
Satwik Dutta: University of Texas at Dallas (USA); PhD, Electrical Engineering. Satwik is awarded copies of The CMU Kids Corpus LDC97263 and CLSU: Kids’ Speech Version 1.1. LDC2007S18 for his work in speech activity detection.  
 
Pedram Hosseini: George Washington University (USA); PhD., Computer Science. Pedram is awarded copies of Penn Discourse Treebank Version 3.0 LDC2019T05 and The New York Times Annotated Corpus LDC2008T19 for his research in automatic detection of causal relations in text.  
 
Mariano Maisonnave: Universidad Nacional del Sur (Argentina); PhD, Computer Science. Mariano is awarded a copy of ACE 2005 Multilingual Training Corpus LDC2006T06 for his work in event extraction.  
 
Mark Sullivan: California State University, Los Angeles (USA); Masters, Applied and Advanced Studies in Education. Mark is awarded a copy of ETS Corpus of Non-Native Written English LDC2014T06 for his research in sentence boundary problems of Chinese L1 speakers in English compositions.  
 
For information about the program, visit the Data Scholarships page. 
 
Membership Year 2021 publication preview 
The 2021 Membership Year is just around the corner and plans for next year’s publications are in progress. Among the expected releases are: 

• Global TIMIT Mandarin Chinese: 6,000 linguistically rich utterances featuring time-aligned lexical and phonetic transcription 
• Columbia Games Corpus: 12 spontaneous task-oriented dyadic conversations elicited from native Standard American English speakers playing computer games, transcribed and annotated for discourse/pragmatic phenomena 
• My Science Tutor Children’s Conversational Speech: 400+ hours of speech from 1,371 US third, fourth, and fifth grade students participating in sessions with a virtual science tutor, transcripts included 
• The SSNCE Database of Tamil Dysarthric Speech: Tamil speech from 20 dysarthric speakers aged 12-40 years and a control group (10 speakers) with time-aligned phonetic transcripts 
• Icelandic Parliamentary Speech: 6,493 Icelandic Parliament recordings from 2005-2016 with 196 speakers, aligned and segmented and divided into training, development, and evaluation sets for ASR development 
• LORELEI: representative and incident language packs containing monolingual text, bi-text, translations, annotations, supplemental resources, and related tools (Akan, Kinyarwanda, and Wolof) 
• BOLT: co-reference, treebank, propbank, and translation resources for discussion forum, SMS/Chat, and conversational telephone speech data in all languages (Chinese, Egyptian Arabic, and English) 
• TAC KBP: training and evaluation data for English surprise slot filling (2010) and English sentiment slot filling (2013-2014) tasks  

Check your inbox in the coming weeks for more information about membership renewal.  
 
LDC data and commercial technology development 
For-profit organizations are reminded that an LDC membership is a pre-requisite for obtaining a commercial license to almost all LDC databases. Non-member organizations, including non-member for-profit organizations, cannot use LDC data to develop or test products for commercialization, nor can they use LDC data in any commercial product or for any commercial purpose. LDC data users should consult corpus-specific license agreements for limitations on the use of certain corpora. Visit the Licensing page for further information. 
 
New publications: 

(1) Global TIMIT Learner Treebank English was developed by LDC and LAIX Inc. and consists of approximately 24 hours of L1 and L2 English read speech and transcripts. It is comprised of two separate data sets of 50 speakers reading 120 sentences from Treebank-3 (LDC99T42). Among the 120 sentences, 20 sentences were read by all speakers, 40 sentences were read by 10 speakers, and 60 sentences were read by one speaker, for a total of 3220 sentence types.  
 
L1 English Treebank was recorded at the University of Pennsylvania, USA; participants were 25 female and 25 male native American English speakers. L2 English Treebank was recorded at LAIX Inc., Shanghai, China. L2 speakers (25 female, 25 male) were Chinese learners of English considered fluent and who had passed specified standards on English assessment tests.       
 
The Global TIMIT project aimed to create a series of corpora in a variety of languages with a similar set of key features as in the original TIMIT Acoustic-Phonetic Continuous Speech Corpus (LDC93S1) which was designed for acoustic-phonetic studies and for the development and evaluation of automatic speech recognition systems. Specifically, those features include: 

• A large number of fluently-read sentences, containing a representative sample of phonetic, lexical, syntactic, semantic, and pragmatic patterns
• A relatively large number of speakers 
• Time-aligned lexical and phonetic transcription of all utterances 
• Some sentences read by all speakers, others read by a few speakers, and others read by just one speaker 

Global TIMIT Learner Treebank English is distributed via web download.   
 
2020 Subscription Members will automatically receive copies of this corpus. 2020 Standard Members may request a copy as part of their 16 free membership corpora. Non-members may license this data for a fee. 

* 

(2) Corpus of Law, Academic and News consists of 400 Persian documents divided into three genres: legal, academic, and news. The legal section contains texts from official publications, including the civil penal code, the criminal penal code, and the constitution of the Islamic Republic of Iran. The academic sub-corpus is comprised of published academic abstracts in various disciplinary areas, such as Art and Humanities, Social Sciences, and Natural Sciences. The news sub-corpus was extracted from an archive of ten Iranian news outlets spanning the period 2010- 2020. 
 
Each document contains metadata in the file's header with information such as specific text type, dates, and source, and also contains annotations marking title and body paragraphs.  
 
Corpus of Law, Academic and News is distributed via web download. 
 
2020 Subscription Members will automatically receive copies of this corpus provided they have submitted a completed copy of the special license agreement. 2020 Standard Members may request a copy as part of their 16 free membership corpora. Non-members may license this data for a fee.  

* 

(3) IARPA Babel Mongolian Language Pack IARPA-babel401b-v2.0b was developed by Appen for the IARPA (Intelligence Advanced Research Projects Activity) Babel program. It contains approximately 204 hours of Halh Mongolian conversational and scripted telephone speech collected in 2014 along with corresponding transcripts. The gender distribution among speakers is approximately equal; speakers' ages range from 16 years to 61 years. Calls were made using different telephones (e.g., mobile, landline) from a variety of environments including the street, a home or office, a public place, and inside a vehicle. 

  

The Babel program focused on underserved languages and sought to develop speech recognition technology that could be rapidly applied to any human language to support keyword search performance over large amounts of recorded speech.   

  

This is the last release in the IARPA Babel series which consists of 25 language packs in total. 

  

IARPA Babel Mongolian Language Pack IARPA-babel-401b-v2.0b is distributed via web download.  

2020 Subscription Members will automatically receive copies of this corpus provided they have submitted a completed copy of the special license agreement. 2020 Standard Members may request a copy as part of their 16 free membership corpora. Non-members may license this data for a fee.