LDC November 2016 Newsletter

In this newsletter:

Join LDC for Membership Year 2017

Commercial use and LDC data

Spring 2017 Data Scholarship Program

LDC closed November 24-25 for US Thanksgiving Holiday

New publications:

JANA: A Human-Human Dialogues Corpus for Egyptian Dialect

Multi-Language Conversational Telephone Speech 2011 – Slavic Group

IARPA Babel Georgian Language Pack IARPA-babel404b-v1.0a

GALE Phase 3 and 4 Chinese Newswire Parallel Text

Join LDC for Membership Year 2017

Organizations engaged in language-related research, education and technology development are invited to join LDC for Membership Year (MY) 2017. Consortium members enjoy unparalleled access and continuing rights to new data releases and to an archive of close to 700 holdings.

Membership fees have not increased for 2017. In addition, discounts are available for organizations who keep their membership current and for those who join before March 1, 2017.

           • MY 2016 members receive a 10% discount if they renew their membership before March 1, 2017. After March 1, MY2016 members receive a 5% discount if they renew their membership any time in 2017.

           • New members and returning former members receive a 5% discount off the membership fee if they join/renew before March 1, 2017.

Plans for MY2017 publications are in progress. Among the expected releases are:

2010 NIST Speaker Recognition Evaluation data set

Multilanguage conversational telephone speech: developed to support language identification research in related languages

UCLA High Speed Laryngeal Database: audio recordings and high-speed videoendoscopic images of the vocal folds while sustaining vowels

Noisy TIMIT: TIMIT with added artificial noise

CHiME shared task data: noisy read WSJ speech

First Year Law Students’ Memoranda: memos to a hypothetical court with annotations

IARPA Babel Language Packs: languages include Vietnamese, Haitian Creole, Zulu, Kazakh and Lithuanian

BOLT: source, parallel and word-aligned data in all languages

RATS Keyword Spotting data set

GALE Phases 3 and 4: all tasks and languages   

Visit Join LDC for details on membership, user accounts and payment.

Commercial use and LDC data

For-profit organizations are reminded that an LDC membership is a pre-requisite for obtaining a commercial license to almost all LDC databases. Non-member organizations, including non-member for-profit organizations, cannot use LDC data to develop or test products for commercialization, nor can they use LDC data in any commercial product or for any commercial purpose. LDC data users should consult corpus-specific license agreements for limitations on the use of certain corpora. Visit the Licensing page for further information. 

Spring 2017 Data Scholarship Program

Applications are now being accepted through January 15, 2017 for the Spring 2017 LDC Data Scholarship program which provides university students with no-cost access to LDC data. Consult the LDC Data Scholarship page for further information about program rules and submission requirements.

LDC closed November 24-25 for US Thanksgiving Holiday

LDC will be closed on Thursday, November 24, 2016 and Friday, November 25, 2016 in observance of the US Thanksgiving Holiday. The office will reopen on Monday, November 28, 2016.

New Corpora

(1) JANA: A Human-Human Dialogues Corpus for Egyptian Dialect was developed by researchers at Cairo University. This is a special release in addition to the LDC scheduled corpora for membership year 2016, available under separate terms.

This corpus consists of 82 transcribed dialogues from call center inquiries annotated for dialogue acts. Data was collected from call centers for banks, airlines and mobile network providers in the form of spontaneous spoken telephone dialogues (52) and instant messaging dialogues (30) amounting to over 20,000 words.

Not-for-profit organizations may license this data set for a fee under the LDC Not-for-Profit Membership Agreement or under the LDC User Agreement for Non-Members for use in linguistic research, education and non-commercial technology development. For-profit organizations may license this data for a fee under a commercial license.

(2) Multi-Language Conversational Telephone Speech 2011 – Slavic Group was developed by LDC and is comprised of approximately 60 hours of telephone speech in Polish, Russian and Ukrainian. The data was collected to support research and technology evaluation in automatic language identification, specifically language pair discrimination for closely related languages/dialects.

Call were made using LDC’s telephone collection infrastructure. Human auditors labeled calls for gender, dialect type and noise.  Audio data is presented in FLAC-compressed MS-WAV (RIFF) file format. Each uncompressed file is two channels, recorded at 8000 samples/second with samples stored as 16-bit signed integers.

Multi-Language Conversational Telephone Speech 2011 – Slavic Group is distributed via web download.

2016 Subscription Members will receive copies of this corpus. 2016 Standard Members may request a copy as part of their 16 free membership corpora. Non-members may license this data for a fee.

(3) IARPA Babel Georgian Language Pack IARPA-babel404b-v1.0a was developed by Appen for the IARPA (Intelligence Advanced Research Projects Activity) Babel program. It contains approximately 190 hours of Georgian conversational and scripted telephone speech collected in 2014-2015 along with corresponding transcripts.

The Babel program focuses on underserved languages and seeks to develop speech recognition technology that can be rapidly applied to any human language to support keyword search performance over large amounts of recorded speech.

The Georgian speech in this release represents that spoken in the Eastern and Western dialect regions in Georgia. The gender distribution among speakers is approximately equal; speakers' ages range from 16 years to 73 years. Calls were made using different telephones (e.g., mobile, landline) from a variety of environments including the street, a home or office, a public place, and inside a vehicle.

Transcripts are encoded in UTF-8.

IARPA Babel Georgian Language Pack IARPA-babel404b-v1.0a is distributed via web download.

2016 Subscription Members will receive copies of this corpus provided they have submitted a completed copy of the special license agreement. 2016 Standard Members may request a copy as part of their 16 free membership corpora. Non-members may license this data for a fee.

(4) GALE Phase 3 and 4 Chinese Newswire Parallel Text was developed by LDC and contains Chinese source text and corresponding English translations selected from newswire data collected by LDC in 2007-2008 and translated by LDC or under its direction.

This release includes 367 source-translation document pairs drawn from five distinct newswire sources, comprising 210,048 tokens of Chinese source text and its English translation. Source data and translations are distributed in TDF format. All data is encoded in UTF-8.

GALE Phase 3 and 4 Chinese Newswire Parallel Text is distributed via web download.

2016 Subscription Members will receive copies of this corpus. 2016 Standard Members may request a copy as part of their 16 free membership corpora. Non-members may license this data for a fee.

LDC 2015 November Newsletter

Invitation to Join for Membership Year (MY) 2016

Commercial use and LDC data

Spring 2016 Data Scholarship Program

LDC closed for Thanksgiving Break

New publications:

Articulation Index LSCP
GALE Phase 4 Chinese Newswire Parallel Sentences

KHATT: Handwritten Arabic Text

---

Invitation to Join for Membership Year (MY) 2016

Membership Year (MY) 2016 is open for joining.  We would like to invite all current and previous members of LDC to renew their membership as well as welcome new organizations to join the Consortium.  For MY2016, LDC is pleased to maintain membership fees at last year’s rates – membership fees will not increase.  Additionally, LDC will extend discounts on membership fees to members who keep their membership current and who join early in the year.

The details of our early renewal discounts for MY2015 are as follows: 

• Organizations who joined for MY2015 will receive a 10% discount when renewing before March 1, 2016. After March 1, 2016, MY2015 members are eligible for a 5% discount when renewing through the end of the year.
• New members as well as organizations who did not join for MY2015, but who held membership in any of the previous MYs (1993-2014), will also be eligible for a 5% discount provided that they join/renew before March 1, 2016.

Publications for MY2016 are still being planned but we plan to release the following:

• Arabic Treebank - Weblog ~ part-of-speech/morphological annotation and syntactic tree annotation of web text from various sources
• BOLT ~ all phases, languages, genres, tasks
• DEFT ~ Spanish and Chinese resources
• Digital Archive of Southern Speech - NLP Version ~ colloquial speech in the Southern United States; NLP version normalizes filenames and formats
• GALE Phase 3 and 4 data ~ all tasks and languages    
• HAVIC ~ amateur video and transcripts
• NewSoMe Corpus of Opinion in Blogs ~ opinion annotated English and Spanish blogs

Commercial use and LDC data

For-profit organizations are reminded that an LDC membership is a pre-requisite for obtaining a commercial license to almost all LDC databases.  Non-member organizations, including non-member for-profit organizations, cannot use LDC data to develop or test products for commercialization, nor can they use LDC data in any commercial product or for any commercial purpose.  LDC data users should consult corpus-specific license agreements for limitations on the use of certain corpora. Visit our Licensing page for further information.

Spring 2016 Data Scholarship Program

Applications are now being accepted through Friday, January 15, 2016 for the Spring 2016 LDC Data Scholarship program. The LDC Data Scholarship program provides university students with access to LDC data at no-cost.  This program is open to students pursuing both undergraduate and graduate studies in an accredited college or university. LDC Data Scholarships are not restricted to any particular field of study; however, students must demonstrate a well-developed research agenda and a bona fide inability to pay.

The application consists of two parts:

(1) Data Use Proposal. Applicants must submit a proposal describing their intended use of the data. The proposal should state which data the student plans to use and how the data will benefit their research project as well as information on the proposed methodology or algorithm.

(2) Letter of Support. Applicants must submit one letter of support from their thesis adviser or department chair. The letter must verify the student's need for data and confirm that the department or university lacks the funding to pay the full nonmember fee for the data or to join the Consortium.

For further information on application materials and program rules, please visit the LDC Data Scholarship page.

Students can email their applications to the LDC Data Scholarship program. Decisions will be sent by email from the same address.

The deadline for the Spring 2016 program cycle is January 15, 2016.

LDC closed for Thanksgiving Break
LDC will be closed on Thursday, November 26, 2015 and Friday, November 27, 2015 in observance of the US Thanksgiving Holiday.  Our offices will reopen on Monday, November 30, 2015.

New publications
(1) Articulation Index LSCP was developed by researchers at Laboratoire de Sciences Cognitives et Psycholinguistique (LSCP), Ecole Normale Supérieure. It revises and enhances a subset of Articulation Index (AIC) (LDC2005S22), a corpus of persons speaking English syllables. Changes include the addition of forced alignment to sound files, time alignment of syllable utterances and format conversions. AIC consists of 20 American English speakers (12 males, 8 females) pronouncing syllables, some of which form actual words, but most of which are nonsense syllables. All possible Consonant-Vowel (CV) and Vowel-Consonant (VC) combinations were recorded for each speaker twice, once in isolation and once within a carrier-sentence, for a total of 25768 recorded syllables.

Articulation Index LSCP alters AIC in the following ways.

1. Time-alignments for the onset and offset of each word and syllable were generated through forced-alignment with a standard HMM-GMM (Hidden Markov Model-Gaussian Mixture Model) ASR system.
2. The time-alignments for the beginning and end of the syllables (whether in isolation or within a carrier sentence) were manually adjusted. The time-alignments for the other words in carrier sentences were not manually adjusted
3. The recordings of isolated syllables were cut according to the manual time-alignments to remove the silent portions at the beginning and end, and the time-alignments were altered to correspond to the cut recordings
4. The file naming scheme was slightly altered for compatibility with the Kaldi speech recognition toolkit.de
5. AIC contains a wide-band (16 KHz, 16-bit PCM) and a narrow-band (8 KHz, 8 bit u-law) version of the recordings distributed in sphere format. The LSCP version contains the wide-band version only distributed as wave files.

Articulation Index LSCP is distributed via web download.

2015 Subscription Members will automatically receive two copies of this corpus.  2015 Standard Members may request a copy as part of their 16 free membership corpora.  This data is being made available at no-cost for non-member organizations under a research license.

*

(2) GALE Phase 4 Chinese Newswire Parallel Sentences was developed by LDC. Along with other corpora, the parallel text in this release comprised training data for Phase 4 of the DARPA GALE (Global Autonomous Language Exploitation) Program. This corpus contains Chinese source sentences and corresponding English translations selected from newswire data collected by LDC in 2008 and translated by LDC or under its direction.

GALE Phase 4 Chinese Newswire Parallel Sentences includes 627 source-translation document pairs, comprising 90,434 tokens of Chinese source text and its English translation. Data is drawn from six distinct Chinese newswire sources.

Sentences were selected for translation in two steps. First, files were chosen using sentence selection scripts.  Selected files were reformatted into a human-readable translation format and assigned to translation vendors. Translators followed LDC's Chinese to English translation guidelines and were provided with the full source documents containing the target sentences for their reference. Bilingual LDC staff performed quality control procedures on the completed translations.

GALE Phase 4 Chinese Newswire Parallel Sentences is distributed via web download.

2015 Subscription Members will automatically receive two copies of this corpus.  2015 Standard Members may request a copy as part of their 16 free membership corpora.  Non-members may license this data for a fee.

*

(3) KHATT: Handwritten Arabic Text was developed by King Fahd University of Petroleum & Minerals, Technical University of Dortmund and Braunschweig University of Technology. It is comprised of scanned Arabic handwriting from 1,000 distinct male and female writers representing diverse countries, age groups, handedness and education levels. Participants produced text on a topic of their choice in an unrestricted style. KHATT was designed to promote research in areas such as text recognition and writer identification.

The majority of participants were natives of Saudi Arabia; the next largest group was from a collection of regional countries (Egypt, Jordan, Kuwait, Morocco, Palestine, Tunisia and Yemen). Most writers were between 16-25 years of age with high school or university qualifications.

KHATT: Handwritten Arabic Text is distributed on one USB drive.

2015 Subscription Members will automatically receive a copy of this corpus.  2015 Standard Members may request a copy as part of their 16 free membership corpora.  Non-members may license this data for a fee.

LDC 2014 November Newsletter

Fall 2014 Data Scholarship Recipients

Invitation to Join for Membership Year (MY) 2015

Spring 2015 Data Scholarship Program

LDC is now on Twitter

LDC closed for Thanksgiving Break

New publications:

Boulder Lies and Truth
GALE Chinese-English Word Alignment and Tagging -- Broadcast Training Part 2
GALE Phase 2 Chinese Web Parallel Text

---

Fall 2014 Data Scholarship Recipients

LDC is pleased to announce the student recipients of the Fall 2014 LDC Data Scholarship program.  The following students will receive no-cost copies of LDC data:

Mohammed Abumatar ~ University of Jordan (Jordan), Bsc Candidate, Computer Engineering.  Mohammed has been awarded a copies of MADCAT Phase 1-3 Training Data for his work in handwriting recognition.

Ramy Baly ~ American University of Beirut (Lebanon), PhD candidate, Electrical and Computer Engineering.  Ramy has been awarded a copies of Arabic Treebank Parts 1-3 for his work in opinion mining.

Abbas Khosravanai ~ Amirkabir University of Technology (Iran), PhD candidate, Computer Engineering.  Abbas has been awarded a copy of 2008 NIST Speaker Recognition for his work in robust speaker recognition.

Phuc Nguyen ~ University of North Texas (USA), PhD candidate, Computer Science and Engineering.  Phuc has been awarded a copy of Message Understanding Conference (MUC) 7 for his work in named entity recognition.

Invitation to Join for Membership Year (MY) 2015
Membership Year (MY) 2015 is open for joining.  We would like to invite all current and previous members of LDC to renew their membership as well as welcome new organizations to join the Consortium.  For MY2015, LDC is pleased to maintain membership fees at last year’s rates – membership fees will not increase.  Additionally, LDC will extend discounts on membership fees to members who keep their membership current and who join early in the year.

The details of our early renewal discounts for MY2015 are as follows:

Organizations who joined for MY2014 will receive a 10% discount when renewing before March 2, 2015. After March 2, 2015, MY2014 members are eligible for a 5% discount when renewing through the end of the year.

New members as well as organizations who did not join for MY2014, but who held membership in any of the previous MYs (1993-2013), will also be eligible for a 5% discount provided that they join/renew before March 2, 2015.

Publications for MY2015 are still being planned but we plan to release the following:

• CIEMPIESS - Mexican Spanish radio broadcast audio and transcripts   
• GALE Phase 3 and 4 data – all tasks and languages   
• Mandarin Chinese Phonetic Segmentation and Tone Corpus - phonetic segmentation and tone labels   
• RATS Speech Activity Detection  – multilanguage audio for robust speech detection and language identification
• SEAME - Mandarin-English code-switching speech
• SenSem Spanish and Catalan Lexicon and Databank - sentence semantics and verbal lexicons

Spring 2015 Data Scholarship Program

Applications are now being accepted through Thursday, January 15, 2015, 11:59PM EST for the Spring 2015 LDC Data Scholarship program. The LDC Data Scholarship program provides university students with access to LDC data at no-cost. During previous program cycles, LDC has awarded no-cost copies of LDC data to over 40 individual students and student research groups. This program is open to students pursuing both undergraduate and graduate studies in an accredited college or university. LDC Data Scholarships are not restricted to any particular field of study; however, students must demonstrate a well-developed research agenda and a bona fide inability to pay.

The application consists of two parts:

(1) Data Use Proposal. Applicants must submit a proposal describing their intended use of the data. The proposal should state which data the student plans to use and how the data will benefit their research project as well as information on the proposed methodology or algorithm.

(2) Letter of Support. Applicants must submit one letter of support from their thesis adviser or department chair. The letter must verify the student's need for data and confirm that the department or university lacks the funding to pay the full non-member fee for the data or to join the Consortium.

For further information on application materials and program rules, please visit the LDC Data Scholarship page.

Students can email their applications to the LDC Data Scholarship program. Decisions will be sent by email from the same address.

The deadline for the Spring 2015 program cycle is January 15, 2015, 11:59PM EST.

LDC is now on Twitter
LDC now has a Twitter feed. Start following us today for updates on new corpora releases and the latest LDC news.

LDC closed for Thanksgiving Break
LDC will be closed on Thursday, November 27, 2014 and Friday, November 28, 2013 in observance of the US Thanksgiving Holiday.  Our offices will reopen on Monday, December 1, 2014.


New publications

(1) Boulder Lies and Truth was developed at the University of Colorado Boulder and contains approximately 1,500 elicited English reviews of hotels and electronics for the purpose of studying deception in written language. Reviews were collected by crowd-sourcing with Amazon Medical Turk.

Each review was required to be original and was checked for plagiarism against the web. Reviews were annotated with respect to the following three dimensions:

Domain: Electronics (e.g., iPhone) or Hotels

Sentiment: Positive or Negative

Truth Value:

a) Truthful: a review about an object known by the writer reflecting the real sentiment of the writer toward the object of the review

b) Opposition: A review about an object known by the writer reflecting the opposite sentiment of the writer toward the object of the review (i.e., if the writer liked the object they were asked to write a negative review; if the writer did not like the object, they were asked to write a positive review)

c) Deceptive (i.e., fabricated): a review written about an object not known by the writer either positive or negative in sentiment; the objects reviewed were provided via a URL from the tasks in (a) and (b)

Each review was judged a total of 30 times: (1) 10 times to evaluate its perceived quality (on a range from 1-5); (2) 10 times with judgments about its perceived truthfulness (e.g., truthful or somehow deceptive, a lie or a fabrication); and (3) 10 times for its perceived sentiment (i.e., star rating).

Boulder Lies and Truth is distributed via web download.

2014 Subscription Members will receive two copies of this data on disc, provided they have completed the user license agreement.  2014 Standard Members may request a copy as part of their 16 free membership corpora.  This data is available at no-cost for non-members under the same user license agreement.

*

(2) GALE Chinese-English Word Alignment and Tagging -- Broadcast Training Part 2 was developed by LDC and contains 65,069 tokens of word aligned Chinese and English parallel text enriched with linguistic tags. This material was used as training data in the DARPA GALE (Global Autonomous Language Exploitation) program.

Some approaches to statistical machine translation include the incorporation of linguistic knowledge in word aligned text as a means to improve automatic word alignment and machine translation quality. This is accomplished with two annotation schemes: alignment and tagging. Alignment identifies minimum translation units and translation relations by using minimum-match and attachment annotation approaches. A set of word tags and alignment link tags are designed in the tagging scheme to describe these translation units and relations. Tagging adds contextual, syntactic and language-specific features to the alignment annotation.

This release consists of Chinese source broadcast conversation (BC) programming collected by LDC in 2008.

The Chinese word alignment tasks consisted of the following components:

Identifying, aligning, and tagging eight different types of links

Identifying, attaching, and tagging local-level unmatched words

Identifying and tagging sentence/discourse-level unmatched words

Identifying and tagging all instances of Chinese 的(DE) except when they were a part of a semantic link

GALE Chinese-English Word Alignment and Tagging -- Broadcast Training Part 2 is distributed via web download.

2014 Subscription Members will automatically receive two copies of this data on disc.  2014 Standard Members may request a copy as part of their 16 free membership corpora.  Non-members may license this data for a fee.

*

(3) GALE Phase 2 Chinese Web Parallel Text was developed by LDC and along with other corpora, the parallel text in this release comprised training data for Phase 2 of the DARPA GALE (Global Autonomous Language Exploitation) Program. This corpus contains Chinese source text and corresponding English translations selected from weblog and newsgroup data collected by LDC and translated by LDC or under its direction.

This release includes 46 source-translation document pairs, comprising 66,779 tokens of translated data. Data is drawn from four Chinese weblog and newsgroup sources.

Data was manually selected for translation according to several criteria, including linguistic features and topic features. The files were formatted into a human-readable translation format and assigned to translation vendors. Translators followed LDC's Chinese to English translation guidelines. Bilingual LDC staff performed quality control procedures on the completed translations.

GALE Phase 2 Chinese Web Parallel Text is distributed via web download.

2014 Subscription Members will automatically receive two copies of this data on disc.  2014 Standard Members may request a copy as part of their 16 free membership corpora.  Non-members may license this data for a fee.

LDC February 2014 Newsletter

Spring 2014 LDC Data Scholarship recipients

Membership fee savings and publications pipeline

New LDC website enhancements coming soon

New publications:

GALE Arabic-English Parallel Aligned Treebank -- Broadcast News Part 2

King Saud University Arabic Speech Database

NIST 2012 Open Machine Translation (OpenMT) Progress Test Five Language Source

Spring 2014 LDC Data Scholarship recipients

LDC is pleased to announce the student recipients of the Spring 2014 LDC Data Scholarship program!  This program provides university students with access to LDC data at no-cost. Students were asked to complete an application which consisted of a proposal describing their intended use of the data, as well as a letter of support from their thesis adviser. We received many solid applications and have chosen two proposals to support. The following students will receive no-cost copies of LDC data:

• Skye Anderson ~ Tulane University (USA), BA candidate, Linguistics.  Skye has been awarded a copy of LDC Standard Arabic Morphological Analyzer (SAMA) Version 3.1 for her work in author profiling.
  
  
• Hao Liu ~ University College London (UK), PhD candidate, Speech, Hearing and Phonetic Sciences.  Hao has been awarded a copy of Switchboard-1 Release 2, and NXT Switchboard Annotations for his work in prosody modeling.

Membership fee savings and publications pipeline

Members can still save on 2014 membership fees, but time is running out. Any organization which joins or renews membership for 2014 through Monday, March 3, 2014, is entitled to a 5% discount. Organizations which held membership for MY2013 can receive a 10% discount on fees provided they renew prior to March 3, 2014.

Planned publications for this year include:

• 2009 NIST Language Recognition Evaluation ~  development data from VOA broadcast and CTS telephone speech in target and non-target languages.
• ETS Corpus of Non-Native Written English ~ contains 1100 essays written for a college-entrance test sampled from eight prompts (i.e., topics)  with score levels (low/medium/high) for each essay.
• GALE data ~ including Word Alignment, Broadcast Speech & Transcripts, Parallel Text, Parallel Aligned Treebanks in Arabic, Chinese, and English.
  
  
• Hispanic Accented English ~ contains approximately 30 hours of spontaneous speech and read utterances from non-native speakers of English with corresponding transcripts.
• Multi-Channel Wall Street Journal Audio-Visual Corpus (MC-WSJ-AV) ~  re-recording of parts of the WSJCAM0 using a number of microphones as well as three recording conditions resulting in 18-20 channels of audio per recording.
• TAC KBP Reference Knowledge Base ~ TAC KBP aims to develop and evaluate technologies for building and populating knowledge bases (KBs) about named entities from unstructured text.  KBP systems must either populate an existing reference KB, or else build a KB from scratch. The reference KB for is based on a snapshot of English Wikipedia snapshot from October 2008 and contains a set of entities, each with a canonical name and title for the Wikipedia page, an entity type, an automatically parsed version of the data from the infobox in the entity's Wikipedia article, and a stripped version of the text of the Wiki article.
• USC-SFI MALACH Interviews and Transcripts Czech ~ developed by The University of Southern California's Shoah Foundation Institute (USC-SFI) and the University of West Bohemia as part of the MALACH (Multilingual Access to Large Spoken ArCHives) Project. It contains approximately 143 hours of interviews from 420 interviewees along with transcripts and other documentation.

New LDC website enhancements coming soon
Look for LDC’s new website enhancements in the coming weeks. We've revamped our membership services to make it easier than ever for you to manage your membership and access data more quickly.

New publications

(1) GALEArabic-English Parallel Aligned Treebank -- Broadcast News Part 2 was developed by LDC and contains 141,058 tokens of word aligned Arabic and English parallel text with treebank annotations. This material was used as training data in the DARPA GALE (Global Autonomous Language Exploitation) program.

Parallel aligned treebanks are treebanks annotated with morphological and syntactic structures aligned at the sentence level and the sub-sentence level. Such data sets are useful for natural language processing and related fields, including automatic word alignment system training and evaluation, transfer-rule extraction, word sense disambiguation, translation lexicon extraction and cultural heritage and cross-linguistic studies. With respect to machine translation system development, parallel aligned treebanks may improve system performance with enhanced syntactic parsers, better rules and knowledge about language pairs and reduced word error rate.

In this release, the source Arabic data was translated into English. Arabic and English treebank annotations were performed independently. The parallel texts were then word aligned. The material in this corpus corresponds to a portion of the Arabic treebanked data in Arabic Treebank - Broadcast News v1.0 (LDC2012T07).

The source data consists of Arabic broadcast news programming collected by LDC in 2007 and 2008. All data is encoded as UTF-8. A count of files, words, tokens and segments is below.

Language	Files	Words	Tokens	Segments
Arabic	31	110,690	141,058	7,102

The purpose of the GALE word alignment task was to find correspondences between words, phrases or groups of words in a set of parallel texts. Arabic-English word alignment annotation consisted of the following tasks:

• Identifying different types of links: translated (correct or incorrect) and not translated (correct or incorrect)
• Identifying sentence segments not suitable for annotation, e.g., blank segments, incorrectly-segmented segments, segments with foreign languages
• Tagging unmatched words attached to other words or phrases

GALE Arabic-English Parallel Aligned Treebank -- Broadcast News Part 2 is distributed via web download.

2014 Subscription Members will automatically receive two copies of this data on disc. 2014 Standard Members may request a copy as part of their 16 free membership corpora.  Non-members may license this data for a fee.

*

(2) King Saud University Arabic Speech Database was developed by King Saud University and contains 590 hours of recorded Arabic speech from male and female speakers. The utterances include read and spontaneous speech. The recordings were conducted in varied environments representing quiet and noisy settings.

The corpus was designed principally for speaker recognition research. The speech sources are sentences, word lists, prose and question and answer sessions. Read speech text includes the following:

• Sets of sentences devised to cover allophones of each phoneme, phonetic balance, and differentiation of accents.
• Word lists developed to minimize missing phonemes and to represent nasals fricatives, commonly used words, and numbers.
• Two paragraphs, one from the Quran and another from a book, selected because they included all letters of the alphabet and were easy to read.

Spontaneous speech was captured through question and answer sessions between participants and project team members. Speakers responded to questions on general topics such as the weather and food.

Each speaker was recorded in three different environments: a sound proof room, an office, and a cafeteria. The recordings were collected via microphone and mobile phone and averaged between 16-19 minutes. The data was verified for missing recordings, problems with the recording system or errors in the recording process.

King Saud University Arabic Speech Database is distributed on one hard disk.

2014 Subscription Members will receive a copy of this data provided that they have completed the User License Agreement. 2014 Standard Members may request a copy as part of their 16 free membership corpora.  Non-members may license this data for a fee.

*

(3) NIST2012 Open Machine Translation (OpenMT) Progress Test Five Language Source was developed by NIST Multimodal Information Group. This release contains the evaluation sets (source data and human reference translations), DTD, scoring software, and evaluation plan for the OpenMT 2012 test for Arabic, Chinese, Dari, Farsi, and Korean to English on a parallel data set. The set is based on a subset of the Arabic-to-English and Chinese-to-English progress tests from the OpenMT 2008, 2009 and 2012 evaluations with new source data created by humans based on the English reference translation. The package was compiled, and scoring software was developed, at NIST, making use of newswire and web data and reference translations developed by the Linguistic Data Consortium  and the Defense Language Institute Foreign Language Center.

The objective of the OpenMT evaluation series is to support research in, and help advance the state of the art of, machine translation (MT) technologies -- technologies that translate text between human languages. Input may include all forms of text. The goal is for the output to be an adequate and fluent translation of the original. The 2012 task included the evaluation of five language pairs: Arabic-to-English, Chinese-to-English, Dari-to-English, Farsi-to-English and Korean-to-English in two source data styles. For general information about the NIST OpenMT evaluations, refer to the NIST OpenMT website.

This evaluation kit includes a single Perl script (mteval-v13a.pl) that may be used to produce a translation quality score for one (or more) MT systems. The script works by comparing the system output translation with a set of (expert) reference translations of the same source text. Comparison is based on finding sequences of words in the reference translations that match word sequences in the system output translation.

This release consists of 20 files, four for each of the five languages, presented in XML with an included DTD. The four files are source and reference data in the following two styles:

• English-true: an English-oriented translation this requires that the text read well and not use any idiomatic expressions in the foreign language to convey meaning, unless absolutely necessary.
• Foreign-true: a translation as close as possible to the foreign language, as if the text had originated in that language.

NIST 2012 Open Machine Translation (OpenMT) Progress Test Five Language Source is distributed via web download.

2014 Subscription Members will automatically receive two copies of this data on disc. 2014 Standard Members may request a copy as part of their 16 free membership corpora.  Non-members may license this data for a fee.