LDC December 2012 Newsletter

Spring 2013 LDC Data Scholarship Program - deadline approaching!

Two New LDC Podcasts for your Listening Pleasure 

Penn Discourse Treebank Version 2.0 Update
LDC to close for Winter Break

New publications:

GALE Chinese-English Word Alignment and Tagging Training Part 3 -- Web

Russian-English Computer Security Parallel Text

---

 Spring 2013 LDC Data Scholarship Program - deadline approaching!

The deadline for the Spring 2013 LDC Data Scholarship Program is one month away!   Student applications are being accepted now through January 15, 2013, 11:59PM EST.  The LDC Data Scholarship program provides university students with access to LDC data at no cost.  This program is open to students pursuing both undergraduate and graduate studies in an accredited college or university. LDC Data Scholarships are not restricted to any particular field of study; however, students must demonstrate a well-developed research agenda and a bona fide inability to pay. 

Students will need to complete an application which consists of a data use proposal and letter of support from their adviser.  For further information on application materials and program rules, please visit the LDC Data Scholarship page.  

Students can email their applications to the LDC Data Scholarship program. Decisions will be sent by email from the same address.

Two New LDC Podcasts for your Listening Pleasure

Two new podcasts are available on a the LDC blog continuing the  20^th Anniversary series. The first features Natalia Bragilevskaya, LDC’s Business Administrator, Membership Coordinator Ilya Ahtaridis and Marian Reed, Marketing Coordinator. They recall the early days of LDC and describe the growth of sponsored projects work and LDC’s interactions with its membership.

Click here for Natalia, Ilya and Marian’s podcast.

The third podcast in the series introduces the community to two  LDC, researchers Yiwola Awoyale and Moussa Bamba, whose work focuses on West African languages. 

Yiwola has been teaching Linguistics, Yoruba language studies and various aspects of African linguistics since 1975. At LDC, he developed the Global Yoruba Lexical Database, a set of related dictionaries based on Yoruba and its diaspora. Moussa’s work in the Manding languages of the Niger-Congo family has resulted in the release of the Mawukakan Lexicon, to be followed by similar resources for Maninkakan, Bambara, and Jula. 

In their podcast, Yiwola and Moussa discuss how they came  to LDC, their current research and how it benefits multiple communities. Click here for Yiwola and Moussa’s podcast. 

Other podcasts will be published via the LDC blog, so stay tuned to that space.

Penn Discourse Treebank Version 2.0 Update

The developers of the Penn Discourse Treebank Version 2.0 LDC2008T05 (PDTB) have updated this release to add metadata to the Wall Street Journal (WSJ) news stories in the corpus. The goal is to aid understanding PDTB files as texts and to support distinguishing texts from different genres within the WSJ. 

The metadata includes the following fields:

• DD: the date the article appeared in the WSJ
• AN: unique identifier for the article
• HL: the column name (for regular features such as Who's News, Marketing & Media, Technology), its headline and by-line
• SO: the source of the article
• IN: manually-assigned codes or keywords for the article
• CO: manually-assigned codes for companies or other organizations
• DATELINE: normally the location where the article was filed, but sometimes has very unexpected contents
• GV: Branch of Government or Government Agency mentioned in the article
• SBREAKS: the byte position of section breaks present in the file
• ARTICLEBREAK: separates files that contain more than one article

All new downloads of PDTB will contain the complete updated corpus.  Current PDTB licensees can re-download the file to obtain the updated data. 

LDC to close for Winter Break

LDC will be closed from Monday, December 24, 2012 through Tuesday, January 1, 2013 in accordance with the University of Pennsylvania Winter Break Policy. Our offices will reopen on Wednesday, January 2, 2013. Requests received for membership renewals and corpora during the Winter Break will be processed at that time.

Best wishes for a happy and safe holiday season!

New publications

(1) GALE Chinese-English Word Alignment and Tagging Training Part 3 -- Web was developed by LDC and contains 154,541 tokens of word aligned Chinese and English parallel text enriched with linguistic tags. This material was used as training data in the DARPA GALE (Global Autonomous Language Exploitation) program.

Some approaches to statistical machine translation include the incorporation of linguistic knowledge in word aligned text as a means to improve automatic word alignment and machine translation quality. This is accomplished with two annotation schemes: alignment and tagging. Alignment identifies minimum translation units and translation relations by using minimum-match and attachment annotation approaches. A set of word tags and alignment link tags are designed in the tagging scheme to describe these translation units and relations. Tagging adds contextual, syntactic and language-specific features to the alignment annotation. 

GALE Chinese-English Word Alignment and Tagging Training Part 1 -- Newswire and Web (LDC2012T16) and GALE Chinese-English Word Alignment and Tagging Training Part 3 -- Web (LDC2012T20) are also available through LDC.

This release consists of Chinese source web data (newsgroup, weblog) collected by LDC in 2008 and 2009. The distribution by words, character tokens and segments appears below: 

Language: Chinese

Files: 1249

Words: 103027

CharTokens: 154541 

Segments: 4842

Note that all token counts are based on the Chinese data only. One token is equivalent to one character and one word is equivalent to 1.5 characters.

The Chinese word alignment tasks consisted of the following components:

• Identifying, aligning, and tagging 8 different types of links
• Identifying, attaching, and tagging local-level unmatched words
• Identifying and tagging sentence/discourse-level unmatched words
• Identifying and tagging all instances of Chinese 的(DE) except when they were a part of a semantic link.

GALE Chinese-English Word Alignment and Tagging Training Part 3 -- Web is distributed via web download.2012 Subscription Members will automatically receive two copies of this data on disc. 2012 Standard Members may request a copy as part of their 16 free membership corpora. Non-members may license this data for US$1750.

*

(2) Russian-English Computer Security Parallel Text was developed by The MITRE Corporation. It consists of parallel sentences from a set of computer security reports published in Russian and translated into English by translators with particular expertise in the technical area. Translators were instructed to err on the side of literal translation if required, but to maintain the technical writing style of the source and to make the resulting English as natural as possible. The translators followed specific guidelines for translation, and those are included in this distribution.

There are 6,276 lines of parallel Russian and English, with a total of 60,059 words of Russian and 76,437 words of English, presented in a separate UTF-8 plain text file for each language. The sentences were translated in sequential order and presented in a scrambled order, such that parallel sentences at identical line numbers are translations. For example, the 31st line of the English file is a translation of the 31st line of the Russian file. The original line sequence is not provided. 1,694 untranslated lines (such as code snippets) are included as a separate file.

Russian-English Computer Security Parallel Text is distributed via web download. 2012 Subscription Members will automatically receive two copies of this data on disc. 2012 Standard Members may request a copy as part of their 16 free membership corpora.  Non-members may license this data for US$1500.

LDC 20th Anniversary Podcasts: Yiwola Awoyale and Moussa Bamba

The third podcast in the series shifts gears and introduces two LDC researchers, Yiwola Awoyale and Moussa Bamba, whose work focuses on West African languages. 

Yiwola has been teaching linguistics, Yoruba language studies and various aspects of African linguistics since 1975. At LDC, he developed the Global Yoruba Lexical Database, a set of related dictionaries based on Yoruba and its diaspora. Moussa’s work in the Manding languages of the Niger-Congo family has resulted in the release of the Mawukakan Lexicon, to be followed by similar resources for Maninkakan, Bambara, and Jula. 

In their podcast, Yiwola and Moussa discuss how they came to LDC, their current research and how it benefits multiple communities.

Click here for Yiwola and Moussa's podcast.

LDC 20th Anniversary Podcasts: Natalia Bragilevskaya, Ilya Ahtaridis and Marian Reed

LDC has put together a second podcast for your listening enjoyment. This issue builds upon the origin of LDC while focusing on LDC's interactions with the outside world. Natalia is the head of LDC's Business Office while Ilya and Marian are the twin engines of LDC's External Relations group (LDC's Membership Coordinator and Marketing Coordinator, respectively). As was the case with David Graff's October 2012 podcast, John Vogel conducted each of these interviews.

As a reminder, LDC podcasts are issued as part of our celebration of our 20th Anniversary year. Please stay tuned for later editions.

Click here for Natasha, Ilya and Marian's podcast.  

LDC November 2012 Newsletter

Spring 2013 LDC Data Scholarship Program

Invitation to Join for Membership Year 2013

Why become an LDC member?

2012 User Survey Results
LDC to Close for Thanksgiving Break

New publications:

Annotated English Gigaword

Chinese-English Semiconductor Parallel Text

GALE Phase 2 Arabic Newswire Parallel Text

---

Spring 2013 LDC Data Scholarship Program

Applications are now being accepted through January 15, 2013, 11:59PM EST for the Spring 2013 LDC Data Scholarship program! The LDC Data Scholarship program provides university students with access to LDC data at no-cost. During previous program cycles, LDC has awarded no-cost copies of LDC data to over 25 individual students and student research groups.

This program is open to students pursuing both undergraduate and graduate studies in an accredited college or university. LDC Data Scholarships are not restricted to any particular field of study; however, students must demonstrate a well-developed research agenda and a bona fide inability to pay. The selection process is highly competitive.

The application consists of two parts:

(1)  Data Use Proposal. Applicants must submit a proposal describing their intended use of the data. The proposal should state which data the student plans to use and how the data will benefit their research project as well as information on the proposed methodology or algorithm.

Applicants should consult the LDC Corpus Catalog for a complete list of data distributed by LDC. Due to certain restrictions, a handful of LDC corpora are restricted to members of the Consortium. Applicants are advised to select a maximum of one to two datasets; students may apply for additional datasets during the following cycle once they have completed processing of the initial datasets and publish or present work in some juried venue.

(2)  Letter of Support. Applicants must submit one letter of support from their thesis adviser or department chair. The letter must verify the student's need for data and confirm that the department or university lacks the funding to pay the full Non-member Fee for the data or to join the consortium.

For further information on application materials and program rules, please visit the LDC Data Scholarship page. Students can email their applications to the LDC Data Scholarship program. Decisions will be sent by email from the same address. The deadline for the Spring 2013 program cycle is January 15, 2013, 11:59 PM EST.

Invitation to Join for Membership Year 2013

Membership Year (MY) 2013 is open for joining! We would like to invite all current and previous members of LDC to renew their membership as well as welcome new organizations to join the consortium. For MY2013, LDC is pleased to maintain membership fees at last year’s rates – membership fees will not increase. 

Additionally, LDC will extend discounts on membership fees to members who keep their membership current and who join early in the year.

The details of our early renewal discounts for MY2013 are as follows:

    ·  Organizations who joined for MY2012 will receive a 5% discount when renewing. This discount will apply throughout 2013, regardless of time of renewal. MY2012 members renewing before March 1, 2013 will receive an additional 5% discount, for a total 10% discount off the membership fee.

    ·  New members as well as organizations who did not join for MY2012, but who held membership in any of the previous MYs (1993-2011), will also be eligible for a 5% discount provided that they join/renew before March 1, 2013.

The following table provides exact pricing information.

		MY2013 Fee	MY2013 Fee with 5% Discount*	MY2013 Fee with 10% Discount**
Not-for-Profit /US Government
	Standard	US$2400	US$2280	US$2160
	Subscription	US$3850	US$3658	US$3465
For-Profit
	Standard	US$24000	US$22800	US$21600
	Subscription	US$27500	US$26125	US$24750

*  For new members, MY2012 Members renewing for MY2013, and any previous year Member who renews before March 1, 2013

** For MY2012 Members renewing before March 1, 2013


Publications for MY2013 are still being planned; here are the working titles of data sets we intend to provide:

-      Arabic Treenbank – Weblog

-      Chinese-English Biomedical Parallel Text

-      GALE data – all phases and tasks

-      Hispanic-English Speech

-      Maninkakan Lexicon

-      OpenMT 2008-2012 Progress set

In addition to receiving new publications, current year members of the LDC also enjoy the benefit of licensing older data at reduced costs; current year for-profit members may use most data for commercial applications.

This past year, LDC members who joined early or kept their membership current saved almost US$70,000 collectively on membership fees. Be sure to keep an eye on your mail - all previous and current LDC members will be sent an invitation to join letter and renewal invoice for MY2013. Renew early for MY2013 to save today!

Why become an LDC member?

LDC is offering early renewal discounts on membership fees for Membership Year 2013 making now a good time to consider joining or renewing membership. LDC membership has the following advantages:

• LDC membership provides cost-effective access to an extensive and growing catalog that spans 20 years and includes over 500 multilingual speech, text, and video resources. Even if your organization only needs a few datasets from a given membership year, membership is often the most economical way to obtain current corpora. Additionally, the generous discounts that member organizations receive on older corpora reduce the cost of acquiring such datasets.

• All members enjoy unlimited use of LDC data within their organizations.  For universities, there is no difference in cost between a departmental membership and one that is university-wide. Departments can therefore combine resources and establish one LDC membership for use by the entire university community.  Likewise, for-profit members with multiple branches can maintain one membership for use by their entire organizations.

For-profit organizations are reminded that an LDC membership is a pre-requisite for obtaining a commercial license to almost all LDC databases. Non-member organizations, including non-member for-profit organizations, cannot use LDC data to develop or test products for commercialization, nor can they use LDC data in any commercial product or for any commercial purpose. LDC data users should consult corpus-specific license agreements for limitations, including commercial restrictions, on the use of certain corpora. In the case of a small group of corpora, commercial licenses must be obtained separately from the owners of the data.

2012 User Survey Results

 Earlier this year, LDC sent a survey to its user communities. Like previous iterations in 2006 and 2007, the survey solicited community input and suggestions on key LDC-related topics, including:

-      Satisfaction levels with LDC’s data, homepage and Catalog

-      Reflections on LDC’s 20^th Anniversary year

-      Suggestions for future publications

-      Speculations on the future of HLT-related fields, specifically on mobile technologies, cloud computing, social networking and open data

Survey respondents were generally satisfied with LDC’s data, membership options, homepage and Catalog, though there were requests for additional data options and data acquisition methods. Some of the data respondents requested are already in our pipeline for the end of 2012 or for Membership Year (MY) 2013, so please be on the lookout for Publications updates.

Respondents were also very supportive of LDC’s 20^th Anniversary, posting testimonials and well-wishes in the 20^th Anniversary section.

LDC would like to thank all survey participants. Survey participants will receive access to full survey results shortly.

LDC to Close for Thanksgiving Break

LDC will be closed on Thursday, November 22, 2012 and Friday, November 23, 2012 in observance of the US Thanksgiving Holiday.  Our offices will reopen on Monday, November 26, 2012.

New publications

(1) Annotated English Gigaword was developed by Johns Hopkins University's Human Language Technology Center of Excellence. It adds automatically-generated syntactic and discourse structure annotation to English Gigaword Fifth Edition (LDC2011T07) and also contains an API and tools for reading the dataset's XML files. The goal of the annotation is to provide a standardized corpus for knowledge extraction and distributional semantics which enables broader involvement in large-scale knowledge-acquisition efforts by researchers.

Annotated English Gigaword contains the nearly ten million documents (over four billion words) of the original English Gigaword Fifth Edition from seven news sources:

• Agence France-Presse, English Service (afp_eng)
• Associated Press Worldstream, English Service (apw_eng)
• Central News Agency of Taiwan, English Service (cna_eng)
• Los Angeles Times/Washington Post Newswire Service (ltw_eng)
• Washington Post/Bloomberg Newswire Service (wpb_eng)
• New York Times Newswire Service (nyt_eng)
• Xinhua News Agency, English Service (xin_eng)

The following layers of annotation were added:

• Tokenized and segmented sentences
• Treebank-style constituent parse trees
• Syntactic dependency trees
• Named entities
• In-document coreference chains

The annotation was performed in a three-step process: (1) the data was preprocessed and sentences selected for annotation (sentences with more than 100 tokens were excluded); (2) syntactic parses were derived; and (3) the parsed output was post-processed to derive syntactic dependencies, named entities and coreference chains. Over 183 million sentences were parsed.

Annotated English Gigaword is distributed on one hard drive. 2012 Subscription Members will automatically receive one copy of this data on hard drive.  2012 Standard Members may request a copy as part of their 16 free membership corpora. 2011 Members who licensed English Gigaword Fifth Edition (LDC2011T07) may request a no-cost copy of Annotated English Gigaword. Non-member organizations who licensed English Gigaword Fifth Edition may request a copy of Annotated English Gigaword for the US$200 media fee.

*

(2) Chinese-English Semiconductor Parallel Text was developed by The MITRE Corporation. It consists of parallel sentences from a collection of abstracts from scientific articles on semiconductors published in Mandarin and translated into English by translators with particular expertise in the technical area. Translators were instructed to err on the side of literal translation if required, but to maintain the technical writing style of the source and to make the resulting English as natural as possible. The translators followed specific guidelines for translation, and those are included in this distribution.

There are 2,169 lines of parallel Mandarin and English, with a total of 125,302 characters of Mandarin and 64,851 words of English, presented in a separate UTF-8 plain text file for each language. The sentences were translated in sequential order and presented in a scrambled order, such that parallel sentences at identical line numbers are translations. For example, the 31st line of the English file is a translation of the 31st line of the Mandarin file. The original line sequence is not provided.

Chinese-English Semiconductor Parallel Text is distributed via web download.

2012 Subscription Members will automatically receive two copies of this data on disc. 2012 Standard Members may request a copy as part of their 16 free membership corpora.

*

(3) GALE Phase 2 Arabic Newswire Parallel Text was developed by LDC.  Along with other corpora, the parallel text in this release comprised training data for Phase 2 of the DARPA GALE (Global Autonomous Language Exploitation) Program. This corpus contains Modern Standard Arabic source text and corresponding English translations selected from newswire data collected in 2007 by LDC and transcribed by LDC or under its direction.

GALE Phase 2 Arabic Newswire Parallel Text includes 400 source-translation pairs, comprising 181,704 tokens of Arabic source text and its English translation. Data is drawn from six distinct Arabic newswire sources: Al Ahram, Al Hayat, Al-Quds Al-Arabi, An Nahar, Asharq Al-Awsat and Assabah.

The files in this release were transcribed by LDC staff and/or transcription vendors under contract to LDC in accordance with the Quick Rich Transcription guidelines developed by LDC. Transcribers indicated sentence boundaries in addition to transcribing the text. Data was manually selected for translation according to several criteria, including linguistic features, transcription features and topic features. The transcribed and segmented files were then reformatted into a human-readable translation format and assigned to translation vendors. Translators followed LDC's Arabic to English translation guidelines. Bilingual LDC staff performed quality control procedures on the completed translations.

GALE Phase 2 Arabic Newswire Parallel Text is distributed via web download.

2012 Subscription Members will automatically receive two copies of this data on disc. 2012 Standard Members may request a copy as part of their 16 free membership corpora.