Skip to content

Commit cddb958

Browse files
Huang17Huang17
authored
Merge pull request #28 from ryskina/master
Updated coreference resolution and word segmentation
2 parents 764288f + f34ed5e commit cddb958

File tree

2 files changed

+60
-26
lines changed

2 files changed

+60
-26
lines changed

‎docs/co-reference_resolution.md

Lines changed: 32 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -24,13 +24,12 @@ Average of F1-scores returned by these three precision/recall metrics:
2424
- B-cubed.
2525
- Entity-based CEAF.
2626

27-
2827
## <span class="t">CoNLL 2012 Co-reference task</span>.
2928

3029
CoNLL 2012 introduced a co-reference task in Chinese.
3130
- http://conll.cemantix.org/2012/introduction.html
3231

33-
Data for this evaluation is part of Ontonotes, distributed by the Linguistic Data Consortium (LDC).
32+
Data for this evaluation is part of OntoNotes, distributed by the Linguistic Data Consortium (LDC).
3433
- https://catalog.ldc.upenn.edu/LDC2013T19
3534

3635
| Test set | # of co-referring mentions | Genre |
@@ -47,16 +46,44 @@ Scoring code: https://github.com/conll/reference-coreference-scorers
4746

4847
| System | Average F1 of MUC, B-cubed, CEAF |
4948
| --- | --- |
50-
| [[Clark & Manning, 2016](https://nlp.stanford.edu/static/pubs/clark2016deep.pdf)] | 63.88 |
51-
| [[Clark & Manning, 2016](https://nlp.stanford.edu/static/pubs/clark2016improving.pdf)] | 63.66 |
49+
| [Kong & Jian (2019)](https://www.ijcai.org/Proceedings/2019/700) | 63.85 |
50+
| [Clark & Manning (2016b)](https://nlp.stanford.edu/static/pubs/clark2016deep.pdf) | 63.88 |
51+
| [Clark & Manning (2016a)](https://nlp.stanford.edu/static/pubs/clark2016improving.pdf) | 63.66 |
5252

5353
### Resources
5454

55-
Data for this evaluation is part of Ontonotes, distributed by the Linguistic Data Consortium (LDC).
55+
Data for this evaluation is part of OntoNotes, distributed by the Linguistic Data Consortium (LDC).
5656
- https://catalog.ldc.upenn.edu/LDC2013T19
5757

5858
---
5959

60+
## <span class="t">Subtask: zero pronoun resolution (CoNLL 2012 / OntoNotes 5.0) </span>.
61+
62+
### Metrics
63+
64+
F1 score computed on resolution hits ([Zhao & Ng 2007](https://www.aclweb.org/anthology/D07-1057.pdf)).
65+
66+
### Results
67+
68+
| System | Overall F1 (w/ gold syntactic info) | Overall F1 (w/o gold syntactic info) |
69+
| --- | --- | --- |
70+
| [Aloraini & Poesio (2020)](https://www.aclweb.org/anthology/2020.lrec-1.11/) | 63.5 | |
71+
| [Song et al. (2020)](https://www.aclweb.org/anthology/2020.acl-main.482/) | 58.5 | 26.1 |
72+
| [Yang et al. (2019)](https://www.aclweb.org/anthology/W19-4108/) | 58.1 | |
73+
| [Yin et al. (2018)](https://www.aclweb.org/anthology/C18-1002/) | 57.3 | |
74+
| [Liu et al. (2017)](https://www.aclweb.org/anthology/P17-1010/) | 55.3 | |
75+
| [Yin et al. (2017)](https://www.aclweb.org/anthology/D17-1135/) | 54.9 | 22.7 |
76+
77+
### Resources
78+
79+
Training and testing is performed on the train and dev splits of OntoNotes 5.0 respectively (statistics reported by [Yin et al. (2018)](https://www.aclweb.org/anthology/C18-1002/))
80+
81+
| Split | Documents | Sentences | Words | Anaphoric Zero Pronouns |
82+
| --- | --- | --- | --- | --- |
83+
| Train | 1,391 | 36,487 | 756K | 12,111 |
84+
| Dev | 172 | 6,083 | 110K | 1,713 |
85+
86+
6087
**Suggestions? Changes? Please send email to [[email protected]](mailto:[email protected])**
6188

6289

‎docs/word_segmentation.md

Lines changed: 28 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -52,11 +52,14 @@ F1 = 0.857
5252

5353
| Model | AS | CITYU | MSR | PKU |
5454
| --- | --- | --- | --- | --- |
55-
| [Meng et al. (2019)](https://arxiv.org/pdf/1901.10125.pdf) | 96.7 | 97.9 | 98.3 | 96.7 |
56-
|[Huang et al. (2019)](https://arxiv.org/pdf/1903.04190.pdf)|96.6|97.6|97.9|96.6|
55+
| [Tian, Song, Xia, Zhang, Wang (2020)](https://www.aclweb.org/anthology/2020.acl-main.734/) | 96.6 | 97.9 | 98.4 | 96.5 |
56+
| [Meng et al. (2019)](https://arxiv.org/abs/1901.10125) | 96.7<sup>*</sup> | 97.9<sup>*</sup> | 98.3 | 96.7 |
57+
| [Huang et al. (2019)](https://arxiv.org/abs/1903.04190)| 96.6 | 97.6 | 97.9 | 96.6 |
5758
| [Ma et al. (2018)](http://aclweb.org/anthology/D18-1529) | 96.2 | 97.2 | 97.4 | 96.1 |
5859
| [Yang et al. (2017)](http://aclweb.org/anthology/P17-1078) | 95.7 | 96.9 | 97.5 | 96.3 |
59-
| [Zhou et al. (2017)](https://www.aclweb.org/anthology/D17-1079) | | | 97.8 | 96 |
60+
| [Zhou et al. (2017)](https://www.aclweb.org/anthology/D17-1079) | | | 97.8 | 96.0 |
61+
62+
<sup>*</sup> Unlike others, [Meng et al. (2019)](https://arxiv.org/pdf/1901.10125.pdf) do not report converting traditional Chinese to simplified Chinese.
6063

6164
### Resources
6265

@@ -70,36 +73,40 @@ F1 = 0.857
7073

7174
## <span class="t">Chinese Penn Treebank</span>.
7275

73-
* [Website](https://verbs.colorado.edu/chinese/ctb.html)
74-
* Includes 2 datasets:
76+
* [Website](https://www.cs.brandeis.edu/~clp/ctb/)
77+
* Includes 3 datasets:
7578
* [CTB6](https://catalog.ldc.upenn.edu/LDC2007T36): consisting of 780,000 words (over 1.28 million Chinese characters)
7679
* [CTB7](https://catalog.ldc.upenn.edu/LDC2010T07): consists of 2,448 text files, 51,447 sentences, 1,196,329 words and 1,931,381 hanzi (Chinese characters)
77-
80+
* [CTB9](https://catalog.ldc.upenn.edu/LDC2016T13): consists of 3,726 text files, 132,076 sentences, 2,084,387 words, 3,247,331 characters (hanzi or foreign)
7881

7982
|Data set|Test set (Tokens)|
8083
| ---: | ---: |
81-
|CTB6|81,578|
82-
|CTB7|81,578|
84+
|CTB6|82K|
85+
|CTB7|245K|
86+
|CTB9|242K|
8387

8488
### Results
8589

86-
| Model | CTB6 | CBT7 |
87-
| --- | --- | --- |
88-
|[Huang et al. (2019)](https://arxiv.org/pdf/1903.04190.pdf)|97.6|96.6|
89-
| [Meng et al. (2019)](https://arxiv.org/pdf/1901.10125.pdf) | 96.6 | |
90-
| [Ma et al. (2018)](http://aclweb.org/anthology/D18-1529) | 96.7 | 96.6 |
91-
| [Yang et al. (2017)](http://aclweb.org/anthology/P17-1078) | 96.2 | |
92-
| [Zhou et al. (2017)](https://www.aclweb.org/anthology/D17-1079) | 96.2 | |
93-
90+
| Model | CTB6 | CTB7 | CTB9 |
91+
| --- | --- | --- | --- |
92+
| [Tian, Song, Ao, Xia, Quan, Zhang, Wang (2020)](https://www.aclweb.org/anthology/2020.acl-main.735/) | 97.5 | 97.3 | 97.8 |
93+
| [Tian, Song, Xia, Zhang, Wang (2020)](https://www.aclweb.org/anthology/2020.acl-main.734/) | 97.3 | |
94+
| [Yan et al. (2020)](https://transacl.org/ojs/index.php/tacl/article/view/1876) | | 97.1| 97.6 |
95+
| [Huang et al. (2019)](https://arxiv.org/abs/1903.04190)|97.6| | |
96+
| [Ma et al. (2018)](http://aclweb.org/anthology/D18-1529) | 96.7 | 96.6<sup>**</sup> | |
97+
| [Yang et al. (2017)](http://aclweb.org/anthology/P17-1078) | 96.2 | | |
98+
| [Zhou et al. (2017)](https://www.aclweb.org/anthology/D17-1079) | 96.2 | | |
9499

100+
<sup>**</sup> [Ma et al. (2018)](http://aclweb.org/anthology/D18-1529) report different statistics for their CTB7 split (950K/60K/82K), so the results might not be comparable.
95101

96102

97103
### Resources
98104

99-
| Train set | Training Size(Words) |
105+
| Train set | Training Size (Words) |
100106
| --- | ----: |
101-
| CTB6 | 641,368 |
102-
| CTB7 | 950,138 |
107+
| CTB6 | 641K |
108+
| CTB7 | 718K |
109+
| CTB9 | 1,696K |
103110

104111

105112
## <span class="t">Chinese Universal Treebank (UD)</span>.
@@ -114,7 +121,8 @@ F1 = 0.857
114121

115122
| Model | UD |
116123
| --- | --- |
117-
| [Huang et al. (2019)](https://arxiv.org/pdf/1903.04190.pdf)|97.3 |
124+
| [Tian, Song, Ao, Xia, Quan, Zhang, Wang (2020)](https://www.aclweb.org/anthology/2020.acl-main.735/) | 98.3 |
125+
| [Huang et al. (2019)](https://arxiv.org/abs/1903.04190)|97.3 |
118126
| [Ma et al. (2018)](http://aclweb.org/anthology/D18-1529) | 96.9 |
119127

120128
### Resources
@@ -137,7 +145,6 @@ F1 = 0.857
137145

138146
| Model | Weibo |
139147
| --- | --- |
140-
| [Meng et al. (2019)](https://arxiv.org/pdf/1901.10125.pdf) | 96.0 |
141148
| [Yang et al. (2017)](http://aclweb.org/anthology/P17-1078) | 95.5 |
142149

143150

0 commit comments

Comments
 (0)