2021

Token-Level Supervised Contrastive Learning for Punctuation Restoration

Huang, Qiushi, Ko, Tom, Tang, H Lilian et al.

Understand

Punctuation is critical in understanding natural language text.

Currently, most automatic speech recognition (ASR) systems do not generate punctuation, which affects the performance of downstream tasks, such as intent detection and slot filling.
This gives rise to the need for punctuation restoration.
Recent work in punctuation restoration heavily utilizes pre-trained language models without considering data imbalance when predicting punctuation classes.

Built on

D. Beeferman, A. Berger, and J. Lafferty, “Cyberpunc: a lightweight punctuation annotation system for speech,” in ICASSP, 1998 , pp. 689–692 vol.2
1998
Earlier work this paper cites.
H. Christensen, Y. Gotoh, and S. Renals, “Punctuation annotation using statistical prosody models.” in Proc Isca Workshop on Prosody in Speech Recognition and Understanding , 2001
2001
Earlier work this paper cites.
J. Kim and P. Woodland, “A combined punctuation generation and speech recognition system and its performance enhancement using prosody,” Speech Commun. , vol. 41, pp. 563–577, 2003
2003
Earlier work this paper cites.
Y. Liu, E. Shriberg, A. Stolcke, D. Hillard, M. Ostendorf, and M. Harper, “Enriching speech recognition with automatic detection of sentence boundaries and disfluencies,” IEEE Transactions on Audio, Speech and Language Processing , vol. 14, no. 5, p. 1526–1540, Sep. 2006
2006
Earlier work this paper cites.
A. Gravano, M. Jansche, and M. Bacchiani, “Restoring punctuation and capitalization in transcribed speech,” in ICASSP, 2009 , USA, p. 4741–4744
2009
Earlier work this paper cites.
W. Lu and H. T. Ng, “Better punctuation prediction with dynamic conditional random fields,” in EMNLP , 2010, pp. 177–186
2010
Earlier work this paper cites.
N. Ueffing, M. Bisani, and P. Vozila, “Improved models for automatic punctuation prediction for spoken and written text,” in Proc. Interspeech , 2013
2013
Earlier work this paper cites.
F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering.” in CVPR , 2015
2015
Earlier work this paper cites.
X. Che, C. Wang, H. Yang, and C. Meinel, “Punctuation prediction for unsegmented transcript based on word vector,” in LREC , 2016
2016
Earlier work this paper cites.
O. Tilk and T. Alumäe, “Bidirectional recurrent neural network with attention mechanism for punctuation restoration,” in Proc. Interspeech , 2016, pp. 3047–3051
2016
Earlier work this paper cites.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems , 2017
2017
Earlier work this paper cites.

Similar

H. Bredin, “Tristounet: triplet loss for speaker turn embedding,” in ICASSP, 2017 , pp. 5430–5434
2017
Cited alongside, same era.
J. Yi, J. Tao, Z. Wen, and Y. Li, “Distilling knowledge from an ensemble of models for punctuation prediction,” in Proc. Interspeech , 2017, pp. 2779–2783
2017
Cited alongside, same era.
P. Żelasko, P. Szymanski, J. Mizgajski, A. Szymczak, Y. Carmiel, and N. Dehak, “Punctuation prediction model for conversational speech,” in Proc. Interspeech , 2018
2018
Cited alongside, same era.
A. V. D. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748 , 2018
Original
2018
Cited alongside, same era.
J. Yi and J. Tao, “Self-attention based model for punctuation prediction using word and speech embeddings,” in ICASSP, 2019 , pp. 7270–7274
2019
Cited alongside, same era.
G. Szaszák and M. Ákos Tündik, “Leveraging a Character, Word and Prosody Triplet for an ASR Error Robust and Agglutination Friendly Punctuation Approach,” in Proc. Interspeech , 2019, pp. 2988–2992
2019
Cited alongside, same era.
A. Nanchen and P. N. Garner, “Empirical evaluation and combination of punctuation prediction models applied to broadcast news,” in ICASSP, 2019 , pp. 7275–7279
2019
Cited alongside, same era.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of NAACL , 2019, pp. 4171–4186
2019
Cited alongside, same era.
K. Makhija, T.-N. Ho, and E. Chng, “Transfer learning for punctuation prediction,” in Proceedings of APSIPA , 2019, pp. 268–273
2019
Cited alongside, same era.
S. Kim, “Deep recurrent neural networks with layer-wise multi-head attentions for punctuation restoration,” in ICASSP, 2019 , pp. 7280–7284
2019
Cited alongside, same era.
I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in International Conference on Learning Representations , 2019
2019
Cited alongside, same era.

Then

Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “RoBERTa: A robustly optimized BERT pretraining approach,” 2020
2020
Later among the works it cites.
Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “ALBERT: A lite BERT for self-supervised learning of language representations,” in International Conference on Learning Representations , 2020
2020
Later among the works it cites.
B. Lin and L. Wang, “Joint Prediction of Punctuation and Disfluency in Speech Transcripts,” in Proc. Interspeech , 2020, pp. 716–720
2020
Later among the works it cites.
J. Yi, J. Tao, Z. Tian, Y. Bai, and C. Fan, “Focal Loss for Punctuation Prediction,” in Proc. Interspeech , 2020, pp. 721–725
2020
Later among the works it cites.
M. Courtland, A. Faulkner, and G. McElvain, “Efficient automatic punctuation restoration using bidirectional transformers with robust inference,” in Proceedings of the 17th International Conference on Spoken Language Translation , 2020, pp. 272–279
2020
Later among the works it cites.
P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan, “Supervised contrastive learning,” in Advances in Neural Information Processing Systems , vol. 33, 2020, pp. 18 661–18 673
2020
Later among the works it cites.
T. Alam, A. Khan, and F. Alam, “Punctuation restoration using transformer models for high-and low-resource languages,” in W-NUT 2020 , pp. 132–142
2020
Later among the works it cites.
J. Yi, J. Tao, Y. Bai, Z. Tian, and C. Fan, “Adversarial transfer learning for punctuation restoration,” 2020
2020
Later among the works it cites.
Q. Chen, M. Chen, B. Li, and W. Wang, “Controllable time-delay transformer for real-time punctuation prediction and disfluency detection,” in ICASSP, 2020 , pp. 8069–8073
2020
Later among the works it cites.
A. Nagy, B. Bial, and J. Ács, “Automatic punctuation restoration with BERT models,” arXiv preprint arXiv:2101.07343 , 2021
Original
2021
Closest in time.
B. Gunel, J. Du, A. Conneau, and V. Stoyanov, “Supervised contrastive learning for pre-trained language model fine-tuning,” in International Conference on Learning Representations , 2021
2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…