2019

Divide and Conquer: A Deep CASA Approach to Talker-independent Monaural Speaker Separation

Liu, Yuzhou, Wang, DeLiang

Understand

We address talker-independent monaural speaker separation from the perspectives of deep learning and computational auditory scene analysis (CASA).

Specifically, we decompose the multi-speaker separation task into the stages of simultaneous grouping and sequential grouping.
Simultaneous grouping is first performed in each time frame by separating the spectra of different speakers with a permutation-invariantly trained neural network.
In the second stage, the frame-level separated spectra are sequentially grouped to different speakers by a clustering network.

Built on

A. Bregman, Auditory scene analysis . Cambridge MA: MIT Press, 1990
1990
Earlier work this paper cites.
D. S. Brungart, “Informational and energetic masking effects in the perception of two simultaneous talkers,” J. Acoust. Soc. Amer. , vol. 109, pp. 1101–1109, 2001
2001
Earlier work this paper cites.
ITU-R, “Perceptual evaluation of speech quality (PESQ) An objective method for end-to-end speech quality assessment of narrowband telephone networks and speech codecs,” Recommendation P.862 , 2001
2001
Earlier work this paper cites.
E. Vincent, R. Gribonval, and C. Févotte, “Performance measurement in blind audio source separation,” IEEE Trans. Audio, Speech, Lang. Process. , vol. 14, pp. 1462–1469, 2006
2006
Earlier work this paper cites.
D. L. Wang and G. Brown, Eds., Computational Auditory Scene Analysis: Principles, Algorithms and Applications . Wiley-IEEE Press, 2006
2006
Earlier work this paper cites.
G. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov, “Improving neural networks by preventing co-adaptation of feature detectors,” arXiv preprint arXiv:1207.0580 , 2012
Original
2012
Earlier work this paper cites.
K. Hu and D. L. Wang, “An unsupervised approach to cochannel speech separation,” IEEE Trans. Audio, Speech, Lang. Process. , vol. 21, pp. 122–131, 2013
2013
Earlier work this paper cites.
J. Du, Y. Tu, Y. Xu, L. R. Dai, and C. H. Lee, “Speech separation of a target speaker based on deep neural networks,” in Proc. ICSP , 2014, pp. 65–68
2014
Earlier work this paper cites.
P. S. Huang, M. Kim, M. Hasegawa-Johnson, and P. Smaragdis, “Deep learning for monaural speech separation,” in Proc. ICASSP , 2014, pp. 1562–1566
2014
Earlier work this paper cites.
Y. Wang, A. Narayanan, and D. L. Wang, “On training targets for supervised speech separation,” IEEE/ACM Trans. Audio, Speech, Lang. Process. , vol. 22, pp. 1849–1858, 2014
2014
Earlier work this paper cites.
H. Erdogan, J. R. Hershey, and S. Watanabe, “Phase-sensitive and recognition-boosted speech separation using deep recurrent neural networks,” in Proc. ICASSP , 2015, pp. 708–712
2015
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in Proc. ICCV , 2015, pp. 1026–1034
2015
Earlier work this paper cites.
D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proc. ICLR , 2015
2015
Earlier work this paper cites.
O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional networks for biomedical image segmentation,” in Proc. MICCAI , 2015, pp. 234–241
2015
Earlier work this paper cites.

Similar

J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” arXiv preprint arXiv:1607.06450 , 2016
Original
2016
Cited alongside, same era.
D.-A. Clevert, T. Unterthiner, and S. Hochreiter, “Fast and accurate deep network learning by exponential linear units (ELUs),” in Proc. ICLR , 2016
2016
Cited alongside, same era.
Y. Gal and Z. Ghahramani, “A theoretically grounded application of dropout in recurrent neural networks,” in Proc. NIPS , 2016, pp. 1019–1027
2016
Cited alongside, same era.
J. R. Hershey, Z. Chen, J. L. Roux, and S. Watanabe, “Deep clustering: Discriminative embeddings for segmentation and separation,” in Proc. ICASSP , 2016, pp. 31–35
2016
Cited alongside, same era.
J. Jensen and C. H. Taal, “An algorithm for predicting the intelligibility of speech masked by modulated noise maskers,” IEEE Trans. Audio, Speech, Lang. Process. , vol. 24, pp. 2009–2022, 2016
2016
Cited alongside, same era.
C. Lea, R. Vidal, A. Reiter, and G. D. Hager, “Temporal convolutional networks: A unified approach to action segmentation,” in Proc. ECCV , 2016, pp. 47–54
2016
Cited alongside, same era.
S. Semeniuta, A. Severyn, and E. Barth, “Recurrent dropout without memory loss,” arXiv preprint arXiv:1603.05118 , 2016
Original
2016
Cited alongside, same era.
D. S. Williamson, Y. Wang, and D. L. Wang, “Complex ratio masking for monaural speech separation,” IEEE Trans. Audio, Speech, Lang. Process. , vol. 24, pp. 483–492, 2016
2016
Cited alongside, same era.
X. L. Zhang and D. L. Wang, “A deep ensemble learning method for monaural speech separation,” IEEE/ACM Trans. Audio, Speech, Lang. Process. , vol. 24, pp. 967–977, 2016
2016
Cited alongside, same era.
F. Chollet, “Xception: Deep learning with depthwise separable convolutions,” in Proc. CVPR , 2017, pp. 1251–1258
2017
Cited alongside, same era.
E. W. Healy, M. Delfarah, J. L. Vasko, B. L. Carter, and D. L. Wang, “An algorithm to increase intelligibility for hearing impaired listeners in the presence of a competing talker,” J. Acoust. Soc. Amer. , vol. 141, pp. 4230–4239, 2017
2017
Cited alongside, same era.
G. Huang, Z. Liu, L. Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in Proc. CVPR , 2017, pp. 4700–4708
2017
Cited alongside, same era.

Then

S. Merity, N. S. Keskar, and R. Socher, “Regularizing and optimizing LSTM language models,” arXiv preprint arXiv:1708.02182 , 2017
Original
2017
Later among the works it cites.
N. Takahashi and Y. Mitsufuji, “Multi-scale multi-band DenseNets for audio source separation,” in Proc. WASPAA , 2017, pp. 21–25
2017
Later among the works it cites.
S. Bai, J. Z. Kolter, and V. Koltun, “An empirical evaluation of generic convolutional and recurrent networks for sequence modeling,” arXiv preprint arXiv:1803.01271 , 2018
Original
2018
Later among the works it cites.
C. Li, L. Zhu, S. Xu, P. Gao, and B. Xu, “CBLDNN-based speaker-independent speech separation via generative adversarial training,” in Proc. ICASSP , 2018, pp. 711–715
2018
Later among the works it cites.
Y. Liu and D. L. Wang, “A CASA approach to deep learning based speaker-independent co-channel speech separation,” in Proc. ICASSP , 2018, pp. 5399–5403
2018
Later among the works it cites.
Y. Luo, Z. Chen, and N. Mesgarani, “Speaker-independent speech separation with deep attractor network,” IEEE Trans. Audio, Speech, Lang. Process. , vol. 26, pp. 787–796, 2018
2018
Later among the works it cites.
Y. Luo and N. Mesgarani, “TasNet: Surpassing ideal time-frequency masking for speech separation,” arXiv preprint arXiv:1809.07454 , 2018
Original
2018
Later among the works it cites.
Z.-Q. Wang, J. L. Roux, and J. R. Hershey, “Alternative objective functions for deep clustering,” in Proc. ICASSP , 2018, pp. 686–690
2018
Later among the works it cites.
Z.-Q. Wang, J. L. Roux, D. L. Wang, and J. R. Hershey, “End-to-end speech separation with unfolded iterative phase reconstruction,” in Proc. Interspeech , 2018, pp. 2708–2712
2018
Later among the works it cites.
Z.-Q. Wang, K. Tan, and D. L. Wang, “Deep learning based phase reconstruction for speaker separation: A trigonometric perspective,” arXiv preprint arXiv:1811.09010 , 2018
Original
2018
Later among the works it cites.
C. Xu, W. Rao, X. Xiao, E. S. Chng, and H. Li, “Single channel speech separation with constrained utterance level permutation invariant training using grid LSTM,” in Proc. ICASSP , 2018, pp. 6–10
2018
Later among the works it cites.
Z.-X. Li, Y. Song, L.-R. Dai, and I. McLoughlin, “Listening and grouping: an online autoregressive approach for monaural speech separation,” IEEE Trans. Audio, Speech, Lang. Process. , vol. 27, pp. 692–703, 2019
2019
Closest in time.
A. Pandey and D. L. Wang, “TCNN: temporal convolutional neural network for real-time speech enhancement in the time domain,” in Proc. ICASSP , 2019
2019
Closest in time.
Z. Shi, H. Lin, L. Liu, R. Liu, and J. Han, “FurcaNeXt: End-to-end monaural speech separation with dynamic gated dilated temporal convolutional networks,” arXiv preprint arXiv:1902.04891 , 2019
Original
2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…