publications | Kirill Milintsevich

2024

PhD Thesis
Estimation of Depression Level from Text: Symptom-Based Approach, External Knowledge, Dataset Validity

Kirill Milintsevich

Normandie Université; Tartu Ülikool, 2024

Abs Bib

Major Depressive Disorder (MDD) is one of the most prevalent mental disorders globally, often resulting in disability and an increased risk of suicide. The recent COVID-19 pandemic has made depression rates go up around the world. Moreover, stigma and limited treatment access hinder proper diagnosis and care for many. Early studies have found that depressed and non-depressed people use different vocabulary. For example, depressed people tend to use more negative or emotional words. More recently, deep learning models have been developed to detect depression from text. However, most researchers have treated depression detection as a simple classification task with only two possible labels: depressed and non-depressed. When considering two individuals with depression, it is important to note that they may exhibit different underlying symptoms. One person may experience insomnia and difficulty concentrating, while another may struggle with changes in appetite and low self-esteem. These people would require different treatments, so having information about the symptoms is essential. In this work, we developed a neural network that predicts depression symptoms from text. We found that predicting symptoms instead of a simple diagnosis was more accurate while giving us more details at the same time. We further improved the neural network by introducing external knowledge from existing sentiment and emotion lexicons. We used a simplistic yet effective approach of directly marking the words from the lexicons in the text. Finally, while working with a social media dataset, we discovered it was poorly annotated. As a result, we reannotated a part of this dataset with the help of a mental health professional, showing the importance of following medical symptom definitions and establishing clear annotation guidelines.
@phdthesis{milintsevich2024estimation, title = {Estimation of Depression Level from Text: Symptom-Based Approach, External Knowledge, Dataset Validity}, author = {Milintsevich, Kirill}, year = {2024}, school = {Normandie Universit{\'e}; Tartu {\"U}likool}, url = {https://hdl.handle.net/10062/106041}, }
LREC-COLING
Analyzing Symptom-based Depression Level Estimation through the Prism of Psychiatric Expertise

Navneet* Agarwal, Kirill* Milintsevich, Lucie Metivier, and 3 more authors

In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), May 2024

Abs Bib

The ever-growing number of people suffering from mental distress has motivated significant research initiatives towards automated depression estimation. Despite the multidisciplinary nature of the task, very few of these approaches include medical professionals in their research process, thus ignoring a vital source of domain knowledge. In this paper, we propose to bring the domain experts back into the loop and incorporate their knowledge within the gold-standard DAIC-WOZ dataset. In particular, we define a novel transformer-based architecture and analyse its performance in light of our expert annotations. Overall findings demonstrate a strong correlation between the psychological tendencies of medical professionals and the behavior of the proposed model, which additionally provides new state-of-the-art results.
@inproceedings{agarwal-etal-2024-analyzing, title = {Analyzing Symptom-based Depression Level Estimation through the Prism of Psychiatric Expertise}, author = {Agarwal, Navneet and Milintsevich, Kirill and Metivier, Lucie and Rotharmel, Maud and Dias, Ga{\"e}l and Dollfus, Sonia}, editor = {Calzolari, Nicoletta and Kan, Min-Yen and Hoste, Veronique and Lenci, Alessandro and Sakti, Sakriani and Xue, Nianwen}, booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)}, month = may, year = {2024}, address = {Torino, Italia}, publisher = {ELRA and ICCL}, url = {https://aclanthology.org/2024.lrec-main.87}, pages = {974--983} }
Evaluating Lexicon Incorporation for Depression Symptom Estimation

Kirill Milintsevich, Gaël Dias, and Kairit Sirts

In Proceedings of the 6th Clinical Natural Language Processing Workshop, Jun 2024

Abs Bib Code

This paper explores the impact of incorporating sentiment, emotion, and domain-specific lexicons into a transformer-based model for depression symptom estimation. Lexicon information is added by marking the words in the input transcripts of patient-therapist conversations as well as in social media posts. Overall results show that the introduction of external knowledge within pre-trained language models can be beneficial for prediction performance, while different lexicons show distinct behaviours depending on the targeted task. Additionally, new state-of-the-art results are obtained for the estimation of depression level over patient-therapist interviews.
@inproceedings{milintsevich-etal-2024-evaluating, title = {Evaluating Lexicon Incorporation for Depression Symptom Estimation}, author = {Milintsevich, Kirill and Dias, Ga{\"e}l and Sirts, Kairit}, editor = {Naumann, Tristan and Ben Abacha, Asma and Bethard, Steven and Roberts, Kirk and Bitterman, Danielle}, booktitle = {Proceedings of the 6th Clinical Natural Language Processing Workshop}, month = jun, year = {2024}, address = {Mexico City, Mexico}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2024.clinicalnlp-1.28}, doi = {10.18653/v1/2024.clinicalnlp-1.28}, pages = {322--328} }
Your Model Is Not Predicting Depression Well And That Is Why: A Case Study of PRIMATE Dataset

Kirill Milintsevich, Kairit Sirts, and Gaël Dias

In Proceedings of the 9th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2024), Mar 2024

Abs Bib Code

This paper addresses the quality of annotations in mental health datasets used for NLP-based depression level estimation from social media texts. While previous research relies on social media-based datasets annotated with binary categories, i.e. depressed or non-depressed, recent datasets such as D2S and PRIMATE aim for nuanced annotations using PHQ-9 symptoms. However, most of these datasets rely on crowd workers without the domain knowledge for annotation. Focusing on the PRIMATE dataset, our study reveals concerns regarding annotation validity, particularly for the lack of interest or pleasure symptom. Through reannotation by a mental health professional, we introduce finer labels and textual spans as evidence, identifying a notable number of false positives. Our refined annotations, to be released under a Data Use Agreement, offer a higher-quality test set for anhedonia detection. This study underscores the necessity of addressing annotation quality issues in mental health datasets, advocating for improved methodologies to enhance NLP model reliability in mental health assessments.
@inproceedings{milintsevich-etal-2024-model, title = {Your Model Is Not Predicting Depression Well And That Is Why: A Case Study of {PRIMATE} Dataset}, author = {Milintsevich, Kirill and Sirts, Kairit and Dias, Ga{\"e}l}, editor = {Yates, Andrew and Desmet, Bart and Prud{'}hommeaux, Emily and Zirikly, Ayah and Bedrick, Steven and MacAvaney, Sean and Bar, Kfir and Ireland, Molly and Ophir, Yaakov}, booktitle = {Proceedings of the 9th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2024)}, month = mar, year = {2024}, address = {St. Julians, Malta}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2024.clpsych-1.13}, pages = {166--171} }

2023

Brain Informatics

Towards automatic text-based estimation of depression through symptom prediction

Kirill Milintsevich, Kairit Sirts, and Gaël Dias

Brain Informatics, Mar 2023

Bib Code

@article{milintsevich2023towards,
  title = {Towards automatic text-based estimation of depression through symptom prediction},
  author = {Milintsevich, Kirill and Sirts, Kairit and Dias, Ga{\"e}l},
  journal = {Brain Informatics},
  volume = {10},
  number = {1},
  pages = {1--14},
  year = {2023},
  publisher = {SpringerOpen}
}

Calvados at MEDIQA-Chat 2023: Improving Clinical Note Generation with Multi-Task Instruction Finetuning

Kirill Milintsevich, and Navneet Agarwal

In Proceedings of the 5th Clinical Natural Language Processing Workshop, Jul 2023

Abs Bib Code

This paper presents our system for the MEDIQA-Chat 2023 shared task on medical conversation summarization. Our approach involves finetuning a LongT5 model on multiple tasks simultaneously, which we demonstrate improves the model’s overall performance while reducing the number of factual errors and hallucinations in the generated summary. Furthermore, we investigated the effect of augmenting the data with in-text annotations from a clinical named entity recognition model, finding that this approach decreased summarization quality. Lastly, we explore using different text generation strategies for medical note generation based on the length of the note. Our findings suggest that the application of our proposed approach can be beneficial for improving the accuracy and effectiveness of medical conversation summarization.
@inproceedings{milintsevich-agarwal-2023-calvados, title = {{C}alvados at {MEDIQA}-Chat 2023: Improving Clinical Note Generation with Multi-Task Instruction Finetuning}, author = {Milintsevich, Kirill and Agarwal, Navneet}, editor = {Naumann, Tristan and Ben Abacha, Asma and Bethard, Steven and Roberts, Kirk and Rumshisky, Anna}, booktitle = {Proceedings of the 5th Clinical Natural Language Processing Workshop}, month = jul, year = {2023}, address = {Toronto, Canada}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2023.clinicalnlp-1.56}, doi = {10.18653/v1/2023.clinicalnlp-1.56}, pages = {529--535} }

2021

EACL
Enhancing Sequence-to-Sequence Neural Lemmatization with External Resources

Kirill Milintsevich, and Kairit Sirts

In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Apr 2021

Abs Bib Code

We propose a novel hybrid approach to lemmatization that enhances the seq2seq neural model with additional lemmas extracted from an external lexicon or a rule-based system. During training, the enhanced lemmatizer learns both to generate lemmas via a sequential decoder and copy the lemma characters from the external candidates supplied during run-time. Our lemmatizer enhanced with candidates extracted from the Apertium morphological analyzer achieves statistically significant improvements compared to baseline models not utilizing additional lemma information, achieves an average accuracy of 97.25% on a set of 23 UD languages, which is 0.55% higher than obtained with the Stanford Stanza model on the same set of languages. We also compare with other methods of integrating external data into lemmatization and show that our enhanced system performs considerably better than a simple lexicon extension method based on the Stanza system, and it achieves complementary improvements w.r.t. the data augmentation method.
@inproceedings{milintsevich-sirts-2021-enhancing, title = {Enhancing Sequence-to-Sequence Neural Lemmatization with External Resources}, author = {Milintsevich, Kirill and Sirts, Kairit}, editor = {Merlo, Paola and Tiedemann, Jorg and Tsarfaty, Reut}, booktitle = {Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume}, month = apr, year = {2021}, address = {Online}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2021.eacl-main.272}, doi = {10.18653/v1/2021.eacl-main.272}, pages = {3112--3122} }

2020

Baltic HLT
Lexicon-Enhanced Neural Lemmatization for Estonian

Kirill Milintsevich, and Kairit Sirts

In Human Language Technologies–The Baltic Perspective, Apr 2020

Abs Bib

We propose a novel approach for Estonian lemmatization that enriches the seq2seq neural lemmatization model with lemma candidates generated by the rule-based VABAMORF morphological analyser. In this way, the neural decoder can benefit from the additional input considering that it has a high likelihood of including the correct lemma. We develop our model by stacking two interconnected layers of attention in the decoder—one attending to the input word and another to the candidates obtained from the morphological analyser. We show that the lexicon-enhanced model achieves statistically significant improvements in lemmatization compared to baseline models not utilizing additional lemma information and achieves a new best result on lemmatization on the Estonian UD test set.
@incollection{milintsevich2020lexicon, title = {Lexicon-Enhanced Neural Lemmatization for {E}stonian}, author = {Milintsevich, Kirill and Sirts, Kairit}, booktitle = {Human Language Technologies--The Baltic Perspective}, pages = {158--165}, year = {2020}, publisher = {IOS Press}, doi = {10.3233/FAIA200618}, url = {https://ebooks.iospress.nl/doi/10.3233/FAIA200618} }
Baltic HLT
Evaluating Multilingual BERT for Estonian

Claudia Kittask, Kirill Milintsevich, and Kairit Sirts

In Human Language Technologies–The Baltic Perspective, Apr 2020

Abs Bib

Recently, large pre-trained language models, such as BERT, have reached state-of-the-art performance in many natural language processing tasks, but for many languages, including Estonian, BERT models are not yet available. However, there exist several multilingual BERT models that can handle multiple languages simultaneously and that have been trained also on Estonian data. In this paper, we evaluate four multilingual models—multilingual BERT, multilingual distilled BERT, XLM and XLM-RoBERTa—on several NLP tasks including POS and morphological tagging, NER and text classification. Our aim is to establish a comparison between these multilingual BERT models and the existing baseline neural models for these tasks. Our results show that multilingual BERT models can generalise well on different Estonian NLP tasks outperforming all baselines models for POS and morphological tagging and text classification, and reaching the comparable level with the best baseline for NER, with XLM-RoBERTa achieving the highest results compared with other multilingual models.
@incollection{kittask2020evaluating, title = {Evaluating Multilingual {BERT} for {E}stonian}, author = {Kittask, Claudia and Milintsevich, Kirill and Sirts, Kairit}, booktitle = {Human Language Technologies--The Baltic Perspective}, pages = {19--26}, year = {2020}, publisher = {IOS Press}, doi = {10.3233/FAIA200597} }

2019

Char-RNN for Word Stress Detection in East Slavic Languages

Ekaterina Chernyak, Maria Ponomareva, and Kirill Milintsevich

In Proceedings of the Sixth Workshop on NLP for Similar Languages, Varieties and Dialects, Jun 2019

Abs Bib

We explore how well a sequence labeling approach, namely, recurrent neural network, is suited for the task of resource-poor and POS tagging free word stress detection in the Russian, Ukranian, Belarusian languages. We present new datasets, annotated with the word stress, for the three languages and compare several RNN models trained on three languages and explore possible applications of the transfer learning for the task. We show that it is possible to train a model in a cross-lingual setting and that using additional languages improves the quality of the results.
@inproceedings{chernyak-etal-2019-char, title = {Char-{RNN} for Word Stress Detection in {E}ast {S}lavic Languages}, author = {Chernyak, Ekaterina and Ponomareva, Maria and Milintsevich, Kirill}, editor = {Zampieri, Marcos and Nakov, Preslav and Malmasi, Shervin and Ljube{\v{s}}i{\'c}, Nikola and Tiedemann, J{\"o}rg and Ali, Ahmed}, booktitle = {Proceedings of the Sixth Workshop on {NLP} for Similar Languages, Varieties and Dialects}, month = jun, year = {2019}, address = {Ann Arbor, Michigan}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/W19-1404}, doi = {10.18653/v1/W19-1404}, pages = {35--41} }

2017

Automated Word Stress Detection in Russian

Maria Ponomareva, Kirill Milintsevich, Ekaterina Chernyak, and 1 more author

In Proceedings of the First Workshop on Subword and Character Level Models in NLP, Sep 2017

Abs Bib Code

In this study we address the problem of automated word stress detection in Russian using character level models and no part-speech-taggers. We use a simple bidirectional RNN with LSTM nodes and achieve accuracy of 90% or higher. We experiment with two training datasets and show that using the data from an annotated corpus is much more efficient than using only a dictionary, since it allows to retain the context of the word and its morphological features.

@inproceedings{ponomareva-etal-2017-automated,
  title = {Automated Word Stress Detection in {R}ussian},
  author = {Ponomareva, Maria and Milintsevich, Kirill and Chernyak, Ekaterina and Starostin, Anatoly},
  editor = {Faruqui, Manaal and Schuetze, Hinrich and Trancoso, Isabel and Yaghoobzadeh, Yadollah},
  booktitle = {Proceedings of the First Workshop on Subword and Character Level Models in {NLP}},
  month = sep,
  year = {2017},
  address = {Copenhagen, Denmark},
  publisher = {Association for Computational Linguistics},
  url = {https://aclanthology.org/W17-4104},
  doi = {10.18653/v1/W17-4104},
  pages = {31--35}
}