Defense mechanisms in adolescents at high risk of developing psychosis: An empirical investigation

  • Tommaso Boldrini | tommaso.boldrini@unipd.it Department of Developmental and Socialization Psychology, University of Padova, Padova, Italy.
  • Gabriele Lo Buglio Department of Developmental and Socialization Psychology, University of Padova, Padova, Italy.
  • Guido Giovanardi Department of Dynamic and Clinical Psychology, Sapienza University of Rome, Rome; Department of Psychology, University of Campania “Luigi Vanvitelli,” Caserta, Italy.
  • Vittorio Lingiardi Department of Dynamic and Clinical Psychology, Sapienza University of Rome, Rome, Italy.
  • Silvia Salcuni Department of Developmental and Socialization Psychology, University of Padova, Padova, Italy.

Abstract

Research within psychotherapy and psychopathology frameworks has been inspired by the central concept of defense mechanisms, which play a pivotal role in psychoanalysis. Defense specificities have only recently been studied systematically in the context of several clinical diagnoses. The present study aimed to explore the specificities of defense mechanisms in adolescents at ultra-high risk of developing psychosis and test the relationship between defensive functioning and attenuated psychotic symptoms. Twenty-six adolescent inpatients at ultra-high risk of developing psychosis and a matched clinical control group of inpatients not at risk of psychosis were interviewed. Two observer-rater methods, the Defense Mechanism Rating Scales and the Psychotic-Defense Mechanism Rating Scales were applied to interview transcripts to assess the individual use of defenses. Ultra-high-risk patients demonstrated lower defense mechanisms (i.e., total instances of defense mechanisms observed) and lower overall adaptiveness of the defenses presented. When specific differences between groups were observed, psychotic defenses were the sole defenses that could partially discriminate between ultra-high-risk and not at-risk patients. Regarding the relationship between defense mechanisms and subthreshold psychotic symptoms, psychotic defenses were associated with negative and disorganization symptoms, rather than with positive symptoms. The psychological vulnerability of ultra-high-risk patients is discussed and treatment implications for psychotherapy with such challenging patients are addressed.

Dimensions

Altmetric

PlumX Metrics

Downloads

Download data is not yet available.

References

American Psychiatric Association (2013) Diagnostic and statistical manual of mental disorders (5th ed). Arlington, VA: Author. DOI: https://doi.org/10.1176/appi.books.9780890425596

American Psychiatric Association. (1994). Diagnostic and statistical manual of mental disorders (4th ed.). Washington, DC: American Psychiatric Press.

Andreou, C., Bailey, B., Borgwardt S. (2019). Assessment and treatment of individuals at high risk for psychosis. BJPsych Advances. 25, 177–184. DOI: https://doi.org/10.1192/bja.2019.3

Armando, M., Hutsebaut, J., & Debbane, M. (2019). A Mentalization-Informed Staging Approach to Clinical High Risk for Psychosis. Frontiers In Psychiatry, 10. DOI: https://doi.org/10.3389/fpsyt.2019.00385

Babl, A., Holtforth, M.G., Perry, J.C., Schneider, N., Dommann, E., Heer, S., Stähli, A., Aeschbacher, N., Eggel, M., Eggenberg, J., Sonntag, M., Berger, T., Caspar, F. (2019). Comparison and changes of defense mechanisms over the course of psychotherapy in patients with depression or anxiety disorder: Evidence from a randomized controlled trial. Journal of Affective Disorders, 252, 212-220. DOI: https://doi.org/10.1016/j.jad.2019.04.021

Berney, S., de Roten, Y., Beretta, V., Kramer, U., & Despland, J. N. (2014). Identifying psychotic defenses in a clinical interview. Journal of Clinical Psychology: In Session, 70(5), 428–439. DOI: https://doi.org/10.1002/jclp.22087

Bion, W. R. (1959). Attacks on linking. International Journal of Psychoanalysis, 40, 308–315.

Boldrini, T., Nazzaro, M. P., Damiani, R., Genova, F., Gazzillo, F., & Lingiardi, V. (2018). Mentalization as a predictor of psychoanalytic outcome: An empirical study of transcribed psychoanalytic sessions through the lenses of a computerized text analysis measure of reflective functioning. Psychoanalytic Psychology, 35(2), 196–204. DOI: https://doi.org/10.1037/pap0000154

Boldrini, T., Tanzilli, A., Pontillo, M., Chirumbolo, A., Vicari, S., & Lingiardi, V. (2019). Comorbid Personality Disorders in Individuals with an At-Risk Mental State for Psychosis: A Meta-Analytic Review. Frontiers In Psychiatry, 10, 429. DOI: https://doi.org/10.3389/fpsyt.2019.00429

Bora, E., & Pantelis, C. (2013). Theory of mind impairments in first-episode psychosis, individuals at ultra-high risk for psychosis and in first-degree relatives of schizophrenia: Systematic review and meta-analysis. Schizophrenia Research, 144(1–3), 31–36. DOI: https://doi.org/10.1016/j.schres.2012.12.013

Borsboom, D., Cramer, A.O.J., 2013. Network analysis: an integrative approach to the structure of psychopathology. Annual Review of Clinical Psychology. 9, 91–121. DOI: https://doi.org/10.1146/annurev-clinpsy-050212-185608

Carrà, G., Crocamo, C., Angermeyer, M., Brugha, T., Toumi, M., & Bebbington, P. (2019). Positive and negative symptoms in schizophrenia: A longitudinal analysis using latent variable structural equation modelling. Schizophrenia Research, 204, 58–64. DOI: https://doi.org/10.1016/j.schres.2018.08.018

Cichocki, L. (2008). Changes in defence mechanisms of people suffering from psychotic disorders and receiving therapy in the Day Treatment Centre. Archives of Psychiatry and Psychotherapy, 4, 27–32.

Constantinides, P., & Beck, S. M. (2010). Toward developing a scale to empirically measure psychotic defense mechanisms. Journal of the American Psychoanalytic Association, 58(6), 1159–1188. DOI: https://doi.org/10.1177/0003065110396875

Cramer, P. (2009). The development of defense mechanisms from preadolescence to early adulthood: do IQ and social class matters? A longitudinal study. Journal of Research in Personality, 43, 464-471. DOI: https://doi.org/10.1016/j.jrp.2009.01.021

Damasio A. R. (1999). The Feeling of What Happens: Body and Emotion in the Making of Consciousness. New York, NY: Harcourt Brace.

de Gracia Dominguez, M., Viechtbauer, W., Simons, C.J., van Os, J., Krabbendam, L. (2009). Are psychotic psychopathology and neurocognition orthogonal? A systematic review of their associations. Psychological Bullettin, 135, 157. DOI: https://doi.org/10.1037/a0014415

Di Giuseppe, M., Di Silvestre, A., Lo Sterzo, R., Hitchcott, P., Gemignani, A., Conversano, C. (2019a). Qualitative and quantitative analysis of the defense profile in Breast Cancer women: A pilot study. Health Psychology Open, Jan-Jun 6(1):2055102919854667. DOI: https://doi.org/10.1177/2055102919854667

Di Giuseppe, M., Gennaro, A., Lingiardi, V., & Perry, J. C. (2019b). The role of defense mechanisms in emerging personality disorders in clinical adolescents. Psychiatry: Interpersonal and Biological Processes, 82(2), 128–142. DOI: https://doi.org/10.1080/00332747.2019.1579595

Di Giuseppe, M., Perry, J.C., Petraglia, J., Janzen, J., Lingiardi, V. (2014). Development of a Q‐sort version of the Defense Mechanism Rating Scales (DMRS‐Q) for clinical use. Journal of Clinical Psychology, In Session 70: 452–465.

Euler, S., Stalujanis, E., Allenbach, G., Kolly S., de Roten, Y., Despland, N.J., Kramer, U. (2018). Dialectical behavior therapy skills training affects defense mechanisms in borderline personality disorder: An integrative approach of mechanisms in psychotherapy. Psychotherapy Research, 29(8), 1074-1085. DOI: https://doi.org/10.1080/10503307.2018.1497214

Foto-Özdemir, D., Akdemir, D., Çuhadaroglu-Çetin, F. (2016). Gender differences in defense mechanisms, ways of coping with stress and sense of identity in adolescent suicide attempts. Turkish Journal of Pediatry, 58, 271-281. DOI: https://doi.org/10.24953/turkjped.2016.03.007

Galderisi, S., Mucci, A., Buchanan, R. W., & Arango, C. (2018). Negative symptoms of schizophrenia: New developments and unanswered research questions. The Lancet Psychiatry, 5(8), 664–677. DOI: https://doi.org/10.1016/S2215-0366(18)30050-6

Gallagher S, Zahavi D. (2012). The Phenomenological Mind. 2nd ed. Abingdon, UK: Routledge.

Gallagher, S. (Ed.). (2011). The oxford handbook of the self. Oxford: Oxford University Press. DOI: https://doi.org/10.1093/oxfordhb/9780199548019.001.0001

George, C., Kaplan, N., & Main, M. (1984). The Berkeley Adult Attachment Interview. Unpublished manuscript. Department of Psychology, University of California, Berkeley.

George, C., Kaplan, N., & Main, M. (1996). Adult Attachment Interview (3rd ed.). Unpublished manuscript, Department of Psychology, University of California, Berkley.

Gerritsen, C., Bagby, R. M., Sanches, M., Kiang, M., Maheandiran, M., Prce, I., & Mizrahi, R. (2019). Stress precedes negative symptom exacerbations in clinical high risk and early psychosis: A time-lagged experience sampling study. Schizophrenia Research, 210, 52–58. DOI: https://doi.org/10.1016/j.schres.2019.06.015

Gur, R.E., March, M., Calkins, M.E., Weittenhiller, L., Wolf, D.H., Turetsky, B.I., Gur, R.C., 2015. Negative symptoms in youths with psychosis spectrum features: com- plementary scales in relation to neurocognitive performance and function. Schizophrenia Research, 166, 322–327. DOI: https://doi.org/10.1016/j.schres.2015.05.037

Hauser, S. T. (1986). Conceptual and empirical dilemmas in the assessment of defences. In G. E. Vaillant (Ed.), Empirical studies of ego mechanisms of defense (pp. 89–100). Washington, DC: American Psychiatric Press.

Hilsenroth, M. J., Katz, M., & Tanzilli, A. (2018). Psychotherapy research and the psychodynamic diagnostic manual (PDM-2). Psychoanalytic Psychology, 35(3), 320-327. DOI: https://doi.org/10.1037/pap0000207

Hörz-Sagstetter, S., Mertens, W., Isphording, S., Buchheim, A., & Taubner, S. (2015). Changes in reflective functioning during psychoanalytic psychotherapies. Journal of the American Psychoanalytic Association, 63(3), 481–509. DOI: https://doi.org/10.1177/0003065115591977

Januzzi, G., Verga, M., Piasentin, V., Azzone, P., Vignano, D., & Freni, S. (1997). The psychodynamic impact of psychopharmacological treatment: Change in defense organization. New Trends in Experimental and Clinical Psychiatry, 13(4), 267–270.

Kernberg O.F. (2016). What is personality? Journal of Personality Disorders, 30, 145–156. DOI: https://doi.org/10.1521/pedi.2106.30.2.145

Kernberg, O. F. (1975). Borderline conditions and pathological narcissism. New York, NY: Jason Aronson.

Kernberg, O. F. (1984). Severe personality disorders: Psychotherapeutic strategies. New Haven, CT: Yale University Press.

Klein, M. (1946). Some notes on schizoid mechanisms. International Journal of Psychoanalysis, 27, 99–110.

Kramer, U., de Roten, Y., Perry, J. C., & Despland, J. N. (2009). Specificities of defense mechanisms in bipolar affective disorder relations with symptoms and therapeutic alliance. Journal of Nervous and Mental Disease, 197, 675–681. DOI: https://doi.org/10.1097/NMD.0b013e3181b3b224

Kramer, U., Despland, J. N., Michel, L., Drapeau, M., & de Roten, Y. (2010). Change in defense mechanisms and coping over the course of short-term dynamic psychotherapy for adjustment disorder. Journal of Clinical Psychology, 66, 1232–1241. DOI: https://doi.org/10.1002/jclp.20719

Labouvie-Vief, G., Hakim-Larson, J., & Hobart, C. J. (1987). Age, ego level and the life-span development of coping and defense processes. Psychology and Aging, 2, 286–293. DOI: https://doi.org/10.1037/0882-7974.2.3.286

Leanza, L., Egloff, L., Studerus, E., Andreou, C., Heitz, U., Ittig, S., … Riecher-Rössler, A. (2018). The relationship between negative symptoms and cognitive functioning in patients at clinical high risk for psychosis. Psychiatry Research, 268, 21–27. DOI: https://doi.org/10.1016/j.psychres.2018.06.047

Levy, K. N., Meehan, K. B., Kelly, K. M., Reynoso, J. S., Weber, M., Clarkin, J. F., & Kernberg, O. F. (2006). Change in attachment patterns and reflective function in a randomized control trial of transference-focused psychotherapy for borderline personality disorder. Journal of Consulting and Clinical Psychology, 74(6), 1027–1040. DOI: https://doi.org/10.1037/0022-006X.74.6.1027

Lieberman, J. A., Small, S. A., & Girgis, R. R. (2019). Early Detection and Preventive Intervention in Schizophrenia: From Fantasy to Reality. American Journal of Psychiatry, 176(10), 794–810. DOI: https://doi.org/10.1176/appi.ajp.2019.19080865

Lingiardi, V., & Boldrini, T. (2019). The diagnostic dilemma of psychosis: Reviewing the historical case of pseudoneurotic schizophrenia. Journal of Nervous and Mental Disease, 207(7), 577–584. DOI: https://doi.org/10.1097/NMD.0000000000001008

Lingiardi, V., & McWilliams, N. (2017). Psychodynamic diagnostic manual – 2nd ed. (PDM-2). New York, NY: Guilford Press.

Maffei, C., Fossati, A., Lingiardi, V., Madeddu, F., Borellini, C., & Petrachi, M. (1995). Personality maladjustment, defenses, and psychopathological symptoms in nonclinical subjects. Journal of Personality Disorders,9(4), 330-345. DOI: https://doi.org/10.1521/pedi.1995.9.4.330

Mahler, M. S., & Furer, M. (1968). On human symbiosis and the vicissitudes of individuation: I. Infantile psychosis. Oxford: International Universities Press.

McGlashan, T. H., Walsh, B. C., Woods, S. W., Addington, J., Cadenhead, K., Cannon, T., ... Walker, E. (2013). Structured interview for psychosis-risk syndromes. New Haven, CT: PRIME Research Clinic, Yale School of Medicine.

McWilliams, N. (2011). Psychoanalitic diagnosis. Understanding personality structure in the clinical process (2nd ed.). New York, NY: Guildford Press.

Meyer, E.C., Carrion, R.E., Cornblatt, B.A., Addington, J., Cadenhead, K.S., Cannon, T.D., McGlashan, T.H., Perkins, D.O., Tsuang, M.T., Walker, E.F., Woods, S.W., Heinssen, R., Seidman, L.J., group, N. (2014). The relationship of neurocognition and negative symptoms to social and role functioning over time in individuals at clinical high risk in the first phase of the North American Prodrome Longitudinal Study. Schizophrenia Bullettin, 40, 1452–1461. DOI: https://doi.org/10.1093/schbul/sbt235

Miller, T. J., McGlashan, T. H., Rosen, J. L., Cadenhead, K., Cannon, T., Ventura, J., McFarlane, W., Perkins, D. O., Pearlson, G. D., & Woods, S. W. (2003). Prodromal assessment with the structured interview for prodromal syndromes and the scale of prodromal symptoms: Predictive validity, interrater reliability, and training to reliability. Schizophrenia Bulletin, 29, 703–715. DOI: https://doi.org/10.1093/oxfordjournals.schbul.a007040

Miller, T. J., McGlashan, T. H., Woods, S. W., Stein, K., Driesen, N., Corcoran, C. M., Hoffman, R., & Davidson, L. (1999). Symptoms assessment in schizophrenic prodromal states. Psychiatric Quarterly, 70(4), 273–287. DOI: https://doi.org/10.1023/A:1022034115078

Millman, Z. B., Gold, J. M., Mittal, V. A., & Schiffman, J. (2019). The Critical Need for Help-Seeking Controls in Clinical High-Risk Research. Clinical Psychological Science, 7(6), 1171–1189. DOI: https://doi.org/10.1177/2167702619855660

Nazzaro, M. P., Boldrini, T., Tanzilli, A., Muzi, L., Giovanardi, G., & Lingiardi, V. (2017). Does reflective functioning mediate the relationship between attachment and personality? Psychiatry Research, 256, 169-175. DOI: https://doi.org/10.1016/j.psychres.2017.06.045

Palmier-Claus, J. E., Dunn, G., & Lewis, S. W. (2012). Emotional and symptomatic reactivity to stress in individuals at ultra-high risk of developing psychosis. Psychological Medicine, 42(5), 1003–1012. DOI: https://doi.org/10.1017/S0033291711001929

Pao, P. N. (1979). Schizophrenic disorders. New York, NY: International Universities Press.

Parnas, J. (2012). The core Gestalt of schizophrenia. World Psychiatry (WPA), 11(2), 67–69. DOI: https://doi.org/10.1016/j.wpsyc.2012.05.002

Perry, J. C. (1990). Defense Mechanism Rating Scales (DMRS) (5th ed.). Cambridge: Author.

Perry, J. C., & Bond, M. (2012). Change in defense mechanisms during long-term dynamic psychotherapy and five-year outcome. American Journal of Psychiatry, 169, 916–925. DOI: https://doi.org/10.1176/appi.ajp.2012.11091403

Perry, J. C., & Cooper, S. H. (1987). Empirical studies of psychological defenses. In R. Michels & J. J. Cavenar (Eds.), Psychiatry (vol.1). Philadelphia, PA: Lippincot.

Perry, J. C., & Cooper, S. H. (1989). An empirical study of defense mechanisms: I. Clinical interview and life vignette ratings. Archives of General Psychiatry, 46, 444–452. DOI: https://doi.org/10.1001/archpsyc.1989.01810050058010

Perry, J. C., & Henry, M. (2004). Studying defense mechanisms in psychotherapy using the Defense Mechanism Rating Scales. In U. Hentschel, G. Smith, J. G. Draguns, & W. Ehlers (Eds.), Defense mechanisms: Theoretical, research and clinical perspectives (pp. 165–192). Amsterdam: Elsevier.

Perry, J. C., & Hoglend, P. (1998). Convergent and discriminant validity of overall defensive functioning. Journal of Nervous and Mental Disease, 186, 529–535. DOI: https://doi.org/10.1097/00005053-199809000-00003

Perry, J.C. (1993). A guided clinical interview to assess Axis II disorders. Paper presented at the third Congress of the International Society for the Study of Personality Disorders. Cambridge.

Perry, J.C., Knoll, M., Tran, V. (2019). Motives, defenses, and conflicts in the dynamic formulation for psychodynamic psychotherapy using the Idiographic Conflict Formulation method. In U Kramer (Ed.), Case Formulation for Personality Disorders. New York: Elsevier, 203-224. DOI: https://doi.org/10.1016/B978-0-12-813521-1.00011-4

Perry, J.C., Presniak, M.D., Olson, T.R. (2013). Defense Mechanisms in Schizotypal, Borderline, Antisocial, and Narcissistic Personality Disorders, Psychiatry, 76, 32-52. DOI: https://doi.org/10.1521/psyc.2013.76.1.32

Presniak, M.D., Olson, T., & MacGregor, M.W. (2010). The role of defense mechanisms in Borderline and Antisocial personalities. Journal of Personality Assessment, 92, 137-145. DOI: https://doi.org/10.1080/00223890903510373

Prout, T.A., Malone, A., Rice, T.R., & Hoffman, L. (2019). Resilience, defense mechanisms, and implicit emotion regulation in psychodynamic child psychotherapy. Journal of Contemporary Psychotherapy, 49, 235-244. DOI: https://doi.org/10.1007/s10879-019-09423-w

Sass, L., Borda, J. P., Madeira, L., Pienkos, E., & Nelson, B. (2018). Varieties of self disorder: A bio-pheno-social model of schizophrenia. Schizophrenia Bulletin, 44(4), 720–727. DOI: https://doi.org/10.1093/schbul/sby001

Selzer, M. A., Sullivan, B. T., Carsky, M., & Terkelsen, K. G. (1989). Working with the person with schizophrenia: The treatment alliance. New York, NY: New York University Press.

Semrad, E. (1967). The organization of ego defenses and object loss. In D. M. Moriarity (Ed.), The loss of loved ones (pp 126–134). Springfield, IL: Charles C. Thomas.

Shaffer, D., Gould, M. S., Brasic, J., Ambrosini, P., Fisher, P., Bird, H., & Aluwahlia, S. (1983). A Children's Global Assessment Scale (CGAS). Archives of General Psychiatry, 40(11), 1228–1231. DOI: https://doi.org/10.1001/archpsyc.1983.01790100074010

Spitzer, R. I., Williams, J. B. W., Gibbon, M., & First, M. B. (1990). Structured clinical interview for DMS-III personality disorders (SCID-II). Washington, DC: American University Press.

Vaillant, G. E. (1971), Theoretical hierarchy of adaptive ego mechanisms: A 30-year follow-up of 30 men selected for psychological health. Archives of General Psychiatry, 24, 107–118. DOI: https://doi.org/10.1001/archpsyc.1971.01750080011003

Vaillant, G. E. (1992). Ego mechanisms of defense: A guide for clinicians and researchers. Washington: American Psychiatric Press.

Vaillant, G. E. (2011). Involuntary coping mechanisms: A psychodynamic perspective. Dialogues in Clinical Neuroscience, 13, 366–370. DOI: https://doi.org/10.31887/DCNS.2011.13.2/gvaillant

Vaillant, G. E., & Drake, R. E. (1985). Maturity of ego defenses in relation to DSM-III Axis II personality disorder. Archives of General Psychiatry, 42, 597– 601. DOI: https://doi.org/10.1001/archpsyc.1985.01790290079009

Van der Steen, Y., Gimpel-Drees, J., Lataster, T., Viechtbauer, W., Simons, C.J.P., Lardinois, M., ... Myin-Germeys, I. (2017). Clinical high risk for psychosis: the association between momentary stress, affective and psychotic symptoms. Acta Psychiatrica Scandinavica, 136 (1), 63–73. DOI: https://doi.org/10.1111/acps.12714

Wallerstein, R. S. (1967). Development and metapsychology of the defensive organization of the ego. Panel report. Journal of th American Psychoanalytic Association, 15, 130–149. DOI: https://doi.org/10.1177/000306516701500105

Whitty, M. T. (2003). Coping and defending: Age differences in maturity of defence mechanisms and coping strategies. Aging and Mental Health, 7, 123–132. DOI: https://doi.org/10.1080/1360786031000072277

Winnicott D. W. (1965). Ego distortion in terms of true and false self. In D. W. Winnicott (Ed.), Maturational processes and the facilitating environment (pp. 140–152). New York, NY: International Universities Press.

Winston, B., Winston, A., Samstag, L. W., & Mu- ran, J. C. (1994). Patient defense/therapist inter- ventions. Psychotherapy, 31, 478-491. DOI: https://doi.org/10.1037/0033-3204.31.3.478

Yung, A. R., & McGorry, P. D. (1996) The prodromal phase of first-episode psychosis: Past and current conceptualizations. Schizophrenia Bulletin, 22(2), 353–370. DOI: https://doi.org/10.1093/schbul/22.2.353

Yung, A. R., McGorry, P. D., McFarlane, C. A., Jackson, H. J., Patton, G. C., & Rakkar, A. (1996). Monitoring and care of young people at incipient risk of psychosis. Schizophrenia Bulletin, 22(2), 283–303. DOI: https://doi.org/10.1093/schbul/22.2.283

Zahavi D. (2005). Subjectivity and Selfhood: Investigating the First-Person Perspective. Cambridge, MA: MIT Press. DOI: https://doi.org/10.7551/mitpress/6541.001.0001

Ziegler, D. (2016). Defense mechanisms in Rational Emotive Cognitive Behavior Therapy Personality Theory. Journal of Rational-Emotive & Cognitive-Behavior Therapy, 34 135-148. DOI: https://doi.org/10.1007/s10942-016-0234-2

Published
2020-05-20
Info
Issue
Section
Special Section: Clinical high risk for mental illness
Keywords:
Defense mechanisms, ultra-high risk for psychosis, at-risk mental state for psychosis, attenuated psychotic syndrome, adolescence
Statistics
  • Abstract views: 864

  • PDF: 337
  • HTML: 0
How to Cite
Boldrini, T., Lo Buglio, G., Giovanardi, G., Lingiardi, V., & Salcuni, S. (2020). Defense mechanisms in adolescents at high risk of developing psychosis: An empirical investigation. Research in Psychotherapy: Psychopathology, Process and Outcome, 23(1). https://doi.org/10.4081/ripppo.2020.456