Investigating the Impact of the COVID-19 Pandemic on Computing Students' Sense of Belonging

Sense of belonging, or belongingness, describes how accepted one feels in their academic community and is an important factor in creating inclusive learning environments. Belongingness is influenced by many factors including: students’ backgrounds and experiences; other people; environments (physical and virtual); academic discipline; external factors such as local, regional, and global issues; and time. 2020 has been dominated by several major events including the COVID-19 pandemic which dramatically impacted education. The Black Lives Matter movement has further raised global awareness of equality, diversity and inclusion not just in society, but in educational contexts. Climate change concerns, and politically-charged news are also increasingly affecting our students. We have been monitoring our undergraduate computing students’ sense of belonging for over three years, providing us with a unique opportunity to gauge recent changes during the pandemic. Our results surprised us. We found statistically significant reductions in the belongingness of students identifying as men as well as those not identifying as being part of a minority. However, investigating intersectionality of self-identified gender and minority status revealed more complicated and nuanced trends, illustrating important shifts in the belongingness of our students that we are only beginning to understand.

S ense of belonging, or belongingness, describes how accepted one feels in their academic community and is an important factor in creating inclusive learning environments. Belongingness is influenced by many factors including: students' backgrounds and experiences; other people; environments (physical and virtual); academic discipline; external factors such as local, regional, and global issues; and time. 2020 has been dominated by several major events including the COVID-19 pandemic which dramatically impacted education. The Black Lives Matter movement has further raised global awareness of equality, diversity and inclusion not just in society, but in educational contexts. Climate change concerns, and politically-charged news are also increasingly affecting our students.
We have been monitoring our undergraduate computing students' sense of belonging for over three years, providing us with a unique opportunity to gauge recent changes during the pandemic. Our results surprised us. We found statistically significant reductions in the belongingness of students identifying as men as well as those not identifying as being part of a minority. However, investigating intersectionality of self-identified gender and minority status revealed more complicated and nuanced trends, illustrating important shifts in the belongingness of our students that we are only beginning to understand. education, and is likely influenced by student characteristics (or what students 'bring with them') as well as environmental factors [7]. It is likely that there are also many factors affecting sense of belonging that cannot be controlled for [14]. Sense of belonging has been shown to be influenced not only by traditional educational environmental factors, but by activities such as video games [31], formal research activities [36], and specific technologies and tools such as web interfaces [21].

SENSE OF BELONGING IN OTHER DISCIPLINES
Belongingness is well studied outside computing, in STEM and beyond. Here we only refer to work that directly affects ours. We guide the reader to our recent work [22,23] as well as the following for more on: pSTEM [15], physics [16], and arts & humanities [5].
Good et al. developed and validated a 'Math Sense of Belonging Scale' in 2012 [11]. Using this scale, the authors found that women had a lower sense of belonging which negatively affected their intent to pursue mathematics in the future as well as their academic performance. They found two specific perceptions that damaged the sense of belonging of women, but not of men. The first was the belief that mathematical ability is an inherently fixed trait -akin to the "geek gene" in computing [20,25]. The second is the stereotype that women have less mathematical ability than men -also reminiscent of a computing stereotype. They also found that the perception that mathematical ability could be acquired could protect women from negative stereotypes and allow them to maintain a higher sense of belonging along with the intention to pursue mathematics in the future. This agrees with other research that the difference between men and women in science and mathematics is confidence, not ability [18]. In this work we use an adapted 'Math Sense of Belonging Scale' [11] as described in Section 3.

SENSE OF BELONGING IN COMPUTING
Sax et al. [28] provided impetus for computing educators to foster sense of belonging. Investigating the change in sense of belonging during the introductory course they showed that sense of belonging is a product of incoming student characteristics as well as college environments and experiences. Biggers et al. [3] also provided important evidence that sense of belonging is important in computing, showing that the 'most significantly important factor' causing women to leave the major was feeling like they don't belong. This was the second most important factor for men.
Nguyen & Lewis [24] gathered data from over 1,000 students in dozens of computing faculties along with a collection of departmental policies to show that competitive enrollment in computing degree programs negatively affected student sense of belonging. This study also replicated prior results that women have a significantly lower sense of belonging than men, and that Black and Latinx students have lower sense of belonging than White students.
Other authors have shown that the learning experiences

INTRODUCTION
'Sense of belonging' , or 'belongingness' has been described as "one's personal belief that one is an accepted member of an academic community whose presence and contributions are valued" [11, p711]. Belongingness is important for creating equitable and inclusive learning environments and is likely influenced by many things including students' experiences and backgrounds [12], people (colleagues [34] and professors [10]), place/environment (physical [35] and virtual [17]), context (such as the discipline [11] and part-time/full-time status [13]), external factors (such as local, regional, and global landscapes) [6] and time [1]. In early 2020, the COVID-19 pandemic caused a nearly universal shift of physically delivered education, such as that taking place on traditional college campuses, to online delivery. We are motivated by a recent university-wide survey of our students where half of students ranked "COVID-related stress and anxiety" as "extremely challenging" or "very challenging".
In a study conducted during the early months of COVID-19, Markel & Guo found what many suspected -that many online learning environments fail to adequately replicate critical facets of physical delivery, and that although remote technologies can help with inclusivity in some ways, they can also raise additional barriers [19]. Such shifts are likely to have effects on student sense of belonging and are bound to affect some students more than others, possibly depending on gender, ethnicity, socio-economic background, etc. For example, Stebleton et al. found that the sense of belonging of students who are immigrants is positively affected by "strong peer relationships" and "faculty interactions with students" [34]. Although this was a pre-COVID-19 study, few would argue that the recent shift to online learning, and a future of socially-distanced physical learning, would not affect peer and student-faculty interactions and relationships and therefore sense of belonging. We discuss this in more detail in Section 2.
When the pandemic struck we already had a research study focusing on sense of belonging underway for over three years, providing an ideal opportunity to gauge the impact of the pandemic. In this study, our guiding research question is: What is the impact of the COVID-19 pandemic and the ensuing shift to virtual delivery on the sense of belonging of our undergraduate computing students?

RELATED WORK
In Section 1 we discussed some factors that influence belongingness. Belongingness has also been found to be associated with motivation [2], achievement [39], persistence [26] and student retention [37]. Known predictors of belongingness include race and gender identity [11,14,28], two facets that we explore in this paper.
Sense of belonging is intrinsically tied to groups and communities with defined identities such as academic disciplines. It has been explored within several disciplines, is known to play a role in the transition from second-to third-level factors that intersect. We designed our survey intentionally to allow students to define their own intersectionality as described in the following section.

METHODS & PARTICIPANTS
For over three years we have been exploring the sense of belonging of our undergraduate Computer Science students at a large, European, research-intensive university. The language of instruction is English and many students, but not all, are native speakers. Those that are not fluent have a proficient command of English.
The survey was conducted with ethical approval from our Human Research Ethics Committee, number (LS-17-30-Mooney and LS-19-12-Mooney). Responses were gathered via a Google form advertised on email and student social media groups (e.g. Year 1 Facebook group, etc.). All undergraduate students in our school were invited to take part. Our school has one undergraduate degree (BSc in Computer Science) with a total of approximately 450 students fairly evenly distributed across four years. The gender distribution is approximately 20% women.
The survey questions were adapted from the "Math Sense of Belonging Scale" [11] containing 18 positively framed questions (e.g. I feel accepted/I feel respected/etc.) and 12 negatively-framed questions (e.g. I feel excluded/I feel disregarded/etc.). All items were preceded by "When I am in a computer science setting... ". For each item, participants rated their agreement on an 8-point Likert scale (1 strongly disagree ➝ 8 strongly agree). Belongingness was measured as the sum of positively framed question scores minus the sum of negatively-framed question scores.
In addition to the questions adapted from the Math Sense of Belonging Scale we asked the following questions. L denotes Likert scale questions (1 lowest ➝ 5 highest). For example, for the question "In your opinion, how much does success in Computer Science depend on inherent ability?", the choices were: 1 (not at all), 2, 3, 4, 5 (success is based purely on talent). 1 Which of the following are you a member of, or have you taken part in any of the following? (Check all that apply) "Participation": This was followed by a list of student networking, mentoring and outreach activities. 5. In your opinion, how much does success in Computer Science depend on inherent ability? L 6. How much do you interact socially with other students in Computer Science? L 7. How much do you enjoy problem solving? L 8. How confident are you in your mathematics abilities? L 9. How much experience with computer science did you have before you came to university? (check all that apply). This was followed by a list of various experiences such as of computing students can impact sense of belonging. Stout, Tamer & Alvarado [36] found that first year formal research experiences were associated with a strong sense of mentor support the following year. This mentor support then predicted a positive sense of belonging for underrepresented students, but interestingly not for students in majority groups. Promisingly, the gap in sense of belonging between underrepresented and majority students vanished for students with good mentor support.
Even the physical places where we do computing, and the tools we use and build, have been shown to affect sense of belonging. Fincher, Dizallas & Knox connected physical spaces in the computing faculty including formal and informal/ social, with sense of community, disciplinary practice, and student affinity [8]. Metaxa et al. showed that the design of web interfaces for an introductory programming course could affect the sense of belonging of men and women differently [21]. This is important as our spaces, places and tools are all things that computing educators can control.
Although the pandemic is relatively new, it has been reported that about one-quarter of college students have experienced anxiety due to it [4]. The shift to online delivery has had palpable effects. For instance, Markel & Guo found that many online learning environments cannot foster as much ambient awareness or spontaneous interaction as tools used in physical environments, and that remote technologies and learning can make experiential learning more inclusive, but can also raise additional barriers [19].
Fortunately, it has been claimed that "many strategies to improve students' well-being during stressful times like the COVID-19 pandemic can be supported in an online remote environment" [30, p681]. These authors advise that educators should implement strategies to build students' resilience and well-being, acknowledging that any single approach will not meet the needs of all students. Further they remind educators that not all students are responding to the pandemic similarly as each person has a unique set of resilience factors, and that these only become particularly apparent during traumatic events. These factors include personality traits, attributional style, social support systems, and coping self-efficacy [30]. The authors also find that some students (in pharmacy) have performed better during the transition to virtual delivery, likely due to their environments at home being conducive for success, stating that "some students may prefer the flexibility of asynchronous online learning, which eliminates the need for alarms and leads to better sleep and overall well-being" [30, p679].

INTERSECTIONALITY
Schlesinger, Edwards & Grinter [29] emphasized the need for research to recognize the intersections of diverse aspects of identities. These intersections can involve gender, race and socio-economic factors, as well as nationality, sexuality and others [41]. Intersectionality has been used as a lens to better understand underrepresentation [38]. In this study we observe effects on sense of belonging that differ depending on the such as male and female in studies where gender is the focus is not appropriate. Terminology used to describe racial and ethnic groups can also be insensitive if not used appropriately [40]. Additionally, having a flexible and understanding approach is important. For instance, in surveys students should be asked what groups, genders, etc. they identify with or consider themselves to be part of [42]. Binary gender categories, and making assumptions about participants should be avoided [33]. Many publications such as those cited above, and groups such as the American Association of Public Opinion Researchers [9] provide excellent advice in this regard, which we have tried to adhere to. A guiding principle is that it takes humility in attempting to "get it right" [33, p62]. It is also important to understand local context, and when reading others' results not to make assumptions about local context. Different locales have different demographics, and different realities. For instance, certain terms (or how certain terms are used locally) only reflect certain points of view. Local contexts will likely prevent a completely universal language, but that does not mean that researchers should not strive to do their best in their context, while considering other contexts also.
In this work we asked students questions such as: What gender do you identify with? Woman/Man/Non-binary/Prefer not to disclose/Prefer to self-describe, plus open answer field; Do you consider yourself part of a minority in Computer Science? If so, what minority group(s) are you a part of? By providing for flexibility of expression, we have striven to use inclusive, respectful, non-racist, and non-biased language.

STATISTICAL ANALYSIS
To test for statistical significance, we compared the means between groups and utilized Welch's two-sample t-test. Groups were considered to be significantly different if p-value ( ) < 0.05. For presentation we used the guidelines in [27] and [32].
having a family member involved in computer science, Coder Dojo-style activities, taking courses, and self-taught programming.
Unfortunately, we had to exclude a small number of participants who did not identify as either men or women from these published results to preserve anonymity. However, their responses are valuable and have been taken into careful consideration by our school as the primary aim of this project is to improve the experience of our students. We removed an additional three students who did not respond to more than four questions. Eighteen students had one missing answer, three students had two missing answers and one student has four missing answers. It is likely that these questions were simply overlooked. For these we replaced the missing values with the mean Likert score for the question.
We had 130 students complete the survey between Apr. 2017 and Dec. 2019. We refer to these responses as the "pre-COVID" set. As campus had closed in March 2020, the 68 students who completed the survey in the summer of 2020 were studying remotely (from home) and form our "post-COVID" set. We use the terms pre-/post-COVID-19 to refer to pre-and post-onset of the pandemic, as we are not "post-pandemic" at the time of this study. Our responses were from the following program years: year 1 (42); year 2 (65); year 3 (41); and year 4 (50). Of the 62 respondents identifying as women, 45 self-identified as belonging to a minority and 17 did not. Of the 136 identifying as men, 33 self-identified as belonging to a minority and 103 did not.

INTERSECTIONALITY
Not all women identified as being part of a minority and earlier work found that women not identifying as a minority had a similar belongingness profile to men that did not identify as a minority [23]. Those that did identify as being part of a minority did not always do so because they identify as women. For example, two women identified as belonging to a minority because of race, ethnicity and/or nationality only. Figure 1 shows a Venn diagram depicting the intersection of students identifying as women with those identifying as being part of a minority because of gender, race, ethnicity, nationality and/or LGBTQIA+. Of the 38 women who did identify as a minority and listed being a woman as the reason, 16 also gave additional reasons. In addition to race, ethnicity, nationality and/or LGBTQIA+ other reasons that women gave for identifying as a minority in computer science were being a mature student, having a disability, being religious and lack of prior coding experience.

IMPORTANT NOTES ON TERMINOLOGY & APPROACH
We are conscious that a careful approach and appropriate terminology are important, not just in work such as this. Certain terminology or approaches (or combinations thereof ) can lead to individuals (or groups) feeling boxed-in, left-out, or with negative feelings [40]. For instance, using biological sex terms   Figure 2A shows a histogram of the distributions of belongingness scores broken down by the time the survey was completed i.e. pre-/post-COVID-19 (onset). Figure 2B Figure 5 shows a visualization of the mean belongingness scores of four groups: women and men who did and did not self-identify as belonging to a minority. The last pre-COVID-19 data collection was December 2019, just prior to the onset of the COVID-19 pandemic, and months prior to the shift to remote learning. Five data collection periods of several months' duration are represented, therefore these paths are approximate. It can be seen that students self-identifying as men or women, but not self-identifying as being part of a minority suffered sizable drops in belongingness scores post COVID-19. These are also reflected in Figure 4A and B. However, of these, only the drop for non-minority men is statistically significant. The sizable increase in belongingness post COVID-19 for minority women is not statistically significant, as shown in Figure 4A.

RESULTS
The variation between 2017 and 2019 is discussed in our prior work [23]. It is worth noting that we observed that women who do not self-identify as being part of a minority had a very similar belongingness to non-minority men. This trend continues post COVID-19 as shown in Figure 5. In fact, the belongingness scores of non-minorities regardless of gender are remarkably similar to each other pre-and post-COVID-19.
When we categorized belongingness as follows (< 0: very low, 0-49: low, 50-99: medium, and ≥ 100: high) and investigated the students that fell into either the very low (N = 21) or high (N = 15) categories we could not find any significant differences between groups (e.g. self-identified gender, race, ethnicity, etc.) other than a statistically significant difference between the two groups in how much they socialize with other students (t(30.313) = -5.197, p<0.001). This supports and extends our previous findings [22,23].

INTERSECTIONALITY
We then investigated the intersectionality of self-identified gender and minority status. Figure 4A shows that women not self-identifying as being a member of a minority had a lower mean belongingness   up with small numbers of many groups. For instance, we only had 17 women not identifying as belonging to a minority. Additionally, we were unable to explore gender beyond those self-identifying as men or women to protect the anonymity of those identifying as non-binary (due to small n). However, we do have evidence that these students do not have the same sense of belonging as students identifying as gender binary.

CONCLUSION
Creating inclusive learning environments for all students is of primary importance. We measured student 'Sense of Belonging' or 'belongingness' which has been shown in other studies to vary according to factors such as race, ethnicity and gender, and to impact attributes such as achievement, motivation, persistence, and student retention. We utilized a survey adapted from the "Math Sense of Belonging Scale" to examine the sense of belonging pre-and post-onset of the COVID-19 pandemic.
This study has provided us with important insight into how a shift to online learning affected the belongingness of our students. At this point we can only conclude that these effects are real, complex, and affect individual students and groups differently -some negatively. The mechanisms of these effects are not yet understood. Solid evidence of causation is likely to take time to establish. If others have observed similar trends at different institutions, working together would likely help us understand these issues better.
It is known that the interrelation of belongingness and other factors is complicated. Unprecedented global events affecting our students only make this more complex. However, work like this is the first step towards determining actions which may help to improve the sense of belonging of our students in all situations. 

DISCUSSION & FINDINGS
Our work is motivated by a university-wide survey of our students where 50% of students ranked "Covid-related stress and anxiety" as "extremely challenging" or "very challenging". In light of recent events surrounding COVID-19, it is important to highlight that these temporary online/hybrid learning situations will undoubtedly challenge the sense of belonging for many students, particularly incoming first year students. We anticipated that this was likely to disproportionately (and possibly negatively) affect students who identify as women and as minorities in computer science, however, contrary to our expectations we found the following: 1. COVID-19 had a larger impact on the sense of belonging of all students in the space of a few months than we otherwise observed over the two prior years. 2. Men and women who do not identify as being part of any minority appear to have had similar downward shifts in their senses of belonging. Although in our results only men showed a statistically significant decrease, the lack of a significant difference in women is likely due to sample size. 3. Women who do identify as being part of a minority had a large mean increase in sense of belonging post-COVID-19 although this was not statistically significant. Again, sample size may be a factor. 4. Men who do not identify as being part of any minority saw the largest statistically significant drop in belongingness post-COVID-19.
At this point we can only hypothesize as to why the belongingness of students who do not identify as belonging to a minority decreased during the pandemic. Based on previous research we know that online environments fail to replicate critical facets of physical delivery [19], and that sense of belonging and physical campus space are tightly coupled [34]. Additionally, we observed that belongingness appears to increase with social interaction [23]. We also know that gender and race impact sense of belonging [11,14,28] and that we previously observed a lower sense of belonging in students who self-identified as women and as belonging to a minority [23]. However, the reasons that these students experienced an increase in belongingness since the onset of the pandemic seems unintuitive and requires further exploration. Recent research has shown that, for some students, working from home could provide a more supportive environment [30] and it is possible that using remote technologies can help to create a more inclusive environment [19]. Additionally, some of our results may not be due only to the COVID-19 pandemic. Other factors such as the Black Lives Matter movement may be impacting students' belongingness in ways that we do not understand.
The present study is not without limitations, and we are currently working on a multi-institutional study to provide more robust results. Perhaps the chief limitation is sample size. Having access to data from other institutions would help with triangulation. Although we have approximately 450 undergraduate students and 198 responses, we quickly ended