Introduction

The use of AI in education (AIEd) has been a discrete area of study for several decades, albeit the majority of studies have been from a technical development standpoint with less involvement of educators (Zawacki-Richter et al., 2019). Development of AIEd has tended to be concentrated on Intelligent Tutoring Systems (Guan et al., 2020). Use of such technologies in an educational context has not been without its critics (e.g. Selwyn, 2019). Meanwhile, AI has already made a relatively unheralded appearance in low-level features of much technology supporting everyday knowledge work such as search, recommendation, transcription and translation. It has also appeared increasingly within writing support tools, such as grammar checkers, as well as in plagiarism detection.

This picture of gradual change was dramatically disrupted in November 2022 by the launch of ChatGPT. Particularly in education, generative AI has created excitement but is also a considerable concern (Kasneci et al., 2023, Trust et al., 2023; Lo, 2023). The usage figures of ChatGPT show an incredibly rapid rise in popularity and the potential benefits claimed for it are wide-ranging. Much of the fear has revolved around its potential impact on academic integrity. What is lacking to date are in-depth studies that explore how ChatGPT is actually used and experienced by students. Since it is in writing text that generative AI excels, and because writing is central to many forms of learning, including assessment, this paper focuses on how generative AI is changing how students write. With the increasing use of AI in many domains of activity there is a growing interest in defining AI literacy (Long and Magerko, 2020). We build on this work to propose a model of generative AI literacy as a framework to assess student use of ChatGPT in their writing.

In this context, the present study had the aim of uncovering how postgraduate students’ practices of writing were impacted by ChatGPT, with the specific research questions for the study being:

  1. (1)

    How were postgraduate students using ChatGPT and other digital writing tools for writing tasks in the summer of 2023?

  2. (2)

    What do students consider the benefits and problems of ChatGPT’s use?

  3. (3)

    What are the strengths and weaknesses in student generative AI literacy?

The digitisation of writing

The impacts of generative AI tools such as ChatGPT in education, need to be understood in the context of long-term digitisation of writing. The digitisation of writing is a major shift in writing as a fundamental process of expression and learning (Strobl et al. 2019). It carries particular significance because of the cultural value and status placed on good writing, particularly in some academic disciplines. Through the introduction of word processors, then spelling, grammar and style checking, then connectivity, and now generative AI tools, this digitisation process is having profound effects on writing, albeit they remain difficult to pinpoint because they are primarily mental rather than directly visible changes (Kruse and Rapp, 2019).

There has been an acceleration of this digitisation process in the last decade with the growing number of AI-powered writing assistants that are appearing. Godwin-Jones (2022) differentiates four types of such tools:

  • Automatic writing evaluation (AWE) which provides feedback on completed work;

  • Automatic written corrective feedback (AWCF) which offers synchronous feedback on spelling, grammar and or style as text is written;

  • Translation tools;

  • Text generation tools which create bodies of text from a short stimulus.

We could add to this other tools and apps used during the research process at the beginning of writing such as for search, text summarisation (e.g., Scholarcy, iris.ai, summarisebot) and literature reviewing (e.g., ResearchRabbit, Gecko, connectedpapers). There are also well-established tools for referencing which link search and reading to producing a final reference list for a completed assignment (e.g., EndNote, RefWorks, Zotero, and Mendeley).

If writing as a process consists of the stages of “prewriting, planning, drafting, revising, and editing” (Strobl et al., 2019, p. 38), then AWE and AWCF are mostly used in the latter two stages. Our previous research revealed, however, that rephrasing tools such as Wordtune are used both in improving text at the revision stage, but also in breaking through mental blocks in the early stages of writing or even planning (Zhao et al., 2023). Similarly, Malik et al. (2023) found Indonesian students also using a wide range of AI-based tools in their writing. Translation tools might also be used at various stages, such as in processing reading and drafting text (Zhao et al., 2024). So just as writing is a complex iterative process, the use of digital writing tools is complex. Our study of Wordtune also found it being commonly being combined with other writing tools (Zhao et al., 2023). While many such tools have multiple functionalities, they tend to be used for specific tasks for which they are best known. For example, many tools will offer some support for translation, but users tend to have a preferred tool for this function.

Thus, it is important to recognise that many learners were already using multiple tools at different stages of the writing process prior to the dramatic debut of ChatGPT, so the use of generative AI appears in a pre-existing landscape of digital writing. Yet the panoply of tools to support the writing process has been little analysed, particularly from the perspective of how they are used in practice and in conjunction with each other, during writing as a complex, iterative process.

Generative AI

ChatGPT’s launch has been a dramatic, potentially paradigm-shifting intervention, influencing how writing as a central aspect of learning is performed, but also the general perception of AI in Education. It has seen an extraordinary explosion of use, with a claimed 100 million users within two months of its launch (Trust et al., 2023). At the same time, it has been deeply controversial, particularly within education, and has been linked to many of the wider debates on the ethics of AI around bias, privacy and impact on society. Whereas the widespread use in the writing of tools such as Grammarly and Google Translate seems to have been tacitly accepted with relatively little controversy, ChatGPT has drawn huge debate to the digitisation of writing (Adeshola and Adepoju, 2023; AlAfnan et al., 2023; Memarian and Doleck, 2023). This may be partly because tools such as Turnitin have at least partly given teachers the lead in the “arms race” with unfair means. From an educational point of view, it is the impossibility of detecting generative AI’s use that makes it so controversial (Uzun, 2023).

Yet it is hard to deny the power and user-friendliness of ChatGPT. While it remains essentially a form of narrow AI (as opposed to a general AI that mimics the breadth of human intelligence), it does perform a wide range of tasks across the writing process, potentially composing a complete essay, but also including the ability to: summarise readings or a topic, produce an outline for a text, draft text, rewrite text in different styles or lengths, and check grammar and spelling (UNESCO, 2023). Thus, ChatGPT has the ability to write entire texts from a prompt or support specific processes in writing. In addition, it can also write computer code, solve math problems, etc. Yet ChatGPT, at least in its early manifestation, poses many informational and ethical problems (EPIC, 2023) (Table 1).

Table 1 Informational and ethical problems of ChatGPT.
Full size table

Some of these problems are being addressed in later versions of ChatGPT or in other text generation tools such as Bard or the new Bing. Moreover, they are not inherent to large language models but rather features of systems built by BigTech. Nevertheless, given the great AI capabilities of BigTech in terms of resources including data and so their power to define the definition of AI, we anticipate that users need to be aware of such potential issues.

AI literacy

In the last 5 years, there have been growing suggestions of the need to define AI literacy, because of the increasingly pervasive presence of AI in everyday lives and work. A widely cited definition is offered by Long and Magerko (2020):

We define AI literacy as a set of competencies that enables individuals to critically evaluate AI technologies; communicate and collaborate effectively with AI; and use AI as a tool online, at home, and in the workplace.

The authors break AI literacy down under five headings, with 17 components under those headings:

  • What AI is—this is knowledge such as how to recognise AI when it is encountered and understanding distinctions between general and narrow AI.

  • What it can do—this consists of differentiating the tasks AI is good at doing from those it is not good at, and also being able to imagine future uses, reflecting the evolving nature of AI.

  • How AI works—includes ideas such as representation and has an emphasis on data literacy, emphasising learning from data and the need for critical interpretation of data.

  • How it should be used—under which ethics is placed.

  • How people perceive it.

This is useful in contrasting to more technically oriented definitions such as that of Pinski and Benlian (2023) which lack the ethical and critical dimension. Another useful definition is offered by Ridley and Pawlick-Potts (2021) when they suggest that:

Algorithmic literacy is the skill, expertise, and awareness to, understand and reason about algorithms and their processes; recognise and interpret their use in systems (whether embedded or overt); create and apply algorithmic techniques and tools to problems in a variety of domains; assess the influence and effect of algorithms in social, cultural, economic, and political contexts; position the individual as a co-constituent in algorithmic decision-making.

This is a concise expression of the key aspects, integrating notions of algorithmic literacy, which focuses on the way that AI is often encountered indirectly through functions such as filtering and personalisation on online platforms. Levels of student AI literacy have been much investigated, often using Long and Magerko’s (2020) framework (e.g. Kong et al. 2022). However, most of this work was done before the advent of ChatGPT.

The capability of ChatGPT and other generative AI to create a significant body of content from a short prompt has shifted concepts of what AI is. We suggest that this implies the need to update our notion of AI literacy. In the light of the potential and critiques of the technology (discussed above), we can suggest that generative AI literacy might be defined under five headings:

  1. 1.

    Pragmatic understanding: The individual can use generative AI effectively and interpret the information it produces critically

    1. a.

      The individual can pick the right tool for the task, in the context of the proliferation of writing tools (including alternative generative AI to ChatGPT)

    2. b.

      The individual learns to use the chosen tool effectively for a specific task

      1. i.

        Deciding where in the writing process to use it, e.g. for Search, brainstorming, structuring text etc

      2. ii.

        Uses the tool effectively through prompt engineering), such as by

        1. 1.

          Being CLEAR (concise, logical, explicit, adaptive and reflective) (Lo, 2023)

        2. 2.

          Providing context for prompts posed

        3. 3.

          Defining what sort of answer is required

        4. 4.

          Rephrasing questions

        5. 5.

          Asking for sources used

        6. 6.

          Iterating and synthesising results

      3. iii.

        Updating their knowledge as tools develop rapidly

    3. c.

      The individual interprets generative AI outputs critically, given an understanding of how they work and their limits

      1. i.

        Information accuracy, currency, citeability

      2. ii.

        bias

  2. 2.

    Safety understanding: The individual can use generative AI safely

    1. a.

      Is aware of privacy risks

  3. 3.

    Reflective understanding: The individual can assess and take action to manage the impacts of AI on their experience in the educational context

    1. a.

      Impacts on own skills and learning

    2. b.

      Impacts on social connection, including the social aspects of learning

  4. 4.

    Socio-ethical understanding: The individual understands the societal impacts of AI, including

    1. a.

      IPR issues relating to how models are trained

    2. b.

      Impact on information culture, misinformation and disinformation

    3. c.

      Social impacts such as through exploitative process of creation, and the impacts on jobs/ job enrichment

    4. d.

      Equity of access

    5. e.

      Environmental impacts

    6. f.

      Implications of the undue power of BigTech

  5. 5.

    Contextual understanding: The individual understands how to use generative AI appropriately in a particular context and make their own use explicit, as appropriate

    1. a.

      What is appropriate to context

    2. b.

      How to make use transparent and cite appropriately

Generative AI in education

It has been education in particular that has been disrupted by the potential and risks of generative AI in 2023. Yet while much has been written about this in editorials and opinion pieces (Kasneci et al., 2023; Trust et al., 2023) usually to inform educators about how to use it, we are only at the beginning of learning its impact on student behaviour through empirical research. This is important because it seems likely that students have taken up its use far more quickly than teachers.

We do have a few early studies of use by students. A number of surveys by Best Colleges indicate that though US students had concerns about whether it was fair to use ChatGPT for assessments, they were using it and saw it as soon to become the norm (Welding, 2023). Chan and Hu (2023) found Hong Kong students positive about generative AI and willing to use it. This was partly because of its direct uses for brainstorming, individualised assistance with questions, and help with literature reviewing (such as summarisation). But their willingness to use it was partly because they saw it as representing long-term trends in technologies. They had concerns about its accuracy, the transparency of its working, the privacy of their data, the risk of becoming over-reliant, the impacts on employment and conflicts with human values. This accords well with a study by Attewell (2023) based on focus groups with UK students. This again found generative AI being used in a wide range of ways. A similar range of concerns were also expressed such as about the reliability of information from generative AI, privacy, equity of access and fears of becoming over-reliant on it. Students wanted educational institutions to have clearer policies and offer training in the use of generative AI. Interestingly, they also wanted student involvement in generating policy on AI.

If students are generally positive about ChatGPT’s use it is staff who express more concerns. Cardon et al. (2023) conducted a survey of business communication instructors. The main concern of this group of educators was that students would use it to cheat. But they saw a range of negative impacts on learning, such as

  • Less critical thinking/ creativity—itself seen as part of a wider malaise, and the crisis of creativity

  • Less writing skills

  • Less authenticity

  • Less agency because of dependence on such tools

  • Less commitment to authenticity in communication, such as valuing authorial voice and sincerity in communications

They also acknowledged that it can be helpful e.g. in the early stages of writing, and certainly enhanced the efficiency of writing. However they did believe such tools would be used in the workplace, so it was unavoidable that it had to be taught.

Methodology

To answer the research questions, we employed a qualitative methodology within the interpretivist paradigm. We used a combination of semi-structured interviews and observational techniques to gain an understanding of how students selectively employed digital tools in their writing processes and to understand their experiences and concerns regarding the use of generative AI. We recruited participants through an email invitation circulated to students asking for participants who were using “digital tools” for writing. We also asked interviewees to suggest other suitable participants. Our study included 23 participants of diverse nationalities, including students from the UK, China, India, Thailand, Japan, Greece, Malaysia, the USA, and other regions (Table 2). These students were pursuing a range of academic degrees, including postgraduate taught and postgraduate research programmes. All the participants were in the process of undertaking academic tasks, such as writing dissertations or theses. The interviews were conducted in the summer of 2023, this was before the university had issued its policy on AI use.

Table 2 Participant information.
Full size table

At the beginning of the interview, participants were asked to demonstrate their writing process for an academic essay/dissertation and explain how they use digital tools to support their writing. The second part of the interview participants were asked a series of questions, including about the tools they used during the writing process, how they had used ChatGPT, and what their concerns were about it, such as data privacy, inclusivity, accessibility, bias, ethics, and the potential impact of generative AI on education. Thematic Analysis served as our chosen method for analysing the qualitative data, enabling us to gain a nuanced understanding of students’ perceptions of digital writing and ChatGPT in particular (Braun and Clarke, 2006). The research received ethical approval from the University of Sheffield. Voluntary, informed consent was gained from participants. All the data were anonymised for the purposes of analysis and reporting.

Findings

Many tools used in the complex task of writing

One theme that emerged strongly from the data was that students were routinely using a wide range of digital tools (many with an AI component) throughout the academic writing process. The most commonly mentioned tools were grammar checkers (especially Grammarly), paraphrasing tools (Quillbot and Wordtune) and translation tools (e.g., DeepL and Google Translate). Somewhat less commonly other types of tools were in use such as for managing references (Endnote, MyBib) and plagiarism detection (Q-text).

Students who were native English speakers used more basic tools such as Word’s grammar checking or very specialist tools such as to manage references. In contrast, non-English speakers were using a wider range of tools, with considerable experience of having done so built up over time.

Such services seem to be used in quite individualistic ways and critically the impression was that their use had been learned from classmates, social media (such as Youtube or Little Red Book), and trial and error—rather than the institution and educators, although one individual was using Grammarly on supervisory advice.

Sometimes students paid a subscription for such tools; sometimes not. ChatGPT was the most frequently paid-for tool among all those mentioned.

ChatGPT: Used in many different ways

Interviewees talked about other tools as much as about ChatGPT and for these interviewees, it was early days with ChatGPT. Many had used it to only a limited extent. Nevertheless, there was evidence of ChatGPT being employed throughout the writing process. There were frequent mentions of uses to:

  • understand difficult concepts including understanding assignment briefs

  • summarise readings during the research process

  • suggest structures for writing

  • get words down on the page and break through a mental block

  • rephrase text and check grammar

Central uses that ChatGPT was uniquely good at was gaining an understanding of an assignment brief and then structuring ideas:

This is the topic and I first use ChatGPT to give me some idea about this topic. And actually, when I saw this topic, I didn’t really understand it. I use ChatGPT to give me some explanations. [9, Thailand]

So I asked it how to organise an essay about this topic. And then it gave me this structure. So I use this as a reference. [2, China]

Some also used it to check that the final text met the brief:

Then in writing or after writing, I would ask if this paragraph was in line with classwork requirements. I would send it all the requirements, and then ask it if I could write like this? [1, China]

Some used it for searching for literature.

I think the most difficult part of writing an essay is the idea you need to make your own opinions and you need to structure your essay but you need to find examples and literature to support your argument. If you just google it or search […] sometimes you can’t find much relevant information or only a few. […] You can just find less relevant information and articles but with ChatGPT, it will collect the most relevant information for you. [1, China].

For other tasks, such as summarising readings, rephrasing ideas into more academic language and grammar checking participants used other tools as much as ChatGPT. Grammarly, Quillbot, Wordtune and translation tools had heavy use in rewriting. Students tended to use ChatGPT for just one or a few of these tasks, not all of them, linked to which aspect of writing they found hardest.

Students’ individual explorations of the tool had produced quite distinct patterns of use, often linked to the areas of weakness they perceived in their own writing. For example, participant 23 emphasised using it to generate analogies to help them understand complex ideas in the context of moving to study computer science from another discipline:

If I don’t understand, repeat, repeat, repeat, repeat, repeat until I understand and then proceed. And that is how my process is. It’s a lot of repetition, a lot of reclarifying myself and always detecting all this reading and fact checking. [23, Malaysia]

Participant 17 described using in complex way to support reading:

I would copy literally like a whole article into it and then say, summarise this in bullet points. I’ll then write it on a piece of paper because that just helps me to like process it in my head as I write the words. Um, and then after that, and then I’ll read the paper like normal [17, UK]

Participant 15 described using it directly in the writing process:

Sometimes if I’ve written something, what I do is I ask for a feedback and I ask or like, you know, if I want a paraphrasing also. So what I do is I prompt it by asking that the check for flow and paraphrase and then it will kind of give it suggestions and sometimes it adds its own things. And then if I don’t want something, I can just not have it. But most of the times what happens is you get a pretty good idea of like, you know, yeah, this is something good and then you can build up from there. So it gives you that initial kind of a boost and then it becomes easier to build your arguments or build your paragraphs. (15, India)

Participant 20 stands out as potentially using it in a way to simply write ideas for him. He talked about “delegating” tasks to it, constructing himself as in control of the process:

So when I break it down, so usually when I have an academic paper, say it’s 10 pages, I have one thought for each page, like one heading for each page in my head. And once I have that set, then I just start off with ChatGPT straight up. [20, India]

Such complex uses, often involved quite interactive exchanges. Participant 23 used the term “bouncing” ideas off it.

I always have sort of like a conversation with it. [23, Malaysia]

Indeed, one interesting dimension of these interactions was the sense that the experience was something quite close to discussing with another human.

Just directly ask a question. Yes. As if I’m asking a human being. [22, China]

But always, always, always remember it is still a tool. It is not a living being. That is that’s why I keep reminding myself it’s not a living being. Always cross check your back. [23, Malaysia]

The quote implies a genuine struggle not to see ChatGPT as human. Equally, the value of ChatGPT was often linked to not needing to ask tutors or peers for help. Of course, it also gives immediate responses unlike them, but it raises the question of how ChatGPT is impacting the social dimension of learning.

ChatGPT was also being commonly used for non-academic writing, such as for job applications. Several interviewees mentioned using it for computer coding. There were also very specific uses, such as for converting text to Latex.

ChatGPT benefits and worries

The way that they described it, ChatGPT and the other digital tools they used, gave participants a sense of being more efficient and productive.

Thinking about it, basically, there is no big change, because there are still steps in writing, mainly those steps, but the efficiency has been improved. [1, China]

I think he saves me a lot of time when I’m reading like reference that’s really long. Helps me summarise it. I think it’s really efficient. [11, China]

This discourse implied that it did not really change what they did or learned, just speeded the process up and saved them time. This may be a rationalisation and disguise the loss of learning that the “efficiencies” might cause. A lot of the discourse around needing to use tools revolved around “laziness” but also a sense of a lack of confidence and anxiety. Using ChatGPT was often justified for reducing stress.

I’ll say I feel less stressful because we get to work more efficiently by using the tools, by using AI tools compared to the traditional way. [11, China]

There was a sense of ChatGPT being the future and there is an inevitability to it overtaking “traditional” learning practices.

Ten years ago, study style, study style, and the current one is very different. [23, Malaysia]

Yet, while it was being used rather extensively our participants were wary of ChatGPT for a number of reasons. Three were mentioned by most interviewees. The first was the unreliability of the information it output and the need to “fact check” it. There was less mention of specific issues such as failing to acknowledge sources or making up references. A second major concern was that its use would be detected by plagiarism detection tools (a concern they held regardless of the fact that it is generally not detectable). More positively participant 19 wanted to preserve her “voice”

I feel like it doesn’t sound like me and I do want to write in my own voice or in my own words as much as I can. [19, India]

Similarly, participant 23 thought that ChatGPT created a text that often read as “auto-generated” [23] so would not use it in many contexts, e.g. writing job applications because it would not effectively differentiate them from other candidates. Implicitly, they did not see this as a problem in the context of academic writing, suggesting that they did not see individuality as valued in that context.

There were certainly some students who did see using ChatGPT as impacting their learning:

But it sometimes it stopped me from thinking. [12, China]

It will reduce the ability to think independently, that is, reduce the motivation to think independently. Some things need to be understood by yourself. [5, China]

Thus a third major concern was a fear that the use of tools in general produced dependence. There were also a few comments that showed awareness of the risk to privacy of putting personal information into ChatGPT.

Nearly all the concerns reflected on personal impact. Disappointingly, the societal impacts of ChatGPT were not often acknowledged. One interviewee expressed concerns about the impact on low-paid workers. But even when prompted the participants showed little awareness or concern about bias, or ethical issues, e.g., sustainability or the exploitative labour relations used to create ChatGPT. Nevertheless, it seemed that the controversy around ChatGPT had filtered through to problematising its use, in ways that were expressed far less in relation to other tools such as Grammarly.

Discussion

The findings confirm our earlier work from before the release of ChatGPT, that a wide range of tools is used during the writing process (Zhao et al., 2023). Perhaps ChatGPT will displace other tools, but at the time the study was conducted it was reaching a community of users well-versed in using digital tools. It was finding a unique place in helping understand assignment questions, aligning answers to the question and structuring ideas, but was also used to perform tasks such as summarisation and proofreading that others performed with pre-existing tools. Our findings suggested that students with higher English proficiency levels tended to use AI tools less frequently.

The study confirms previous research, that students are keen to use generative AI and see it as part of a general trend in technology development (Chan and Hu, 2023). Our respondents tended to use discourses around time-saving, efficiency, and stress avoidance as justification for using the tools.

ChatGPT was used rather intensively and iteratively but in highly individualistic ways. This differential use may reflect that there was little input at this time from the institution to help students how to use and where to draw the line in terms of appropriate use. Students wanted guidance from the institution on ChatGPT, echoing Attewell’s (2023) findings.

ChatGPT has brought advanced functions to digital writing but also intensified a sense of controversy in this area. For students, the worries focussed particularly on the unreliability of information it produced, fear of being accused of plagiarism if they used it and a concern about growing dependence on technology. While often its use was claimed to be justified for its time and stress reduction this may have underestimated the overall impact on learning. Saving time on learning tasks may unintentionally remove significant opportunities to learn. Many of the deeper ethical and societal issues such as around the exploitative way ChatGPT was developed were not fully understood. Yet it was clear that how writing was done digitally had become controversial. This could be seen as a benefit of ChatGPT, in that a gradual infiltration of digital tools into writing was made more visible institutionally and the controversial dimensions of technology use in education brought to the fore.

Using the model developed above we can point to strengths and weaknesses in students’ emergent generative AI literacy (Table 3).

Table 3 Participants’ Generative AI literacy.
Full size table

Overall our student interviewees showed significant generative AI literacy in most areas, particularly when considering it was early days with their use of it and almost none of them had received support from their teachers or from their institution in understanding how to use it. The weakest area of development was probably appreciation of the societal impacts of generative AI. As instructors catch up with students in understanding AI hopefully they can help students build up a more systematic understanding of pragmatic use of AI, a more reflective approach and a much more critical awareness of the social implications of AI.

Conclusion

While ChatGPT appears as a threat to longstanding practices in education, especially to some genres of academic writing such as essays, it can also be seen as productively bringing to the fore the controversial nature of AI writing technologies which were already creeping unacknowledged into common use. This context produces an opportunity for educators to actively engage with students in a discussion about how writing can be best supported. Our analysis points to areas of weakness in generative AI literacy that need to be strengthened through this process, such as the understanding of the need to select between apps, to have more sophisticated prompt engineering skills, to think more about bias in results, to be more reflective about its use’s impact on learning and have a much stronger appreciation of the societal impacts of generative AI.

The paper is one of the first to explore student use of generative AI in practice and discover in-depth their perception of its benefits and worries about its drawbacks. It has also developed a framework of generative AI literacy as a way of assessing their use. This can be used to plan both institutional policy and instructor support by identifying gaps in AI literacy that need to be filled. For example, educational developers could use the framework to facilitate discussions with teaching staff, aiding in the development of their AI literacy and enhancing their ability to teach AI literacy to students. In addition, universities could apply the dimensions of the AI framework to formulate policies and provide concrete examples that guide learning and teaching practices. The framework also has the potential in evaluating student AI literacy.

The paper has a number of limitations, pointing to where future research can build on its findings. Most participants in this study were using the free version of ChatGPT (3.5) and only a few used the paid version (4). We did not examine in detail the impact of using different versions. While ChatGPT was the main generative AI tool in use at the time of the study, there were others rapidly emerging in popularity. Future research would need to examine how choices of different apps were made and the impact of these choices on writing. Although the participants of this study come from a variety of countries, it is focused on one institution at a particular time. Given the speed of change in technology and educational policy and practice, it is likely that future research will identify rapid shifts in behaviour. But we emphasise the need to examine student writing practices in the context of significant pre-existing use of digital writing tools. Our focus was on postgraduate students. We think less experienced students, such as undergraduates may be quicker and less discriminating in adopting the technology. As generative AI evolves there will be a need to update our definition of generative AI literacy and also to integrate it with notions such as algorithmic literacy, which point to the way that AI operates in rather hidden ways within the infrastructure. This research employed interviews and observations as its main data collection methods. These offer depth of insight but have less power of generalisability. Future studies could usefully seek to validate our findings through quantitative or mixed-methods approaches, such as surveys or experimental studies. Furthermore, future research could expand the scope of this study from AI literacy to the broader concept of writing digitisation, exploring the issue from other perspectives such as psychology and second language acquisition.