Developer perspectives on the ethics of AI-driven neural implants: a qualitative study

Convergence of neural implants with artificial intelligence (AI) presents opportunities for the development of novel neural implants and improvement of existing neurotechnologies. While such technological innovation carries great promise for the restoration of neurological functions, they also raise ethical challenges. Developers of AI-driven neural implants possess valuable knowledge on the possibilities, limitations and challenges raised by these innovations; yet their perspectives are underrepresented in academic literature. This study aims to explore perspectives of developers of neurotechnology to outline ethical implications of three AI-driven neural implants: a cochlear implant, a visual neural implant, and a motor intention decoding speech-brain-computer-interface. We conducted semi-structured focus groups with developers (n = 19) of AI-driven neural implants. Respondents shared ethically relevant considerations about AI-driven neural implants that we clustered into three themes: (1) design aspects; (2) challenges in clinical trials; (3) impact on users and society. Developers considered accuracy and reliability of AI-driven neural implants conditional for users’ safety, authenticity, and mental privacy. These needs were magnified by the convergence with AI. Yet, the need for accuracy and reliability may also conflict with potential benefits of AI in terms of efficiency and complex data interpretation. We discuss strategies to mitigate these challenges.

In the clinical translation (to the patient)?6.
These concerns, are they general concerns (for neurotechnology) or are they more specific for the [device]?-In the broader field of neurotechnology, where do the greatest concerns or threats lie?Conditions for clinical translation 7.
When do you think would be the right moment to move to first-in-human trials?/ When do you think it is acceptable to initiate clinical translation to the first use in humans?Are there any particular conditions that must be met?-For instance: What are the ideal candidates for a trial?8.
When would you consider a (FIH) trial to be successful?-When is it is not a success?-What could be the main risks involved?9.
When would it then be the right moment to allow the [device] to be more broadly implemented in clinical practice?
Potential impact on field of work 10.What impact do you think the [device] could have on science and your work?11.How do you view the involvement/influence of Silicon Valley?
Remaining questions/remarks 12.Is there perhaps anything else you would like to say, that you feel is important to mention or that we may have forgotten to ask?
*The device of topic in each of the focus group (i.e. the AI-driven visual neural implant, the AI-driven cochlear implant and the AIdriven speech-BCI), can be entered in this topic list under "[device]".

Number Argument Illustrative quote 1a
The speech BCI will have to beat the current gold standard eye-tracker technology in terms of speed.
"Well, we have to beat the eye tracker, which is... that's already quite a high aim, I think, currently.Because eye trackers, of course they're way slower than the way in which we communicate, but still, from a BCI perspective, I think eye gaze and eye tracking based communication is much faster than what is currently possible with existing BCI." [Exp.4_1]1b The AI-speech-BCI could provide improvement over current technologies by targeting a population that are unable to use current technologies.
"I think that for those who are unable to use conventional assisted technology in an adequate way and in all situations, I think that's the initial target population that we're currently aiming for."[Exp.4_1]1c The AI-CI could provide an improvement over current technologies when it works just as good as the current system, but is more future-proof.
"So that you then say, okay, instead of the old classical methods, we will move to something that is self-learning or externally steerable, so that patients can upload their own algorithm and that in this way you actually generate a platform for the future into which we can put new things.The speech-BCI can also be improved in terms of versatility and accuracy in order to allow for more natural communication than the eyetracker method.
"Next to speed I also, I guess, wanted to mention also the accuracy, as well as the versatility of the system.Because, I guess, if we talk about eye tracker, then is that mostly yes and no kind of responses?At least I know that there's some technology, current technology that's based on the yes and no type of communication.And of course, they want to bring it a step further and help people communicate in a more natural way.They could actually communicate with sentences, words and something that comes more from them rather than just yes and no responses, which is, I think, a very big step forward too."[Exp.4_5]1h Involving feedback of users in the design process of the speech-BCI contributed to the design and was good for the ivory tower way of thinking.
"The interesting experience we had is that, as far as the human interaction, was that the feedback of the users... so really, they were designing with us.
We changed a lot of things, we added features in the software which they requested, and that helped a lot.[…] That was very good for our ivory tower way of thinking."

[Exp.4_3] 1i
There are many risks involved in brain decoding and in brain modulation, which has to be done safely and as intended by the developers.
"I think in our case there's tons of risks because we have to read information from the brain, and it has to be done accurately.And then we have to modulate brain activity, and it has to be done accurately.And somehow this has to be done in a way which that's been designed safely and tends to behave as we would like it to do." [Exp.4_2]1j Fundamental research on the underlying mechanisms of brainreading and brain activity modulation should be done to mitigate risks on unforeseen issues.
"And there can be foreseen circumstances, activities in the brain that we didn't foresee, and that can cause issues, so there has to be... That's something that we are not doing, because we are really preclinical, but that's something to take into account, all the protection measures that have to be in place.Patients with progressive ALS are highly motivated trial participants.
"I think the motivation is strongest in the disorders that are very progressive, like ALS.We also have the slow progressive disorders.I think ALS patients are a very specific group, because usually they are in the midst of their life, it's a sudden event, and they have a very short horizon.So, in the few years to come, they are willing to contribute to science or to look for solutions for themselves.But I think in this group motivation is very high.We also see it in medicine trials.They're always motivated for everything." Willingness of participants to undergo invasive ventilation and opt for reanimation reduces risks around surgery and shows motivation to continue their lives, which is desirable for lengthy speech-BCI trial.
"our clinical trials are very long, we aim for at least a year, but we are very happy if they continue in the study for many years.
[…] We, as scientists, want to feel both for the risk... It's about invasive ventilation, so if patients are not willing to go in to have invasive ventilation, then the risk of surgery may be too high.If they're willing to do that, then we could take the chance.
[…].I think that shows the will to, the motivation to continue for a long time.And that's important.
[…] They also have... I'm not really sure, so patients who want to get reanimation if something is happening, that also shows that they are very motivated to continue their life with all the limitations they have.The use of language in communication with patients is important for the management of expectations.
"Well, beyond what a patient can and may expect from the chip, use of language is incredibly important.Because the bit about 'seeing' everyone interprets differently.We also used to say: it's not seeing, but it's visual perception, so perceiving.We constantly correct that.But in our eyes, it wasn't seeing, and I don't know how it is for someone else." [Exp.2_1]2l It is important that a potential participant reasonable and accepting in their current situation, as they might return to this situation after the trial ends.
"I think it's important that a patient, the current situation that they're in, that they're solid in that, because there is indeed a return to that situation again eh, when we talk about the temporality of it.And the moment that the desire otherwise becomes so great that it is all going to change so much and that it is wobbly and not anchored, yes, […] there has to be a solid foundation in someone, in a reasonableness and acceptance.I hate that word so much by the way, but at least the situation visually as it is for this patient, that that is calm." The planned explantation of the AI-VNI has to be communicated clearly to trial participants.
"Well, the very first time [name principal investigator] said: 'yes, but at some point we will have to take out that chip again, maybe after three months.'Well, at least it landed with us that we were like: 'Hey, take it out?Okay.There's always a risk, and people have to realise that yes, they will be some kind of a guinea pig.Well, that's always in the pioneer phase.
We have to realise, and we have to be clear to patients, that we are in a pioneer phase, that this might happen.In our organisation it's quite common that these things happen, patients know."[Exp.4_4]2r Even if trial participants for a speech-BCI may not benefit from the device, they can still get a good feeling from participation as they are doing something helpful for society.
"But also, we learned a little bit from our studies, but also from other studies where it was clear there's no benefit, but it is still a large time investment, that they see it as a kind of job or as a kind of doing something for society, which gives them a good feeling that they are doing something helpful for society.In that sense the motivation is on different levels, and that is very interesting"

[EXP.4_3] 2s
There are stories about tetraplegic patients for whom it was a great shock that after participating in a trial for an exoskeleton that they had trained for to control, they had to stop using it "Can I also mention something about the technology abandonment?At least some of the stories that I heard come also from the other side when, in tetraplegic patients and tetraplegic users, they for example invited people to participate in a clinical trial and it took people, participants, a lot of effort to -something that [Exp.4_1] also mentioned -to actually train to control that exoskeleton.But then, after some period of time, the clinical trial was over and they had to stop using that technology and that was a huge change for the person, a huge shock, because they had to give up something that they've worked so hard on for such a long time, and then suddenly it was over.And it was a huge thing.Regaining the option to communicate for people who have lost the ability to speak, may have a strong impact on their quality of life.
"we know that […] adequate communication is really important for the quality of life of people.And at least for those who are tetraplegic and have lost the ability to speak.So regaining that option, I think that could have a very strong impact on the quality of life there, also their ability to be autonomous and have... you now, be able to control their life, basically, independence, etc." [Exp.4_1]3l The speech-BCI is really a success when participants can continue to use it at home.In the AI-CI there is a tension between the advantage of automatic AI-based scene selection and the level of user-control.
"One moment you want to talk to someone during a concert and the next moment you want to hear that very music.And at the same time nobody wants to notice that the system is switching, but actually what normal hearing people can do, they can indeed put their focus on from one to the other.And so that you have these very complex situations with switches of what you want to listen to, how build that into such a system.
[…] And it may well be that then you do have to give a user a choice, I think that works better then, that they have a sense of control."[Exp.3_2]3p Predictive models such as speech and language models used in the speech BCI may lead to autonomy issues when biases leak into the performance.
"speech and using language models in itself that people, or the computer, is going to pronounce sentences that are largely based on language models, but that were not specifically what the user wanted to express.So, there could be also a problematic autonomy issue there, potentially.Preferences regarding the amount of user-control may differ between individual users and it is therefore important to offer the technology which accommodates these varying preferences.
"For example, I have one of those people, and he studies AI then and he, and, yes, he likes to give feedback or play with it himself.We also have an audiologist here, who is quite old, but he is adjusting himself and he wants to have total control.But, for example, older people who are like, please let it just be completely automatic, I don't get this, I don't want this, I want to push a button on or off.Developing a VNI would identify a solution for people who are disadvantaged and could provide hope "I think hope is the big one, that we really identify some solutions or therapies for people who are among the most disadvantaged in society.That gives sighted people hope that if they end up in that position, that they for those who may find themselves in this situation later in life.
at least have something to look forward to and that other people are also trying their best to alleviate difficult conditions."

[Exp.1_4] 3ii
There is a risk of societal disappointment caused by big promises of companies working on BCIs, as the progress will be slow and limited in regards of target population.This could negatively impact societal and patients' acceptance.
"I see also another societal risk and that is that, you know, because of this study and other studies and big promises made by some companies that are working on BCIs, I think the hopes are high, they're growing.And progress will be slow, because much of the validation studies will be limited to, you know, one, two, three participants at most.And also the initial target population is very small, so progress is most likely going to be much slower than what society may expect based on all the news outlets, etc., etc.So, I think that there's also a risk of societal disappointment or so, at some point, which may affect in the end the acceptability or the acceptance of society and patients of this kind of devices."[Exp.4_1]3jj Availability of neural implants may be at risk as their small target group, small study sizes, high costs and effectiveness compared to other assistive technology, may make limit chances of reimbursements."There's also a risk of hope and availability.In the end, when the system is ready, it has to reimbursed or not.And we have the problem of quite a small target group, small experiments, probably quite high cost, and the effectiveness of the system compared to the other assistive technology that's available.I think that's quite a risk.And it's also a risk."[Exp.4_4]3kk The aim to develop a successful AI-CI, that offers a significant improvement over current technologies, is primary one of costefficacy.
"This is more of a matter of costs than an ethical question I think.We think every little step you make is nice [...]

Supplementary Table 1: Topic list and questions Topics Questions
And then you might even say, well, it's just as good as the current state of the art, but we can do a lot more in the future to further steer development." "And in addition to that, I think that that functionality, adequate functionality in all situations and at all times of the day would be, I think, important."[Exp.4_1]1g

Supplementary Table 3: Illustrative quotes on challenges in clinical trials Number Argument Illustrative quote
When you hear how burdensome or how demanding it can be, then it's good to know: God, does someone also have a job and to what extent do you want it to have an impact on the everyday things that someone does and to what extent is someone prepared to either give those up or fill them in differently?So of course, that what life demands varies enormously from one stage of life to another."[Exp.2_3]2e Speech-BCI trials require high levels of motivation in from participants as they requiring a large timeinvestment."So, in our case, because we need so much data, especially if we go to AI-like models, then you need a lot of data to get accurate models.So, they will have to, it's a large time investment for them, it's two mornings or afternoons a week at least, and that's for at least a year and possibly many years if it goes well and we get better and better results.They have to have the motivation."[Exp.4_3]2f " At one point in the end, it was also said with the retina chip that the chip could not be removed from the eye, because it is attached with some kind of nails and that that is actually very difficult to remove.That was not well said, made clear, or received by patients, with the result that one person indeed found it very annoying, or finds it, because they still have pain in the eye.Because they thought: it can come back out.So that turned out not to be the story, and with very big consequences.[…]Sothat also that is made very clear to the patient what consequences it can have and how often you can put it in and take it out.It's important that patients are very aware what they are starting when they get into a trial like this.And it's the same for medicinal products.Medical trials stop because the effectiveness is not good enough or the side effects are too big.
..' So it hadn't sunk in with us.[…].Well, of course, that has to be communicated very well to a patient that it's for a short while, and then we can learn from that and go on developing."[EXP.""

Table 4 : Illustrative quotes on users and society
Second Sight had the obligation, which of course pulled the plug on everything.Yes, you see what happened to [name retinal chip user].Well, luckily some were still able to scrounge things together here and there.But you see, on the internet there has been a whole riot about it from all over the world.And rightly so I think, and rightly so yes.That's something you should be able to cover somehow.With cars, I understood that if the supplier goes bankrupt, parts have to be supplied for another 12 years.stillusetheretinalimplant, yet legal issues prevent them from being able to fix these medical devices.asasupportnetworkfor these Second Sight patients.But the thing is that both Second Sight and the researchers are not actually even allowed to...As far as I understand, they're not even allowed to upgrade or fix the implants because this is a medical device.So even if they know how to fix them, you know, like the electronics, they are legally just not allowed to do anything to the implants.So everyone's hands are tied."[Exp.1_4]2zNotallpatientscan have the implant removed, because of age or medical conditions."Icoulddefinitely imagine situations, or now the patients are too old, or they developed some medical condition, and then they can't have it removed or something and then they are stuck with it."[Exp.1_4] "in the media it is portrayed as though the company somehow screwed them over, […] that it's a bit of a grey area, because we don't know exactly what they agreed to in the consent forms.And I expect that for all these first-inhuman studies, they knew and they were told many times, and "But would that be solved then if you have the implant, if you decide to have the implant, you also always buy something or insurance pays for the explantation or the removal of that implant.So yeah, whatever, if something happens, the whole implant will be, there will be funds to remove the implant.Does that solve it?"[Exp.1_2]2yResearchersaretryingtodevelopa support network for Second Sight "there are now researchers who have worked with Second Sight patients, and they are trying to kind of have a community, pull a community together patients who "Agreed.I still think that at times, for us these seem to be very small changes, that at the moment you can do something visually yourself or that something contributes to less dependence because you are visually stimulated after all […] for us that may not seem so substantial, but I have experienced that sometimes it can be in very small things that a person really, well, does something with the self-esteem."[Exp."Itvariesperpatient,but for most people, look, also, impact on life is, for example, can you function in a work situation?And there you notice that there is still a lot to be gained and [that] the very good CI users can participate in meetings and the like, but as you have also noticed at the, well, the users' association, the slightly lesser users will still use type interpreters, they […] are still missing things no matter how hard they try.[…] that has quite an impact[...]if you can […] function again in that kind of situation, it just gives you a total, yes, a better perspective on the job market."[Exp.3_1]3k Yes. Look, what you now have with scene selection, for instance, if the wrong scene is selected by your processor, yes, you are actually helped backwards instead of forwards.And, yes, those scene selection algorithms are very generic and so in a traffic situation it can be very unpleasant if all the traffic noise is suppressed because just that car arrived."[Exp.3_1]3o […]And in the end, I can imagine that...But that's really speculative.At some point, if people with ALS for example, progressive neuromuscular disease, they know that communication will not be an issue; if we do our job really well, you know, the considerations related to quality of life and the desire to remain alive when tracheostomy, invasive ventilation comes into play for example, those discussions may at some point change, perhaps.But that remains to be seen, if that's indeed the case."[Exp.4_1]" "[Exp.4_1] "Yeah, that's what they call bias in AI, so any kind of biases that inherent to those models can also leek into performance, yeah."[Exp.4_5]3q With the implementation of AI in the VNI, it may be difficult for the user to know what the device may have missed."I think that raises a very interesting point, because if you use AI for the image detection, you kind of have some kind of important intelligence, right?But it's hard to learn for the user what intelligence your algorithm lacks.Whereas, if you just have a Canny edge detector, you know, kind of, and I think you can learn as a user, what the benefits are and what is typically missed by a Canny edge detector.But that's a different discussion." "...especially people who are in control of that environment with their current technology and don't get stuck communicating at important, for them important moments, yes, they don't have such a need for that and they have, and they will react negatively to it.[…] whereas I think most of the users who […] as soon as noise comes in start performing worse, that you can draw those in.So I don't expect so much that you're going to make top performers better, but I think you're going to bring up the middle and the bottom."[Exp.3_1]3s […]with this entirely new technology […] I think it is ultimately very important how you are going to offer it.If you're going to offer it as one package on or off, say, the system does everything itself that's really not going to please some people, because they want to keep in the trials […] the second thing, I will say, is reliability.Because it is very difficult to interact with the brain.What will happen, for example, if you present, if you create perceptions in a person that are misleading or are not complete?And that creates either a safety issue when navigating, or you know, you can fall or bump on something that your algorithm didn't capture, or your stimulation algorithm didn't make possible for the person to see.The other thing, I will say, is trustworthiness, because if I'm a blind person and I realize sometimes the reliability of what I'm seeing is not very good, I would be concerned about that.And this related to the human-Failure of algorithms I think is a bit trickier.Huh, you see that in selfdriving cars already a lot, of course, for example.Who takes responsibility for the fact that the algorithm makes a wrong choice there?And yes, now I don't expect that to become a very big problem with cured blind people.But yes, of course there is a risk in that.It's not for nothing that those manufacturers are all so afraid of it."[Exp.2_4]3v If you let the AI for the AI-CI train on a large external server to develop an individualized AI, this would allow for "But depending on how you, say, implement the AI, if you indeed, say, switch it on, hook it up to a very large server somewhere [...] And you let those things train, train, train, train, train, train, for each CI user separately, …] that would work very well, because then you get that personalized care.But that does mean that the surrounding sounds are all stored, also the conversations, also the doctor's visits, everything is put on that Google server or I don't know what happens.So yes, then you do get ethical, GDPR things you have to think about, but that again depends on how you ultimately implement that."[Exp.3_6]3w Even if it is not necessary to record full conversations for scene selection in the AI-CI, a sound fragment needs to be recorded which may cause privacy issues."No, no.No, no.There's certainly a, and it doesn't have to be that all conversations are recorded, but if you have to characterize the situation through a remote server is that situation, you need a fragment of that.And there, of course, you do have issues with privacy."[Exp.3_1]But, yes, let's be very honest we all know that they can remotely turn on the microphone of your mobile phone and listen in.So, yes, and we all have those on the table as well, so in practice, it is indeed, there is something to this, you make the choice and if you want to use it then some of your conversations will have to go through the world.But I think it's also a bit of a utopia to think that nothing happens now."[Exp.
"["provide the resources, because they and the patients and society think that this is a good enough investment.So I think this is one of the reasons that previous companies have failed, the technology is still not mature enough.So that's a kind of financial/business benchmark."[Exp.1_4]3hh but if you have to make a very large investment that allows ten per cent of CI users to improve 1% of the situations, I don't think [the manufacturer] is going to say 'we're going to design new processes for that".[Exp.3_1]3ll The AI-VNI could reduce societal costs by improving equal societal participation and access to the work environment."So, if people are able to participate more equally.[…] I read some statistics that say that blindness causes a lot of costs in terms of healthcare, and also rehab support.So, if people are a bit more independent and maybe have more access to the work environment, that could also help society in terms of decreasing these costs."[Exp.1_4]3mm CIs can reduce societal costs by improving health, life-expectancy, work-years and social isolation and can improve mental and physical wellbeing, by allowing users to take part in society more."Reduce costs because people live longer, that's what it's all about, but the biggest impact of the CI itself is just simply reduce costs of the tech system.Because people are healthier, live longer, can work longer and have a better life, healthier life, which means less hospitalisation, less risk of falling, less risk of social isolation.So this mental or this physical and mental wellbeing, I think that's one of the biggest impacts of CIs.That you have people who can basically take part in society the more we help them to support them in the situations they need.Which is work, which is at school, I mean, if the things work better at school and universities then you have also new, next generation of people that can work and contribute to society.So I think that's, on the bigger scale that's, it's cost reduction of health costs and so on, that's most probably the biggest impact on the society."[Exp.3_3]