All part of building your career
Career growth looks different for everyone, and there is no single path into tech. At Liberty IT, employees have opportunities to learn, develop their skills and explore work that makes an impact.
A year ago, the question people asked about a new AI model was usually simple: ‘Can it do this?’ Today, for Patrick, a Principal Data Scientist at Liberty IT, the more interesting question is different: ‘How do we know it's still doing this, reliably, six weeks after we shipped it?’ That shift, from proving a model works once to proving it keeps working, is at the heart of how data science is changing as AI moves from answering questions to taking real actions inside a workflow.
We spoke to Patrick about what that shift actually looks like day to day: the guardrails that have to go around an AI system before it's allowed to act, why evaluation has become one of the hardest parts of the job, and what good human-in-the-loop design looks like for the insurance industry, where trust isn't optional.
The questions have changed
Patrick, just to set the scene, how would you describe the way your day-to-day work has changed over the past few months?
The biggest shift is that the questions have changed surrounding the more reasoning-intensive work. A year ago, the question was often centred around "can these large language models do X?" Now it's "how do we know it's still doing X reliably six weeks after we shipped it?" A lot more of my time goes into evaluation infrastructure, prompt governance and guardrails, making sure we minimise any risk associated with these generative models, and thinking about failure modes, rather than focussing on just building it and proving that it works at the time of building it.
Where the human has to stay in control
AI is moving from answering questions to actually taking action within a workflow. What's your role, and the role of data scientists in general, in making that shift happen safely?
When a system is just producing text for a human to read, a bad output is annoying. When it's routing a claim, drafting a letter, or flagging a risk that triggers a decision, a bad output has real consequences. My role is defining where that boundary sits and creating guardrails. This means deciding what the model is allowed to do autonomously, where it has to pause and surface something to a person, and what the handoff looks like. A lot of that is less about the model and more about workflow design.
What changes when AI takes real action
Once something you've built is taking real actions rather than just producing text, what changes about how you think about reliability?
You move from "Is this output good?" and "Does it answer the question it's built to address?" to "What happens when it's wrong, and how often is that acceptable (if it's acceptable at all)?" You need to think about failure modes you can't always test for in advance. You also care much more about consistency, a model that's right 90% of the time might not be a big problem in some areas of the business, but the cost of errors can be critical in others, so a lot of thought goes into designing systems to address these issues.
Closing the evaluation gap
You've said evaluation is one of the trickiest problems right now. What does proving a system's quality is holding up once it's live actually look like?
It means building internal benchmarks that reflect your use case. We can't rely on generic NLP leaderboards, as they often don't capture insurance language, or contain the edge cases that matter to us, or are at all grounded in real-world business applications and how mistakes affects the operation of an organisation. In practice, that's things like curated test sets, which often need to be created by hand and reviewed manually, and automated evaluation pipelines, sometimes using a model to judge another model's outputs at scale, but still with human review.
The goal is to catch drift early, because proprietary models get updated silently, prompts change, data shifts, and a pipeline that worked well in January can quietly degrade by June without anyone noticing unless you design your evaluation pipelines and guardrails well.
Trust, explainability and human oversight in insurance
In an industry like ours, where trust matters as much as capability, what's your role in designing good human-in-the-loop systems?
Insurance is fundamentally a trust business, so explainability, auditability, and human oversight aren't "nice to haves" for us; they're non-negotiable. Our role is making sure those requirements are built into the system from the start, not bolted on at the end as an afterthought. That means working with subject-matter experts to understand the intricacies of their work, being clear about what the model contributes versus what a person decides, and making it easy for the person to understand why they're being asked to review something and making that auditing process as seamless as possible for the reviewer.
What data scientists should focus on now
For data scientists working on this kind of thing today, what should they be paying the most attention to?
The evaluation gap. Teams are moving fast shipping LLM features, but it's critical to have reliable ways to know whether quality is holding up after launch, and the curated datasets to do so. The other thing I'd flag is productionisation. LLM pilots succeed quite regularly, but taking something from a pilot and turning it into a launched product takes time and proper planning. Latency, cost, compliance sign-off, reliability at scale, and post-production monitoring are all factors that can stall projects and should be addressed as early as possible.
Why the person, not just the model, still matters
Across every answer, the same idea keeps returning: capability isn't the hard part anymore. What's hard is knowing when a system can be trusted to act, spotting it quietly getting worse, and ensuring a person is positioned to catch what the model can't. That's not a constraint on moving fast with AI; it's what makes it possible to move fast responsibly.
As Patrick's answers show, making AI usable in a regulated, trust-based industry increasingly comes down to the data scientists who build the evaluation, the guardrails and the handoffs around it, not just the model itself - and it's a role that keeps learning and growing as fast as the technology does, drawing on the same responsible AI thinking that runs through data science work across Liberty IT.
