While executives and managers are enthusiastic about applying generative artificial intelligence (AI) and large language models (LLMs) to their operations, understanding and realizing the returns on investment (ROI) remain significant challenges. This area requires new approaches and skillsets distinct from previous technology waves.
The ROI Challenge
Despite the impressive proofs of concept that AI can deliver, monetizing these innovations is difficult. Steve Jones, executive VP at Capgemini, highlighted this challenge at the recent Databricks conference in San Francisco, noting that ‘Proving the ROI is the biggest challenge of putting 20, 30, 40 GenAI solutions into production.’
Why ROI Matters
Monetizing generative AI requires more than just technical expertise. It demands a deep understanding of business operations and the ability to quantify the benefits of these innovations.
- Generative AI can automate routine tasks, freeing up human resources for more strategic work.
- These models can improve decision-making by providing data-driven insights and reducing biases.
- By streamlining processes, businesses can reduce costs and enhance customer experiences.
However, realizing these benefits is not straightforward. The ROI challenge lies in measuring the impact of generative AI on key performance indicators (KPIs) such as revenue growth, customer satisfaction, or operational efficiency.
The Role of Testing
Essential investments include testing and monitoring LLMs in production. Testing, in particular, is crucial to maintain accuracy and reliability. Jones advised being rigorous and even intentionally ‘poisoning’ models during testing to assess their robustness against erroneous information.
Testing and Validation
Jones shared an example where he prompted a business model with a fictional scenario involving dragons for long-distance haulage. The model responded with detailed but fictional information, highlighting the need for rigorous testing to prevent such errors in real-world applications.
The Risks of Poor Integration
Generative AI, according to Jones, is a technology prone to being poorly integrated into existing systems, adding superficial features while posing security and risk challenges in production. He predicts that generative AI will take two to five years to reach mainstream adoption, a relatively rapid timeline.
Market Competition and Variation
The generative AI market is expected to see intense competition among vendors and platforms. Jones emphasized the need for businesses to focus on efficient and cost-effective use of LLMs, avoiding over-reliance on a single, costly model for all tasks.
Optimizing AI Deployment
Businesses should look for cheaper and more efficient ways to leverage LLMs. Jones suggested being prepared to decommission solutions as quickly as they are commissioned and ensuring all related artifacts are managed in step with the models.
- This requires having a clear strategy for model rotation, ensuring that new models are deployed quickly while old ones are phased out.
- It also involves establishing a robust governance framework to manage the entire AI lifecycle.
- By adopting an agile approach to AI deployment, businesses can reduce costs and improve the overall effectiveness of their AI initiatives.
The Need for Guardrails
Generative AI can produce unexpected results, such as generating irrelevant content from a simple query. Implementing guardrails is essential to prevent such errors and ensure AI solutions deliver meaningful and accurate outputs.
Conclusion
Understanding and proving the ROI of generative AI is a complex task requiring new strategies and rigorous testing. Businesses must focus on efficient deployment, competitive analysis, and robust validation processes to realize the full potential of generative AI technologies.
- This involves adopting an agile approach to AI development, with continuous testing and feedback loops.
- It also requires having a clear strategy for model rotation and governance.
- By prioritizing these factors, businesses can unlock the true value of generative AI and drive meaningful growth and innovation.