How to navigate your engineering team through the generative AI hype

In the last six months, AI, especially generative AI, has been brought into the mainstream by the launch of OpenAI in ChatGPT and DALL-E to the public. For the first time, anyone with an internet connection can interact with an AI that feels smart and useful – not just a cool interesting prototype.

With this rise of AI from sci-fi toy to real-life tool has come a mixture of widely publicized concerns (should we stop AI experiments?) and excitement (four-day work week!). Behind closed doors, software companies are scrambling to get AI into their products, and engineering leaders are already feeling the pressure of higher expectations from the boardroom and customers.

As an engineering leader, you must prepare for the increasing demands placed on your team and take advantage of new technological advances to beat your competition. Following the strategies outlined below will set you and your team up for success.

Channel ideas into realistic projects

Generative AI is approaching the Peak of Inflated Expectations in Gartner’s Hype Cycle. The ideas started to flow. Your peers and the board will come to you with new projects that they see as opportunities to ride the AI ​​wave.

Whenever people think about what’s possible and how technology can make them, it’s a great thing for engineering! But here comes the hard part. Many ideas come to your desk along with a howwhich cannot be anchored in reality.

There might be an assumption that you can plug a model from OpenAI into your application and, presto, high-quality automation. However, if you remove the how and extract the WHAT with the idea, you can discover realistic projects with strong stakeholder support. Skeptics who once doubted that automation is achievable for certain tasks may now be ready to consider new possibilities, regardless of the reason tool you choose to use.

Opportunities and challenges of generative AI

The new-fangled AI that captures headlines is great at quickly generating text, code and images. For some applications, the potential savings in people’s time is enormous. However, it also has some serious weaknesses compared to existing technologies. Consider ChatGPT as an example:

• ChatGPT has no concept of “trust level.” It does not provide a way to determine whether there is more evidence to support its statements versus whether it is making a best guess from word associations. If that best guess turns out to be wrong, it’s still surprisingly realistic, which makes ChatGPT bugs even more dangerous.
• ChatGPT does not have access to “live” information. It can’t even tell you anything about the past few months.
• ChatGPT is ignorant of domain-specific terminology and concepts that are not available to the public so it can be removed from the web. It can associate your internal company project names and acronyms with unrelated concepts from obscure corners of the internet.

But technology has the answers:

• Bayesian machine learning (ML) models (and many classical statistical tools) include confidence limits for reasoning about the probability of errors.
• Modern streaming architectures allow data to be processed with very low latency, either for updating information retrieval systems or machine learning models.
• GPT models (and other pre-trained models from sources such as HuggingFace) can be “fine-tuned” to domain-specific examples. This can greatly improve the results, but it also takes time and effort to curate a meaningful dataset for tuning.

As an engineering leader, you know your business and how to get the requirements from your stakeholders. The next thing you need, if you don’t already have it, is the confidence to evaluate which tool is suitable for the requirements. ML tools, which include a variety of techniques from simple regression models to the large-scale language models (LLMs) behind the latest “AI” buzz, should now become toolbox options you feel confident evaluating.

Evaluating potential machine learning projects

Not every engineering organization needs a team dedicated to ML or data science. But soon, every engineering organization will need someone who can cut through the buzz and articulate what ML can and cannot do for their business. That judgment comes from experience working on successful and failed data projects. If you can’t name this person on your team, I suggest you look them up!

In the interim, as you talk to stakeholders and set expectations for their dream projects, check this checklist:

There is a simpler method, such as rules-based algorithms, already tried for this problem? What specifically doesn’t the simple approach do with ML?

It’s tempting to think that a “smart” algorithm can solve a problem better and with less effort than a dozen “if” statements made by hand from interviewing a domain expert. That is certainly not the case when considering the overhead of maintaining a known production model. If the rules-based approach is prohibitive or too expensive, it’s time to seriously consider ML.

Can someone provide more specific examples of what a successful ML algorithm can provide?

If a stakeholder is hoping to find some odd “observations” or “anomalies” in a data set but can’t provide specific examples, that’s a red flag. Any data scientist can discover statistical outliers but don’t expect them to be useful.

Is high-quality data readily available?

Garbage-in, garbage-out, as they say. Data hygiene and data architecture projects can be essential in an ML project.

Is there a similar problem with a documented ML solution?

If not, it doesn’t mean that ML can’t help, but you have to be prepared for a long research cycle, the need for deeper ML skills in the team and the potential for eventual failure.

Has ‘good enough’ been defined?

For most use cases, an ML model cannot be 100% accurate. Without clear guidance to the contrary, an engineering team can quickly waste time approaching the elusive 100%, with each percentage point of improvement more time-consuming than the last.

In conclusion

Begin evaluating any proposal to introduce a new ML model into production with a healthy dose of skepticism, just as you would a proposal to add a new data store to your production stack. Effective gatekeeping will ensure that ML becomes a useful tool in your team’s repertoire, not something that stakeholders consider a boondoggle.

The dreaded Trough of Disillusionment in the Hype Cycle is inevitable. Its depth, however, is controlled by the expectations you set and the value you provide. Channel new ideas from around your company into realistic projects — with or without AI — and empower your team so you can quickly identify and take advantage of new opportunities created by ML.

Stephen Kappel is the head of data at Code Climate.