Feature engineering, structuring unstructured data and lead scoring

The most common question I’ve received from (AI consulting) clients is: “Build a custom chatbot for me.” While this is a great solution for some problems, it is far from a silver bullet. In this article, I share three alternative ways that companies can use AI to generate value in the context of sales. The approaches include Generative AI, Deep Learning, and Machine Learning.

Image from Canva.

Large Language Models (LLMs) have taken over the business world, and now every company is trying to use Generative AI. While tools like ChatGPT are clearly powerful, it’s not clear how companies can reliably use this technology to create value.

For most of the companies I’ve dealt with, “using AI” means building a chatbot, co-pilot, AI agent, or AI assistant. But as the initial excitement around these solutions fades, organizations are realizing the key challenges of building systems around LLMs.

This is harder than I thought…

A central challenge is that LLMs are inherently unpredictable (even more so than traditional machine learning systems). Therefore, it is it is not easy to get them to solve a specific problem in a predictable way.

For example, one solution to the hallucination problem is to have LLMs evaluate the system responses for accuracy and appropriateness by ‘judges’. However, increasing the number of LLMs increases the cost, complexity and uncertainty of the system.

Solving the right problem

This is not to say that Generative AI (and friends) are not worth pursuing. AI has made countless companies very rich, and I don’t think that will stop anytime soon.

The key point is that value is generated by solving problems, not by using AI (on its own). The promise of AI is realized when companies identify the right problems to solvefor example, Netflix’s personalized recommendations, UPS’s delivery route optimization, Walmart’s inventory management, and much more.

3 AI Sales Use Cases

While “solving the right problem” is easy to say, it’s not easy to do. To help with that, I’m sharing 3 AI use cases for something every business cares about: sales. I hope I’ve sparked your imagination and shown you how to implement them with real-world examples.

The three use cases are:

1. Functional technology — Extract features from text
2. Structuring unstructured data — Making text analysis ready
3. Lead Scoring — Identifying your biggest opportunities

3 AI use cases. Image by author.

Use case 1: Feature engineering

With technologyconsists of creating variables that can be used to train machine learning modelsor perform an analysis. For example, given a set of LinkedIn profiles, extract things like current job title, years of experience, and industry, and then display them numerically.

Extracting years of experience and industry from resume text. Image by author.

Traditionally, this is done in one of two ways. 1) You create features manually, or 2) You buy features from a third party (e.g. credit scores from FICO, business income from D&B). However, LLMs have created a third wayto do this.

Example: Extracting Features from Resumes

Let’s say you qualify leads for a SaaS offering. The software helps protect mid-market businesses from cybersecurity threats. The target customers are IT leaders who decide which vendors are right for their businesses.

You have collected a pile of 100,000 professional profiles and resumes from various sources based on the tags “IT”, “Cybersecurity”, “leader”, “VP” and a few more. The problem is that the leads are of low quality, often including non-IT leaders, entry-level IT professionals and others who do not fit the customer profile.

To ensure that sales efforts are targeted at the right customers, the goal is to filter the leads and only include IT leadersHere are a few ways to solve this problem.

• Idea 1: Manually check all 100,000 leads. Problem: Impractical for one person or a small sales team
• Idea 2: Write rule-based logic to filter resumes. Problem:CVs come in all sorts of formats, which makes logic difficult.
• Idea 3 : Pay a data provider for this information. Problem : This significantly increases customer acquisition costs (~$0.10 per lead)

Given the problems with the ideas above, let’s see how we can solve this problem with a large language model. One simple strategy is to create a prompt that instructs an LLM to extract the desired information from a resume. An example is given below.

Analyze the following text extracted from a resume and determine whether the 
person works in the IT industry. Return a `0` if the person does not work in 
theIT industry, and a `1` if they do. Then, provide a brief explanation for 
your conclusion.

Resume Text:
{resume text}

This solution is a perfect mix of the three ideas above. It (1) evaluates each lead looking for specific information like a person, (2) is automated by a computer program, and (3) you pay less money (~$0.001 per lead).

**Bonus**: For those interested in implementing something like this, I share a sample Python script here that extracts years of experience from a LinkedIn profile using the OpenAI API.

Use case 2: Structuring unstructured data

Data from emails, support tickets, customer reviews, social media profiles, and call transcripts are all examples of unstructured dataThis simply means it is not organized into rows and columnssuch as an Excel spreadsheet or .csv file.

Structured vs. unstructured data. Image by author.

The problem with unstructured data is that it is not analytics-ready, making it difficult to gain insights. This is in contrast to structured data(i.e, numbers organized in rows and columnsTranslating unstructured data into a structured format is another area where recent developments in natural language processing (NLP) and deep learning can help.

Example: Translating CVs into (meaningful) numbers

Consider the same business case from the previous example. Let’s say we’ve successfully selected 10,000 IT leaders from 100,000 leads. While your sales rep can pick up the phone and compose emails, you want to see if you can distill the list prioritize leads similar to previous customers .

One way to do this is to define additional features that provide more granularity to the ideal customer profile (e.g., industry, compliance requirements, tech stack, geographic location), which can be extracted in a similar manner as in Use Case 1. However, Identifying such indicators can be challenging and developing additional automated processes entails costs.

An alternative approach is to use so-called embed textA text embedding is simply a numerical representation of a text fragment that is semantically meaningful. Think of this as translating a resume into a series of numbers.

Convert text to text embeddings. Image by author.

The value of text embeddings is that they translate unstructured text into a structured table of numbers, which is much more amenable to traditional analytical and computational approaches. In this context, one can use text embeddings, for example, to mathematically evaluate which leads are most similar to previous customersand which are the most different.

Use case 3: Lead Scoring

The last usage example is lead score which consists of evaluate the quality of a leadbased on key predictors (e.g. job title, company revenue, customer behavior, etc.). While this is nothing new, recent advancements in AI have made it possible to better parse unstructured data so that it can be fed into lead scoring models.

Example: Evaluating Leads Based on Quality

To conclude our ongoing business case, let’s discuss how we: Use text embeds to prioritize potential customers. Let’s say we have a list of 1,000 past leads, 500 of which have purchased and 500 of which have not. For each lead, we have a profile with key information such as job title, work experience, current company, industry, and key skills.

These leads can be used to train a predictive model that estimates the likelihood that a customer will purchase the product based on their profile. While there are many nuances to developing a model like this, the basic idea is that We can use the predictions from this model to define numbers for each lead(e.g. A, B, C, D), which can be used to categorize and prioritize the 10,000 new items.

**Bonus**: For the more technical readers looking to implement these approaches, I discuss all three use cases applied to real sales data from my company in this article. videoAdditionally, the sample code is freely available at GitHub.

To summarize

AI has enormous potential for businesses. However, to realize that potential, it is necessary to right to solve problems with it.

With the ubiquity of tools like ChatGPT, solution ideas can easily be limited to the AI ​​assistant paradigm. To broaden the space of possibilities, I’ve shared 3 practical AI use cases that leverage alternative approaches.

More about AI for business 👇

AI for Business

My website: https://www.shawhintalebi.com/

3 AI Use Cases (That Are Not a Chatbot) was originally published in Towards Data Science on Medium, where people continued the conversation by bookmarking and commenting on this story.