Brainstorm AI Singapore 2024: How to write effective AI prompts
Teodora DANILOVIC, Prompt Engineer, Autogenai
Category
🤖
TechTranscript
00:00So, in the short time I have with you all today, I want to demonstrate how smart prompting
00:04leads to smart outputs and hopefully give you a few practical tips and techniques that
00:09you can take away and use for your own prompt creation process.
00:14So I work at Autogen AI, where we help organizations around the world write more winning bids,
00:19tenders, and proposals by leveraging large language models and linguistic engineering.
00:24I get a lot of questions around what prompt engineering actually is and what I do in my
00:28role, so I thought it would be useful to clarify that.
00:30Can I get a quick show of hands, who here has used ChatGPT or any other similar platform
00:36for their prompting?
00:38Wonderful, everyone.
00:39Perfect.
00:40So, what most of you have been doing in those instances is called prompt crafting.
00:46Prompt crafting is when you interact real time with a model and give it a prompt for
00:50that individual instance.
00:52You receive useful and relevant responses, but you wouldn't necessarily expect that prompt
00:56to work on any other piece of text that anyone else would use.
01:00So prompt engineering is curating prompts which produce replicable, reliable outputs
01:05to fulfill a specific function, whilst continuously objectively measuring and improving them.
01:11It's about setting up frameworks that scale well in the future with any unknown input
01:15all the time.
01:19There are many prompting techniques out there, but these are some of the most popular.
01:23Today, I will demonstrate a few of these as we consider their benefits and their drawbacks,
01:28and we'll do this by focusing on one task, which is extracting and classifying things
01:32from a data set.
01:36I'll be using AutoGenerate's platform today to demonstrate these prompts, so the feature
01:40that I'll use, it offers multiple options of outputs for each prompt, so you'll see
01:45three options on the right-hand side.
01:47Up here, we have a zero-shot prompt.
01:50It's an instruction with no examples.
01:52It's what everyone does the first time they interact with the large language model.
01:56It works well most of the time, but it does have a few drawbacks.
02:00Sometimes it can lack a nuanced understanding of the task we're trying to achieve, and we'll
02:04see that in practice today.
02:06So in this example, I've asked it to classify a piece of text, and that piece of text has
02:12a debatable output.
02:17So as a human, I would expect the product arrived late, but the quality is excellent,
02:23to generally be a positive statement.
02:25Although it has some positive and some negatives in it, it's weighted heavier on the positive
02:29statement.
02:31We can see that the model has given neutral for all three outputs.
02:37So for this case, zero-shot prompting doesn't encourage that nuanced understanding, and
02:41I probably wouldn't trust it for even larger sets of data.
02:44Here, the model lacks understanding of what I mean by each of those sentiments.
02:48What does positive mean for me?
02:49What does neutral mean for me?
02:53In the previous example, I didn't have enough context.
02:55So one way of providing it with more context is offering examples of what you want.
02:59This is called multi-shot prompting, a shot being an example.
03:04Chain-of-thought prompting means asking the model to think step-by-step and show its reasoning.
03:13So I've given it pretty much the same prompt, but I've given it three examples of what
03:17I think a positive, negative, and neutral statement is.
03:20You can see, as we transform the text, the outputs are far more nuanced.
03:24The first one says positive.
03:26The second one, after explaining its chain of thought, where I've instructed it to think
03:31step-by-step, also concludes to be positive in the end.
03:34It's far closer to what we're looking for.
03:38One thing I will say is that you should be wary of bias with multi-shot prompting.
03:43In this example, the model might take it to mean that positive statement must always
03:47talk about product quality, or it always must mention that the site was confusing, or that
03:51it always has to have one positive and one negative side to it.
03:55If you're using multi-shot prompting for larger sets of data, you have to make sure your examples
03:59cover all bases.
04:01This can be quite difficult to do, as you have to think of every way that something
04:04can be interpreted.
04:06Chain-of-thought also helps in this scenario, because if I can see the model's thought process,
04:10I can see where it went wrong, and it can help with my model debugging.
04:14Multi-shot here was really great for improving this task and giving the model some more understanding
04:18of my intention.
04:19But sometimes I want to do something a bit more complicated, and I don't want it to give
04:23me just a one-word classification for something.
04:26Let's look at how we can introduce even more complexity with some prompting techniques.
04:33Prompt chaining, or multi-step prompting, is best for complex reasoning tasks that cannot
04:37be instructed in one go.
04:39It ensures you're working on the best piece of text at each stage, and it doesn't leave
04:42room for model inconsistency.
04:45It makes sure that the potentially conflicting instructions don't interfere with one another.
04:50Let's say I want a more complex analysis of some sentiment on a larger body of text.
04:54As a human, I might break this process down into a few different steps.
04:58First, classifying the statements as a whole, then extracting themes from the statement,
05:03and then grouping those themes.
05:04This type of breakdown also works great with prompting.
05:10For the sake of brevity, I'm just showing the output of the first prompt that I gave
05:13the model.
05:15The prompt instructed it to classify a list of customer feedback into sentiments using
05:19the same multi-shot prompt that I showed you guys before, except on a larger set of data.
05:24There's around 25, 26 pieces of customer feedback here.
05:27This is the first prompt in this chain.
05:33The second prompt in the chain is a really good zero-shot prompt.
05:36Up at the top here, you can see we've asked it to identify all the themes in the following
05:41list of customer feedback.
05:43Your response should be an exhaustive list of accurate and relevant themes with a number
05:47beside them indicating how many times it appeared.
05:50Do not write the same theme out twice.
05:53Feedback colon.
05:55So I think it's important to note some key parts of that prompt.
06:01So I've asked it to identify all the themes, and then I've repeated myself and said it
06:05should be an exhaustive list.
06:07Repetition is always good.
06:08I've said that it's a list of customer feedback, which provides the model with context of what
06:12I'm talking about.
06:13I've asked it to be accurate and relevant, so it's clear that I want the themes only
06:17to do with customer feedback.
06:19And then I've said how I want the structure to look.
06:22You can see with all of the options, the structure is exactly how I want it to look.
06:26The answers are exactly what I'm looking for.
06:28So I'm happy with this.
06:30Let me bring it over into the editor.
06:33So as we move on to the third prompt in the chain, you can see how with each step, the
06:37output is a combination of everything that came before it.
06:41Context which is implicitly woven into the answer gets carried on to the next prompt.
06:49The prompt that I've given it here, the third prompt in the chain, is classify the following
06:52themes into positive, negative, neutral, or other categories.
06:56The response should have the sentiment as a heading followed by the themes which fall
07:00under that sentiment.
07:01Under each theme, write a brief justification of why it falls under that sentiment.
07:05And then I've given the list of themes.
07:07As you can see on the right-hand side, this is far more nuanced, accurate, and useful
07:13than the first thing we looked at.
07:16This output is actually a combination of all the techniques we've looked at so far.
07:19And we could not have gotten this type of information which is this thorough and this
07:23accurate through only using one of the techniques.
07:29Once you've got an output that you're happy with, the possibilities really are endless.
07:34You can translate it into JSON.
07:38You can play around with the tone.
07:42You can turn it into a PowerPoint presentation and much, much more.
07:48So to conclude this short segment, simplicity is always best.
07:53Although a single-shot prompt wasn't nuanced or accurate enough for our example today,
07:57it is most often the best choice.
07:59The last three examples I showed you of turning it into JSON or translating the tone, all
08:04of those used a really good, successful zero-shot prompt.
08:08A prompt should be direct, unambiguous, and relevant.
08:12And each of the techniques we've gone through today are just different ways of ensuring
08:15that the prompt meets those requirements.
08:17I really hope this offered some insight into what prompt engineering actually is and gave
08:21you some tools that you can take away and implement into your prompt crafting.
08:26I think we might have time for one quick question, if there are any questions in the audience.
08:32Yes, we have someone here.
08:49How do you think about it, and do you use any tools to refine your prompts?
08:55Yeah, so that's actually a really interesting question.
09:00You can prompt models with how you want them to refine your prompts, interestingly.
09:05So you have to have the techniques in order to instruct the models in how you want your
09:09prompts to look anyway.
09:11Sometimes if I'm lacking inspiration on how to begin writing a prompt, I will definitely
09:15use a model.
09:16I'll be direct, unambiguous, relevant, clear about what my parameters are, clear about
09:20what my instructions are, and often it gives me really good framework.
09:25The only problem with models are it's sometimes a bit more nuanced than that, and you're looking
09:29to write a prompt for a specific use case.
09:30So I know my target audience, I know my customers, and I know what they're looking for.
09:35And I can try to put that into prompting, but ultimately it's your subjective opinion
09:39on whether you think it meets these metrics that you've set out.
09:42So yes, absolutely, you can use other models, not only Claude, but any of the providers
09:48to give you a first draft of a good prompt, and to better your own prompts.
09:55Awesome.
09:56I think that's all the time we have today, but if you have any more questions, I'll be
09:59around afterwards, and please feel free to add me on LinkedIn and ask me any questions
10:04there.
10:05Thank you so much.