Expand query beyond 3 snippets | Voters

Expand query beyond 3 snippets

complete

Jason Taylor

Expand the model to examine more than the top 3 most relevant snippets when examining and answering queries

March 15, 2023

Mike Knapp

marked this post as

complete

Mike Knapp

Hi Jason - good news - I've launched this now. You can select up to 5 snippets for a prompt :) I've also made it possible to use GPT-4. The combination of 5 snippets + GPT-4 seems to work really
 well. Thanks so much for the suggestion! 
From what I've read, fine tuning won't really solve hallucinations, and, the technique I use with Mottle is the recommended approach. 
I have some ideas on how to even further improve in the coming weeks. Watch this space.

Mike Knapp

Thanks Jason for submitting this! How many would you like to see? The limiting factor is just the context window for GTP-3. We have up to 4,000 tokens to use for the question and answer, but I'm also conscious that tokens = $$$, and for a lot of questions, you won't need to provide too much context.

Can you give me more info on why you think more than 3 would help? (For example, maybe you have very small snippets, and so I can definitely see that > 3 would help there.)

Jason Taylor

Mike Knapp: Hey Mike. So in my testing it seems like if it could just pull  a few more snippets it might have a bit more context to get a more accurate example.  Working on testing to see if this would be a helpful chatbot to answer questions for our employees.  And there a few snippets that have a similar topic but with a contextual variation.  Pulling more snippets might allow it to better discern which context is appropriate. We don't have super small snippets but the way we have done it is broken up each paragraph (or a few paragraphs if they are very closely related) of our contract into a snippet.  Perhaps this is not the best/ideal method. I admittedly have not done testing to try different snippet break downs.
The short answer is I feel like 5 would be a good starting point. Honestly, in the ideal world this would be configurable by me as the user so I could make the decision of cost vs additional context advantage.
How does this methodology compare to using a fine-tuning model to train the bot on?

Mike Knapp

Jason Taylor: This is launched now! Thanks so much Jason for the idea :)

Jason Taylor

Mike Knapp: Great work and impressed with how fast you roll out new features. One note on this. The debug still only shows 3 snippets used even if you have your settings using more.

Mike Knapp

Jason Taylor: Thank you, great catch. The data is being captured, I just haven't got around to fixing the screen yet. (I'm thinking of overhauling it completely, but may do a quick fix in the meantime.)