LLMs with dynamically loaded tools

As described here, LLMs can be used with tools to access external systems or real-time data. However, front-loaded tools might be problematic. First of all, tool descriptions take up precious space in the context window. Furthermore if many tools are included, it might be hard for the LLM to pick the right one. According to Apple's technote, for their on-device LLM "many tools" means more than 3–5.

An alternative approach is to use "checked" responses. A prompt such as "What is the capital of Greenland?" can be answered by the LLM alone:

let session = LanguageModelSession(model: model)
let options = GenerationOptions(sampling: .greedy, temperature: 0.5)
let prompt = "What is the capital of Greenland?"
let response = try await session.respond(to: prompt, generating: CheckedResponse.self, options: options)

The value of the response.content.isFullyAnswered property indicates whether the LLM was able to generate a meaningful response. A prompt such as "What is the weather forecast for Seattle?" will fail because it requires access to external knowledge.

let prompt = "What is the weather forecast for Seattle?"
let response = try await session.respond(to: prompt, generating: CheckedResponse.self, options: options)

To pick the right tool, the prompt can be converted into a vector (using the NaturalLanguage framework), which can then be used to find the tool with the most similar description vector.

The basic idea is this:

import NaturalLanguage

if !response.content.isFullyAnswered {
    let embedding = NLEmbedding.sentenceEmbedding(for: .english)
    if let vector = embedding?.vector(for: prompt) {
        print("dimension: \(vector.count)")
    }
}

Once a tool has been picked, it can be used to process the prompt in a separate session. We don't want to use the main session to save space in the context window.

This just-in-time approach scales well, it allows us to use tens or even hundreds of tools.