Using tools with Apple's on-device LLM

In macOS/iOS 26 there's a new API for using the on-device LLM with guided generation and tools. It's very simple to load and interact with the model in Swift. First import the FoundationModels framework:

import FoundationModels

Next get the default model. It's a good idea to check whether the model is available (the user might have disabled Apple Intelligence or it might have not been downloaded):

let model = SystemLanguageModel.default
if !model.isAvailable {
    // gracefully handle the error
}

If the model is available, we can send it a prompt:

let session = LanguageModelSession(model: model)
do {
    let response = try await session.respond(to: "What is the weather forecast for Columbus, Ohio?")
    print("output: \(response.content)")
} catch {
    print("error: \(error.localizedDescription)")
}

Unsurprisingly the response is something like:

I'm unable to provide real-time weather updates. However, you can check the latest weather forecast for Columbus, Ohio, by visiting a reliable weather website or app like the National Weather Service, Weather.com, or a local news station's website. These sources will provide the most up-to-date information.

This is understandable, LLMs only know what they "saw" in the training data. To have access to real-time data such as weather forecasts, they need to get data from external sources. To this end, a tool can be used:

struct WeatherTool: Tool {
    let name = "weatherService"
    let description = "Provides weather forecasts for the US."
    
    @Generable
    struct Arguments {
        var latitude: Double
        var longitude: Double
    }

    @Generable
    struct Reply {
        let periods: [Period]
    }

    @Generable
    struct Period {
        let name: String
        let forecast: String
    }
    
    func call(arguments: Arguments) async throws -> Reply {
        print("getting forecast for \(arguments.latitude), \(arguments.longitude)")
        let url = URL(string: "https://arax.ee/weather/forecast")!
        var request = URLRequest(url: url)
        request.setValue("application/json", forHTTPHeaderField: "Content-Type")
        request.httpMethod = "POST"
        request.httpBody = try JSONEncoder().encode(WeatherRequest(latitude: arguments.latitude, longitude: arguments.longitude))
        let (data, _) = try await URLSession.shared.data(for: request)
        let response = try JSONDecoder().decode(WeatherResponse.self, from: data)
        print("got forecast for location: \(response.city)/\(response.state)")
        var periods = [Period]()
        for i in 0..<response.periods.count {
            let period = response.periods[i]
            periods.append(Period(name: period.name, forecast: period.detailedForecast))
            if i == 4 { break }
        }
        return Reply(periods: periods)
    }
}

Now we can "tell" the session that a tool is available and the LLM can decide to use it to get external data.

let session = LanguageModelSession(model: model, tools: [WeatherTool()])

The input arguments of the tool are defined by the Arguments structure. Now the response is something like:

The weather forecast for Columbus, Ohio is as follows:

• Today: Mostly sunny, with a high near 88°F and east wind 2 to 7 mph.
• Tonight: Mostly clear, with a low around 58°F and east wind 2 to 7 mph.
• Wednesday: Sunny, with a high near 86°F and east wind 2 to 6 mph.
• Wednesday Night: Mostly clear, with a low around 59°F and east wind 1 to 6 mph.
• Thursday: Sunny, with a high near 89°F and northwest wind 1 to 5 mph.

Note that the input arguments of the tool are the latitude and longitude of the location we're asking about. When the LLM decides to use the tool, it automatically converts our input "Columbus, Ohio" into the coordinates.

The response of the tool can be a string or some structured data. We return the forecast data as a structured list of periods. The LLM converts the structured response into natural language.

The session has a context window so we can, for example, ask the model to convert the temperature from ˚F to ˚C:

let response = try await session.respond(to: "Convert the temperature in the forecast to degrees Celsius.")
print("output: \(response.content)")

The response is something like:

Here are the converted temperatures for Seattle:

• Today: High near 30.6°C, Afternoon Low around 29.4°C
• Tonight: Clear. Low around 15°C, With temperatures rising to around 61°F (16°C overnight)
• Wednesday: Sunny. High near 24.4°C, Afternoon Low around 23.3°C
• Wednesday Night: Partly cloudy, with a low around 13.3°C.
• Thursday: Mostly sunny, with a high near 21.7°C.