references/middleware.md
# Middleware
Middleware wraps `Generate` calls to add cross-cutting behavior (retries, logging, fallback, gating, sandboxed tools) without touching the flow that uses it. Middleware composes, so a single `Generate` call can stack several behaviors. Built-ins ship in the `plugins/middleware` package; custom middleware is just a Go struct with two methods.
## The mental model
A middleware is a config struct that implements two methods:
```go
type Middleware interface {
Name() string // stable, registered identifier
New(ctx context.Context) (*Hooks, error) // produces a per-call hook bundle
}
```
The same struct value the user passes to `ai.WithUse` is the value the runtime calls `New` on. There is no separate factory parameter and no embedded base type. Per-call state goes in closures captured by `New`. Plugin-level state goes on unexported fields of the struct.
`New` is invoked once per `Generate` call. The returned `*Hooks` is reused across every iteration of the tool loop within that call.
```go
type Hooks struct {
Tools []Tool // injected for this call
WrapGenerate func(ctx, *GenerateParams, GenerateNext) ... // tool-loop iteration
WrapModel func(ctx, *ModelParams, ModelNext) ... // model API call
WrapTool func(ctx, *ToolParams, ToolNext) ... // tool execution
}
```
A nil hook is a pass-through. Implement only what the middleware needs.
## When each hook fires
A `Generate` call executes a tool loop: model produces output, any tool calls execute, results feed back into a new model call, repeat until the model stops. The hooks fire at three different layers of this loop:
| Hook | Fires | Sees |
| --- | --- | --- |
| `WrapGenerate` | Once per tool-loop iteration. `N` tool turns means `N+1` invocations. | The accumulated `ModelRequest`, the iteration index, the streaming callback, and `MessageIndex` (the next streamed-message slot). |
| `WrapModel` | Once per actual model API call, inside the iteration. | The `ModelRequest` about to go to the model and the streaming callback. |
| `WrapTool` | Once per tool execution. May run **concurrently** for parallel tool calls in the same iteration. | The `ToolRequest` and the resolved `Tool`. |
`WrapGenerate` is the right place for logic that needs to see the whole conversation (rewrites, system-prompt injection, message accumulation). `WrapModel` is the right place for logic about the model call itself (retry, fallback, caching). `WrapTool` is the right place for logic about a single tool execution (approval, sandboxing, logging).
## Composition order
`ai.WithUse(A, B, C)` expands to `A { B { C { actual } } }` at call time. Each layer's `next` continuation runs the next inner layer:
```go
ai.WithUse(
&middleware.Retry{MaxRetries: 3}, // outer: retries the whole inner stack
&middleware.Fallback{Models: ...}, // inner: tries fallback models on failure
)
// effective chain: Retry { Fallback { model } }
```
Order matters. `Retry` outside `Fallback` retries the entire fallback cascade as a unit. Swapped, you'd retry the primary first and only fall back after exhausting retries.
## Per-call state
State that should be shared across the hooks of a single `Generate` call lives in closures captured by `New`. Each `Generate` call gets a fresh `Hooks` bundle, so nothing leaks between calls.
```go
type Counter struct{}
func (Counter) Name() string { return "mine/counter" }
func (Counter) New(ctx context.Context) (*ai.Hooks, error) {
var modelCalls int
return &ai.Hooks{
WrapModel: func(ctx context.Context, p *ai.ModelParams, next ai.ModelNext) (*ai.ModelResponse, error) {
modelCalls++
return next(ctx, p)
},
WrapGenerate: func(ctx context.Context, p *ai.GenerateParams, next ai.GenerateNext) (*ai.ModelResponse, error) {
// The same modelCalls variable is visible here because both closures
// capture it from the enclosing New scope.
resp, err := next(ctx, p)
if err == nil {
log.Printf("iteration %d: %d model calls so far", p.Iteration, modelCalls)
}
return resp, err
},
}, nil
}
```
`WrapTool` may be invoked concurrently for parallel tool calls in the same iteration. Any state it mutates must be guarded:
```go
func (Counter) New(ctx context.Context) (*ai.Hooks, error) {
var (
mu sync.Mutex
toolCalls int
)
return &ai.Hooks{
WrapTool: func(ctx context.Context, p *ai.ToolParams, next ai.ToolNext) (*ai.MultipartToolResponse, error) {
mu.Lock()
toolCalls++
mu.Unlock()
return next(ctx, p)
},
}, nil
}
```
`WrapGenerate` and `WrapModel` are not called concurrently within a single `Generate` call.
## Plugin-level state
When a middleware needs resources its config can't carry as JSON (an HTTP client, a database handle, a logger), put them on **unexported** fields of the config struct. The plugin sets them on a prototype, and `ai.NewMiddleware` captures that prototype in a closure that value-copies it across JSON-dispatched invocations:
```go
type Logger struct {
Prefix string `json:"prefix,omitempty"`
out io.Writer // unexported; preserved across JSON dispatch via value-copy
}
func (Logger) Name() string { return "mine/logger" }
func (l Logger) New(ctx context.Context) (*ai.Hooks, error) {
return &ai.Hooks{
WrapModel: func(ctx context.Context, p *ai.ModelParams, next ai.ModelNext) (*ai.ModelResponse, error) {
start := time.Now()
resp, err := next(ctx, p)
fmt.Fprintf(l.out, "%s model call took %s\n", l.Prefix, time.Since(start))
return resp, err
},
}, nil
}
type LoggerPlugin struct{ Out io.Writer }
func (p *LoggerPlugin) Name() string { return "logger" }
func (p *LoggerPlugin) Init(ctx context.Context) []api.Action { return nil }
func (p *LoggerPlugin) Middlewares(ctx context.Context) ([]*ai.MiddlewareDesc, error) {
return []*ai.MiddlewareDesc{
ai.NewMiddleware("logs model call latency", Logger{out: p.Out}),
}, nil
}
```
The Dev UI and other-runtime callers send JSON config; the prototype's value copy preserves `out` (unexported, not in JSON) while `Prefix` is overridden by the unmarshaled config.
## Composition with WithUse
```go
response, _ := genkit.Generate(ctx, g,
ai.WithModelName("googleai/gemini-flash-latest"),
ai.WithPrompt("Explain quantum computing."),
ai.WithUse(
Logger{Prefix: "[trace]"},
&middleware.Retry{MaxRetries: 3},
),
)
```
No registration is required for pure-Go use. `WithUse` calls each value's `New` directly on a fast path; the registry is only consulted for JSON-dispatched calls (Dev UI or cross-runtime). Registration is what makes a middleware visible to the Dev UI and addressable by name.
## Inline middleware
For ad-hoc middleware that does not need Dev UI visibility or a named type, use `ai.MiddlewareFunc`:
```go
ai.WithUse(ai.MiddlewareFunc(func(ctx context.Context) (*ai.Hooks, error) {
return &ai.Hooks{
WrapModel: func(ctx context.Context, p *ai.ModelParams, next ai.ModelNext) (*ai.ModelResponse, error) {
log.Printf("model call: %s", p.Request.Messages[len(p.Request.Messages)-1].Text())
return next(ctx, p)
},
}, nil
}))
```
The adapter satisfies `Middleware` with a placeholder `Name()` of `"inline"`. Inline middleware is resolved on the fast path and never touches the registry, so the placeholder name is fine.
## Application-owned middleware
Use `genkit.DefineMiddleware` to register a middleware your application owns directly. Registration surfaces it in the Dev UI and lets cross-runtime callers reference it by name:
```go
genkit.DefineMiddleware(g, "logs model call latency", Logger{out: os.Stderr})
// Lookup by name (mostly for inspection / cross-runtime dispatch).
desc := genkit.LookupMiddleware(g, "mine/logger")
```
For application code, `DefineMiddleware` is the typical entry point. For plugin authors, `ai.NewMiddleware` (no registration) plus `MiddlewarePlugin.Middlewares()` is the typical entry point. `genkit.Init` registers the returned descriptors automatically.
## Built-in middleware
The `plugins/middleware` package bundles five production-ready implementations. Register the plugin once during `Init` to make them visible to the Dev UI:
```go
import "github.com/genkit-ai/genkit/go/plugins/middleware"
g := genkit.Init(ctx, genkit.WithPlugins(
&googlegenai.GoogleAI{},
&middleware.Middleware{},
))
```
### `Retry`
Retries failed model API calls with exponential backoff and jitter. Hooks `WrapModel`.
```go
ai.WithUse(&middleware.Retry{
MaxRetries: 3, // default 3
InitialDelayMs: 1000, // default 1000
MaxDelayMs: 60000, // default 60000
BackoffFactor: 2, // default 2
NoJitter: false, // default false
// Statuses (default: UNAVAILABLE, DEADLINE_EXCEEDED, RESOURCE_EXHAUSTED, ABORTED, INTERNAL)
// Statuses: []core.StatusName{core.UNAVAILABLE, core.RESOURCE_EXHAUSTED},
})
```
Non-`GenkitError` errors (network, parse, etc.) are always retried. `GenkitError` errors are retried only if their status is in `Statuses`. The backoff respects `ctx.Done()`: a canceled context aborts the retry loop with the last error.
### `Fallback`
Tries alternative models when the primary fails with a fallback-eligible status. Hooks `WrapModel`.
```go
ai.WithUse(&middleware.Fallback{
Models: []ai.ModelRef{
googlegenai.ModelRef("googleai/gemini-flash-latest", nil),
googlegenai.ModelRef("vertexai/gemini-flash-latest", nil),
},
// default Statuses: UNAVAILABLE, DEADLINE_EXCEEDED, RESOURCE_EXHAUSTED,
// ABORTED, INTERNAL, NOT_FOUND, UNIMPLEMENTED
})
```
Each fallback model uses its own `ModelRef.Config()` verbatim; the original request's config is **not** inherited. Compose with `Retry` outside to retry the whole cascade, or inside to retry just the primary before falling back.
### `ToolApproval`
Interrupts any tool call not in the allow list, exposing approval as a human-in-the-loop step. Hooks `WrapTool`.
```go
ai.WithUse(&middleware.ToolApproval{
AllowedTools: []string{"lookup", "search"}, // anything else triggers an interrupt
})
```
The interrupt rides on the existing tool-interrupt machinery. Approve a call by setting `toolApproved: true` in the resume metadata when restarting:
```go
restart, _ := tool.Restart(interruptPart, &ai.RestartOptions{
ResumedMetadata: map[string]any{"toolApproved": true},
})
genkit.Generate(ctx, g, ai.WithMessages(resp.History()...), ai.WithToolRestarts(restart))
```
A bare resume without that flag is **not** treated as approval, so unrelated resume flows can't bypass gating.
### `Filesystem`
Grants the model scoped file access under a single root directory via `list_files`, `read_file`, plus `write_file` and `search_and_replace` when writes are enabled. Hooks `WrapGenerate` and `WrapTool` and contributes `Tools`.
```go
ai.WithUse(&middleware.Filesystem{
RootDir: "./workspace",
AllowWriteAccess: true,
ToolNamePrefix: "", // set distinct prefixes if attaching multiple Filesystem middlewares
})
```
Path safety is enforced by `os.Root` (Go 1.25+), which rejects any path resolving outside the root, including via `..`, absolute paths, or symlinks. `read_file` returns its content as a queued user message on the next turn (not as the tool's direct output) so binary types like images can be inlined as media parts.
### `Skills`
Exposes a local library of `SKILL.md` files as loadable system instructions. Hooks `WrapGenerate` and contributes a `use_skill` tool.
```go
ai.WithUse(&middleware.Skills{SkillPaths: []string{"skills"}}) // default: ["skills"]
```
A skill is a directory containing `SKILL.md`, optionally with YAML frontmatter (`name`, `description`). The middleware injects a system prompt listing available skills, and the model calls `use_skill("name")` to pull the skill body into the conversation on demand. Heavier persona instructions stay off the hot path until actually loaded.
## Practical patterns
### Streaming-aware middleware
If your `WrapGenerate` or `WrapModel` hook emits its own messages (injected user content, system updates), use the streaming callback and the `MessageIndex` cursor in `GenerateParams`:
```go
WrapGenerate: func(ctx context.Context, p *ai.GenerateParams, next ai.GenerateNext) (*ai.ModelResponse, error) {
if p.Callback != nil {
_ = p.Callback(ctx, &ai.ModelResponseChunk{
Role: ai.RoleUser,
Index: p.MessageIndex,
Content: []*ai.Part{ai.NewTextPart("[injected context]")},
})
p.MessageIndex++ // advance so downstream middleware and the model see the shifted index
}
p.Request.Messages = append(p.Request.Messages, ai.NewUserMessage(ai.NewTextPart("[injected context]")))
return next(ctx, p)
},
```
`Filesystem` does this to deliver `read_file` content to the model while preserving streamed-chunk ordering.
### Adding tools from middleware
`Hooks.Tools` registers extra tools for the duration of the call without the user wiring them through `ai.WithTools`. Useful when the middleware's hooks and tools work as a pair (e.g., `Filesystem`'s read/write tools, `Skills`'s `use_skill` tool):
```go
return &ai.Hooks{
Tools: []ai.Tool{myTool},
WrapTool: myInterceptor, // intercepts both myTool and any user-supplied tools
}, nil
```
Duplicate tool names across user-supplied tools and middleware-contributed tools error out at call setup; the call won't run.
### Interrupting from `WrapTool`
`ai.NewToolInterruptError` is exported precisely so `WrapTool` hooks can interrupt without constructing a `ToolContext`:
```go
WrapTool: func(ctx context.Context, p *ai.ToolParams, next ai.ToolNext) (*ai.MultipartToolResponse, error) {
if shouldGate(p.Tool.Name()) {
return nil, ai.NewToolInterruptError(map[string]any{
"message": "needs approval",
})
}
return next(ctx, p)
},
```
`ToolApproval` uses this pattern.
### Modifying the request safely
`p.Request` is the live request for the iteration. Mutating it in place affects later layers. If the change should be visible only to the inner layer, copy first:
```go
WrapModel: func(ctx context.Context, p *ai.ModelParams, next ai.ModelNext) (*ai.ModelResponse, error) {
req := *p.Request
req.Messages = append([]*ai.Message(nil), p.Request.Messages...)
req.Messages = append(req.Messages, extraSystemMessage)
p.Request = &req
return next(ctx, p)
},
```
`Skills.injectSkillsPrompt` shows the same pattern for `ModelRequest` cloning.
### Idempotent re-injection across iterations
`WrapGenerate` runs once per tool-loop iteration. If you inject content into the request, you'll inject it on every iteration unless you mark and detect what you've already added. `Skills` tags its system prompt part with metadata (`skills-instructions: true`) and refreshes that one part in place rather than appending a new one each turn.
## Migration note
The legacy `ai.ModelMiddleware` / `ai.WithMiddleware` API is preserved and marked deprecated. Prefer `ai.Middleware` / `ai.WithUse`, which adds `WrapGenerate` and `WrapTool` hooks plus `Hooks.Tools` for dynamically injected tools.
## See also
- [`tools.md`](tools.md) for tool definition, interrupt/restart machinery used by `ToolApproval`.
- Sample sources under `go/samples/basic-middleware/`: `retry-fallback`, `filesystem`, `skills`.
- The `plugins/middleware` package source for reference implementations.
references/getting-started.md
# Getting Started
## Project Setup
```bash
mkdir my-genkit-app && cd my-genkit-app
go mod init my-genkit-app
go get github.com/genkit-ai/genkit/go@latest
```
Provider plugins ship in the same module under `plugins/`, so they don't need to be fetched separately. Just import the ones you want and run `go mod tidy` afterwards. The available plugins include:
- `plugins/googlegenai` for Google AI and Vertex AI
- `plugins/anthropic` for Anthropic Claude
- `plugins/compat_oai` for OpenAI-compatible APIs (OpenAI, Groq, xAI, etc.)
- `plugins/ollama` for local Ollama models
- `plugins/middleware` for the built-in middleware bundle (`Retry`, `Fallback`, `ToolApproval`, `Filesystem`, `Skills`)
## Initialization
Every Genkit app starts with `genkit.Init`, which returns a `*Genkit` instance:
```go
import (
"context"
"github.com/genkit-ai/genkit/go/genkit"
"github.com/genkit-ai/genkit/go/plugins/googlegenai"
)
ctx := context.Background()
g := genkit.Init(ctx,
genkit.WithPlugins(&googlegenai.GoogleAI{}),
)
```
### The `*Genkit` Instance
The `*Genkit` value `g` is the central registry. Pass it to every Genkit function:
```go
// Defining resources
genkit.DefineFlow(g, "myFlow", ...)
genkit.DefineTool(g, "myTool", ...)
genkit.DefinePrompt(g, "myPrompt", ...)
// Generating content
genkit.GenerateText(ctx, g, ...)
genkit.Generate(ctx, g, ...)
```
Do not store `g` in a global variable. Pass it explicitly through your call chain.
### Init Options
```go
g := genkit.Init(ctx,
// Register one or more plugins
genkit.WithPlugins(&googlegenai.GoogleAI{}, &anthropic.Anthropic{}),
// Set a default model (used when no model is specified)
genkit.WithDefaultModel("googleai/gemini-flash-latest"),
// Set directory for .prompt files (default: "prompts")
genkit.WithPromptDir("my-prompts"),
// Or embed prompts using Go's embed package
// genkit.WithPromptFS(promptsFS),
)
```
### Embedding Prompts
Use `go:embed` to bundle `.prompt` files into the binary:
```go
//go:embed prompts
var promptsFS embed.FS
g := genkit.Init(ctx,
genkit.WithPlugins(&googlegenai.GoogleAI{}),
genkit.WithPromptFS(promptsFS),
)
```
## Genkit CLI
The Genkit CLI provides a local Developer UI for running flows, tracing executions, and inspecting model interactions.
**Install:**
```bash
curl -sL cli.genkit.dev | bash
```
**Verify:**
```bash
genkit --version
```
### Developer UI
Start your app with the Developer UI attached:
```bash
genkit start -- go run .
```
This launches:
- Your app (with tracing enabled)
- The Developer UI at `http://localhost:4000`
- A telemetry API at `http://localhost:4033`
Add `-o` to auto-open the UI in your browser:
```bash
genkit start -o -- go run .
```
The Developer UI lets you:
- Run and test flows interactively
- View traces for each generation call (inputs, outputs, latency, token usage)
- Inspect prompt rendering and tool calls
- Debug multi-step flows with per-step trace data
### Without the CLI
Set `GENKIT_ENV=dev` to enable the reflection API without the CLI:
```bash
GENKIT_ENV=dev go run .
```
## Import Paths
```go
import (
"github.com/genkit-ai/genkit/go/genkit" // Core: Init, Generate*, DefineFlow, etc.
"github.com/genkit-ai/genkit/go/ai" // Types: WithModel, WithPrompt, Message, Part, etc.
"github.com/genkit-ai/genkit/go/core" // Low-level: Run (sub-steps), Flow types
"github.com/genkit-ai/genkit/go/plugins/server" // server.Start for HTTP
)
```
references/flows-and-http.md
# Flows & HTTP
## DefineFlow
Wrap AI logic in a flow for observability, tracing, and HTTP deployment.
```go
jokeFlow := genkit.DefineFlow(g, "jokeFlow",
func(ctx context.Context, topic string) (string, error) {
return genkit.GenerateText(ctx, g,
ai.WithModelName("googleai/gemini-flash-latest"),
ai.WithPrompt("Tell me a joke about %s", topic),
)
},
)
```
### Running a Flow Directly
```go
result, err := jokeFlow.Run(ctx, "cats")
```
## DefineStreamingFlow
Flows that stream chunks back to the caller. Two common patterns:
### Pattern 1: Passthrough Streaming
Pass the stream callback directly through to `WithStreaming`. The callback type is `ai.ModelStreamCallback` = `func(context.Context, *ai.ModelResponseChunk) error`:
```go
genkit.DefineStreamingFlow(g, "streamingJokeFlow",
func(ctx context.Context, topic string, sendChunk ai.ModelStreamCallback) (string, error) {
resp, err := genkit.Generate(ctx, g,
ai.WithModelName("googleai/gemini-flash-latest"),
ai.WithPrompt("Tell me a long joke about %s", topic),
ai.WithStreaming(sendChunk), // passthrough
)
if err != nil {
return "", err
}
return resp.Text(), nil
},
)
```
### Pattern 2: Manual String Streaming
Use `core.StreamCallback[string]` to stream extracted text:
```go
genkit.DefineStreamingFlow(g, "streamingJokeFlow",
func(ctx context.Context, topic string, sendChunk core.StreamCallback[string]) (string, error) {
stream := genkit.GenerateStream(ctx, g,
ai.WithModelName("googleai/gemini-flash-latest"),
ai.WithPrompt("Tell me a long joke about %s", topic),
)
for result, err := range stream {
if err != nil {
return "", err
}
if result.Done {
return result.Response.Text(), nil
}
sendChunk(ctx, result.Chunk.Text())
}
return "", nil
},
)
```
### Typed Streaming Flows
Use `core.StreamCallback[T]` with `GenerateDataStream` for typed chunks:
```go
genkit.DefineStreamingFlow(g, "structuredStream",
func(ctx context.Context, input JokeRequest, sendChunk core.StreamCallback[*Joke]) (*Joke, error) {
stream := genkit.GenerateDataStream[*Joke](ctx, g,
ai.WithModelName("googleai/gemini-flash-latest"),
ai.WithPrompt("Tell me a joke about %s", input.Topic),
)
for result, err := range stream {
if err != nil { return nil, err }
if result.Done { return result.Output, nil }
sendChunk(ctx, result.Chunk)
}
return nil, nil
},
)
```
## Named Sub-Steps
Use `core.Run` inside a flow for traced sub-steps:
```go
genkit.DefineFlow(g, "pipeline",
func(ctx context.Context, input string) (string, error) {
subject, err := core.Run(ctx, "extract-subject", func() (string, error) {
return genkit.GenerateText(ctx, g,
ai.WithPrompt("Extract the subject from: %s", input),
)
})
if err != nil { return "", err }
joke, err := core.Run(ctx, "generate-joke", func() (string, error) {
return genkit.GenerateText(ctx, g,
ai.WithPrompt("Tell me a joke about %s", subject),
)
})
return joke, err
},
)
```
## HTTP Handlers
### genkit.Handler
Convert any flow into an `http.HandlerFunc`:
```go
mux := http.NewServeMux()
for _, f := range genkit.ListFlows(g) {
mux.HandleFunc("POST /"+f.Name(), genkit.Handler(f))
}
log.Fatal(server.Start(ctx, "127.0.0.1:8080", mux))
```
### Request/Response Format
**Non-streaming request:**
```bash
curl -X POST http://localhost:8080/jokeFlow \
-H "Content-Type: application/json" \
-d '{"data": "bananas"}'
```
Response: `{"result": "Why did the banana go to the doctor?..."}`
**Streaming request:**
```bash
curl -N -X POST http://localhost:8080/streamingJokeFlow \
-H "Content-Type: application/json" \
-d '{"data": "bananas"}'
```
Streaming responses use Server-Sent Events (SSE) format.
### genkit.HandlerFunc
For frameworks that expect error-returning handlers:
```go
handler := genkit.HandlerFunc(myFlow)
// handler is func(http.ResponseWriter, *http.Request) error
```
### Context Providers
Inject request context (e.g., auth headers) into flow execution:
```go
mux.HandleFunc("POST /myFlow", genkit.Handler(myFlow,
genkit.WithContextProviders(func(ctx context.Context, rd core.RequestData) (api.ActionContext, error) {
// rd.Headers contains HTTP headers
return api.ActionContext{"userId": rd.Headers.Get("X-User-Id")}, nil
}),
))
```
### ListFlows
Get all registered flows for dynamic route setup:
```go
flows := genkit.ListFlows(g) // []api.Action
for _, f := range flows {
fmt.Println(f.Name())
}
```
references/prompts.md
# Prompts
## DefinePrompt
Define a reusable prompt in code with a default model and template.
```go
jokePrompt := genkit.DefinePrompt(g, "joke",
ai.WithModel(googlegenai.ModelRef("googleai/gemini-flash-latest", nil)),
ai.WithInputType(JokeRequest{Topic: "example"}),
ai.WithPrompt("Tell me a joke about {{topic}}."),
)
```
### Execute
```go
resp, err := jokePrompt.Execute(ctx,
ai.WithInput(map[string]any{"topic": "cats"}),
)
fmt.Println(resp.Text())
```
### ExecuteStream
```go
stream := jokePrompt.ExecuteStream(ctx,
ai.WithInput(map[string]any{"topic": "cats"}),
)
for result, err := range stream {
if err != nil { return err }
if result.Done { break }
fmt.Print(result.Chunk.Text())
}
```
### Override Options at Execution
```go
resp, err := jokePrompt.Execute(ctx,
ai.WithInput(map[string]any{"topic": "cats"}),
ai.WithModelName("googleai/gemini-pro-latest"), // override model
ai.WithConfig(map[string]any{"temperature": 0.9}),
ai.WithTools(myTool),
)
```
## DefineDataPrompt (Typed Input/Output)
Strongly-typed prompts with Go generics.
```go
type JokeRequest struct {
Topic string `json:"topic"`
}
type Joke struct {
Setup string `json:"setup" jsonschema:"description=The setup"`
Punchline string `json:"punchline" jsonschema:"description=The punchline"`
}
jokePrompt := genkit.DefineDataPrompt[JokeRequest, *Joke](g, "structured-joke",
ai.WithModel(googlegenai.ModelRef("googleai/gemini-flash-latest", nil)),
ai.WithPrompt("Tell me a joke about {{topic}}."),
)
```
### Execute (typed)
```go
joke, resp, err := jokePrompt.Execute(ctx, JokeRequest{Topic: "cats"})
// joke is *Joke, resp is *ModelResponse
```
### ExecuteStream (typed)
```go
stream := jokePrompt.ExecuteStream(ctx, JokeRequest{Topic: "cats"})
for result, err := range stream {
if err != nil { return err }
if result.Done {
finalJoke := result.Output // *Joke
break
}
fmt.Print(result.Chunk) // partial *Joke
}
```
## .prompt Files (Dotprompt)
Define prompts in separate files with YAML frontmatter and Handlebars templates.
### Basic .prompt File
`prompts/joke.prompt`:
```
---
model: googleai/gemini-flash-latest
input:
schema:
topic: string
---
Tell me a joke about {{topic}}.
```
### Load and Use
```go
// LookupPrompt returns Prompt (untyped: map[string]any input, string output)
jokePrompt := genkit.LookupPrompt(g, "joke")
resp, err := jokePrompt.Execute(ctx,
ai.WithInput(map[string]any{"topic": "cats"}),
)
```
### Typed .prompt File
`prompts/structured-joke.prompt`:
```
---
model: googleai/gemini-flash-latest
config:
thinkingConfig:
thinkingBudget: 0
input:
schema: JokeRequest
output:
format: json
schema: Joke
---
Tell me a joke about {{topic}}.
```
Register Go types so the .prompt file can reference them by name:
```go
genkit.DefineSchemaFor[JokeRequest](g)
genkit.DefineSchemaFor[Joke](g)
jokePrompt := genkit.LookupDataPrompt[JokeRequest, *Joke](g, "structured-joke")
joke, resp, err := jokePrompt.Execute(ctx, JokeRequest{Topic: "cats"})
```
### LoadPrompt (Explicit Path)
```go
prompt := genkit.LoadPrompt(g, "./prompts/countries.prompt", "countries")
resp, err := prompt.Execute(ctx)
```
### .prompt File Features
**Multi-message prompts with roles:**
```
---
model: googleai/gemini-flash-latest
input:
schema:
question: string
---
{{ role "system" }}
You are a helpful assistant.
{{ role "user" }}
{{question}}
```
**Media in prompts:**
```
---
model: googleai/gemini-flash-latest
input:
schema:
videoUrl: string
contentType: string
---
{{ role "user" }}
Summarize this video:
{{media url=videoUrl contentType=contentType}}
```
**Conditionals and loops:**
```
---
input:
schema:
topic: string
dietaryRestrictions?(array): string
---
Write a recipe about {{topic}}.
{{#if dietaryRestrictions}}
Dietary restrictions: {{#each dietaryRestrictions}}{{this}}{{#unless @last}}, {{/unless}}{{/each}}.
{{/if}}
```
**Inline schema in .prompt file:**
```
---
model: googleai/gemini-flash-latest
input:
schema:
topic: string
style?: string
output:
format: json
schema:
title: string
body: string
tags(array): string
---
Write an article about {{topic}}.
{{#if style}}Write in a {{style}} style.{{/if}}
```
## Schemas
### DefineSchemaFor (from Go type)
Registers a Go struct as a named schema for use in `.prompt` files.
```go
genkit.DefineSchemaFor[JokeRequest](g)
genkit.DefineSchemaFor[Joke](g)
```
The schema name matches the Go type name. Use `jsonschema` struct tags for metadata:
```go
type Recipe struct {
Title string `json:"title" jsonschema:"description=The recipe title"`
Difficulty string `json:"difficulty" jsonschema:"enum=easy,enum=medium,enum=hard"`
Ingredients []Ingredient `json:"ingredients"`
Steps []string `json:"steps"`
}
type Ingredient struct {
Name string `json:"name"`
Amount float64 `json:"amount"`
Unit string `json:"unit"`
}
```
### DefineSchema (manual JSON Schema)
```go
genkit.DefineSchema(g, "Recipe", map[string]any{
"type": "object",
"properties": map[string]any{
"title": map[string]any{"type": "string"},
"ingredients": map[string]any{
"type": "array",
"items": map[string]any{"type": "object"},
},
},
"required": []string{"title", "ingredients"},
})
```
references/generation.md
# Generation
## GenerateText
Simplest form. Returns a string.
```go
text, err := genkit.GenerateText(ctx, g,
ai.WithModelName("googleai/gemini-flash-latest"),
ai.WithPrompt("Tell me a joke about %s", topic),
)
```
## Generate
Returns a full `*ModelResponse` with metadata, usage stats, and history.
```go
resp, err := genkit.Generate(ctx, g,
ai.WithModelName("googleai/gemini-flash-latest"),
ai.WithSystem("You are a helpful assistant."),
ai.WithPrompt("Explain %s", topic),
)
fmt.Println(resp.Text()) // concatenated text
fmt.Println(resp.FinishReason) // ai.FinishReasonStop, etc.
fmt.Println(resp.Usage) // token counts
```
## GenerateData (Structured Output)
Returns a typed Go value parsed from the model's JSON output.
```go
type Joke struct {
Setup string `json:"setup" jsonschema:"description=The setup of the joke"`
Punchline string `json:"punchline" jsonschema:"description=The punchline"`
}
joke, resp, err := genkit.GenerateData[Joke](ctx, g,
ai.WithModelName("googleai/gemini-flash-latest"),
ai.WithPrompt("Tell me a joke about %s", topic),
)
// joke is *Joke, resp is *ModelResponse
```
## Streaming
### GenerateStream
Returns an iterator. Each value has `.Done`, `.Chunk`, and `.Response`.
```go
stream := genkit.GenerateStream(ctx, g,
ai.WithModelName("googleai/gemini-flash-latest"),
ai.WithPrompt("Tell me a long story about %s", topic),
)
for result, err := range stream {
if err != nil {
return err
}
if result.Done {
finalText := result.Response.Text()
break
}
fmt.Print(result.Chunk.Text()) // incremental text
}
```
### GenerateDataStream (Structured Streaming)
Streams typed partial objects as they arrive.
```go
stream := genkit.GenerateDataStream[Joke](ctx, g,
ai.WithModelName("googleai/gemini-flash-latest"),
ai.WithPrompt("Tell me a joke about %s", topic),
)
for result, err := range stream {
if err != nil {
return err
}
if result.Done {
finalJoke := result.Output // *Joke
break
}
partialJoke := result.Chunk // *Joke (partial)
}
```
### Callback-Based Streaming
Use `ai.WithStreaming` with `Generate` for callback-style streaming. The callback receives `*ai.ModelResponseChunk`:
```go
resp, err := genkit.Generate(ctx, g,
ai.WithModelName("googleai/gemini-flash-latest"),
ai.WithPrompt("Tell me a story"),
ai.WithStreaming(func(ctx context.Context, chunk *ai.ModelResponseChunk) error {
fmt.Print(chunk.Text()) // extract text from chunk
return nil
}),
)
// resp contains the final complete response
```
## Common Options
```go
// Model selection
ai.WithModel(googlegenai.ModelRef("googleai/gemini-flash-latest", nil)) // model reference
ai.WithModelName("googleai/gemini-flash-latest") // by name string
// Content
ai.WithPrompt("Tell me about %s", topic) // user message (supports fmt verbs)
ai.WithSystem("You are a pirate.") // system instructions
ai.WithMessages(msg1, msg2) // conversation history
ai.WithDocs(doc1, doc2) // context documents
ai.WithTextDocs("context 1", "context 2") // context as strings
// Model config (provider-specific)
ai.WithConfig(map[string]any{"temperature": 0.7})
```
## Output Formats
Control how the model structures its output.
### By Go Type
```go
// Automatically uses JSON format and instructs model to match the type
ai.WithOutputType(MyStruct{})
```
### By Format String
```go
ai.WithOutputFormat(ai.OutputFormatJSON) // single JSON object
ai.WithOutputFormat(ai.OutputFormatJSONL) // JSON Lines (one object per line)
ai.WithOutputFormat(ai.OutputFormatArray) // JSON array
ai.WithOutputFormat(ai.OutputFormatEnum) // constrained enum value
ai.WithOutputFormat(ai.OutputFormatText) // plain text (default)
```
### Enum Output
```go
type Color string
const (
Red Color = "red"
Green Color = "green"
Blue Color = "blue"
)
text, err := genkit.GenerateText(ctx, g,
ai.WithPrompt("What color is the sky?"),
ai.WithOutputEnums(Red, Green, Blue),
)
```
### Custom Output Instructions
```go
ai.WithOutputInstructions("Return a JSON object with fields: name (string), age (number)")
```
### Combining Format + Schema
```go
// JSONL with a typed schema (useful for streaming lists)
genkit.DefinePrompt(g, "characters",
ai.WithPrompt("Generate 5 story characters"),
ai.WithOutputType([]StoryCharacter{}),
ai.WithOutputFormat(ai.OutputFormatJSONL),
)
```
references/providers.md
# Model Providers
## Google AI (Gemini)
```go
import "github.com/genkit-ai/genkit/go/plugins/googlegenai"
g := genkit.Init(ctx, genkit.WithPlugins(&googlegenai.GoogleAI{}))
```
**Env var:** `GEMINI_API_KEY` or `GOOGLE_API_KEY`
Model names follow the format `googleai/<model-id>`. Look up the latest model IDs at https://ai.google.dev/gemini-api/docs/models.
```go
// By name string
ai.WithModelName("googleai/gemini-flash-latest")
// Model ref with provider-specific config
ai.WithModel(googlegenai.ModelRef("googleai/gemini-flash-latest", &genai.GenerateContentConfig{
ThinkingConfig: &genai.ThinkingConfig{
ThinkingBudget: genai.Ptr[int32](0), // disable thinking
},
}))
// Lookup a model instance
m := googlegenai.GoogleAIModel(g, "gemini-flash-latest")
```
## Vertex AI
```go
import "github.com/genkit-ai/genkit/go/plugins/googlegenai"
g := genkit.Init(ctx, genkit.WithPlugins(&googlegenai.VertexAI{}))
```
**Env vars:** `GOOGLE_CLOUD_PROJECT`, `GOOGLE_CLOUD_LOCATION` (or `GOOGLE_CLOUD_REGION`)
Uses Application Default Credentials (`gcloud auth application-default login`).
Model names follow the format `vertexai/<model-id>`. Same model IDs as Google AI.
```go
ai.WithModelName("vertexai/gemini-flash-latest")
```
## Anthropic (Claude)
```go
import (
"github.com/anthropics/anthropic-sdk-go" // Anthropic SDK types
ant "github.com/genkit-ai/genkit/go/plugins/anthropic" // Genkit plugin
)
g := genkit.Init(ctx, genkit.WithPlugins(&ant.Anthropic{}))
```
**Env var:** `ANTHROPIC_API_KEY`
Model names follow the format `anthropic/<model-id>`. Look up the latest model IDs at https://docs.anthropic.com/en/docs/about-claude/models.
```go
// By name
ai.WithModelName("anthropic/claude-sonnet-4-6")
// With provider-specific config (uses Anthropic SDK types via ai.WithConfig)
ai.WithConfig(&anthropic.MessageNewParams{
Temperature: anthropic.Float(1.0),
MaxTokens: *anthropic.IntPtr(2000),
Thinking: anthropic.ThinkingConfigParamUnion{
OfEnabled: &anthropic.ThinkingConfigEnabledParam{
BudgetTokens: *anthropic.IntPtr(1024),
},
},
})
```
## OpenAI-Compatible (compat_oai)
Works with any OpenAI-compatible API: OpenAI, DeepSeek, xAI, etc.
```go
import "github.com/genkit-ai/genkit/go/plugins/compat_oai"
openaiPlugin := &compat_oai.OpenAICompatible{
Provider: "openai", // unique identifier
APIKey: os.Getenv("OPENAI_API_KEY"),
// BaseURL: "https://custom-endpoint/v1", // for non-OpenAI providers
}
g := genkit.Init(ctx, genkit.WithPlugins(openaiPlugin))
```
Define models explicitly (not auto-discovered):
```go
model := openaiPlugin.DefineModel("openai", "gpt-4o", compat_oai.ModelOptions{})
```
Use with:
```go
ai.WithModel(model)
```
## Ollama (Local Models)
```go
import "github.com/genkit-ai/genkit/go/plugins/ollama"
ollamaPlugin := &ollama.Ollama{
ServerAddress: "http://localhost:11434",
Timeout: 60, // seconds
}
g := genkit.Init(ctx, genkit.WithPlugins(ollamaPlugin))
```
Define models explicitly:
```go
model := ollamaPlugin.DefineModel(g,
ollama.ModelDefinition{
Name: "llama3.1",
Type: "chat", // or "generate"
},
nil, // optional *ModelOptions
)
```
Use with:
```go
ai.WithModel(model)
```
## Multiple Providers
Register multiple plugins in a single Genkit instance:
```go
g := genkit.Init(ctx,
genkit.WithPlugins(
&googlegenai.GoogleAI{},
&ant.Anthropic{},
),
genkit.WithDefaultModel("googleai/gemini-flash-latest"),
)
// Use different models per call
text1, _ := genkit.GenerateText(ctx, g,
ai.WithModelName("googleai/gemini-flash-latest"),
ai.WithPrompt("Hello from Gemini"),
)
text2, _ := genkit.GenerateText(ctx, g,
ai.WithModelName("anthropic/claude-sonnet-4-6"),
ai.WithPrompt("Hello from Claude"),
)
```
references/tools.md
# Tools
## DefineTool
Define a tool the model can call during generation.
```go
type WeatherInput struct {
Location string `json:"location" jsonschema:"description=City name"`
}
type WeatherOutput struct {
Temperature float64 `json:"temperature"`
Conditions string `json:"conditions"`
}
weatherTool := genkit.DefineTool(g, "getWeather",
"Gets the current weather for a location.",
func(ctx *ai.ToolContext, input WeatherInput) (WeatherOutput, error) {
// Call your weather API
return WeatherOutput{Temperature: 72, Conditions: "sunny"}, nil
},
)
```
## Using Tools in Generation
Pass tools to `Generate`, `GenerateText`, or prompts:
```go
resp, err := genkit.Generate(ctx, g,
ai.WithModelName("googleai/gemini-flash-latest"),
ai.WithPrompt("What's the weather in San Francisco?"),
ai.WithTools(weatherTool),
)
// The model calls the tool automatically and incorporates the result
fmt.Println(resp.Text())
```
### Tool Choice
```go
ai.WithToolChoice(ai.ToolChoiceAuto) // model decides (default)
ai.WithToolChoice(ai.ToolChoiceRequired) // model must use a tool
ai.WithToolChoice(ai.ToolChoiceNone) // model cannot use tools
```
### Max Turns
Limit how many tool-call round trips the model can make:
```go
ai.WithMaxTurns(3) // default is 5
```
## DefineMultipartTool
Tools that return both structured output and media content:
```go
screenshotTool := genkit.DefineMultipartTool(g, "screenshot",
"Takes a screenshot of the current page",
func(ctx *ai.ToolContext, input any) (*ai.MultipartToolResponse, error) {
return &ai.MultipartToolResponse{
Output: map[string]any{"success": true},
Content: []*ai.Part{ai.NewMediaPart("image/png", base64Data)},
}, nil
},
)
```
## Tool Interrupts
Pause tool execution to request human input before continuing.
### Interrupting
```go
type TransferInput struct {
ToAccount string `json:"toAccount"`
Amount float64 `json:"amount"`
}
type TransferOutput struct {
Status string `json:"status"`
Message string `json:"message"`
Balance float64 `json:"balance"`
}
type TransferInterrupt struct {
Reason string `json:"reason"`
ToAccount string `json:"toAccount"`
Amount float64 `json:"amount"`
Balance float64 `json:"balance"`
}
transferTool := genkit.DefineTool(g, "transferMoney",
"Transfers money to another account.",
func(ctx *ai.ToolContext, input TransferInput) (TransferOutput, error) {
if input.Amount > accountBalance {
return TransferOutput{}, ai.InterruptWith(ctx, TransferInterrupt{
Reason: "insufficient_balance",
ToAccount: input.ToAccount,
Amount: input.Amount,
Balance: accountBalance,
})
}
// Process transfer...
return TransferOutput{Status: "success", Balance: newBalance}, nil
},
)
```
### Handling Interrupts
```go
resp, err := genkit.Generate(ctx, g,
ai.WithModelName("googleai/gemini-flash-latest"),
ai.WithTools(transferTool),
ai.WithPrompt(userRequest),
)
for resp.FinishReason == ai.FinishReasonInterrupted {
var restarts, responses []*ai.Part
for _, interrupt := range resp.Interrupts() {
meta, ok := ai.InterruptAs[TransferInterrupt](interrupt)
if !ok {
continue
}
switch meta.Reason {
case "insufficient_balance":
// RestartWith: re-execute the tool with adjusted input
part, err := transferTool.RestartWith(interrupt,
ai.WithNewInput(TransferInput{
ToAccount: meta.ToAccount,
Amount: meta.Balance, // transfer what's available
}),
)
if err != nil { return err }
restarts = append(restarts, part)
case "confirm_large":
// RespondWith: provide a response directly without re-executing
part, err := transferTool.RespondWith(interrupt,
TransferOutput{Status: "cancelled", Message: "User declined"},
)
if err != nil { return err }
responses = append(responses, part)
}
}
// Continue generation with the resolved interrupts
resp, err = genkit.Generate(ctx, g,
ai.WithMessages(resp.History()...),
ai.WithTools(transferTool),
ai.WithToolRestarts(restarts...),
ai.WithToolResponses(responses...),
)
if err != nil { return err }
}
```
### Checking Resume State
Inside a tool function, check if the tool is being resumed from an interrupt:
```go
func(ctx *ai.ToolContext, input TransferInput) (TransferOutput, error) {
if ctx.IsResumed() {
// This is a resumed call after an interrupt
original, ok := ai.OriginalInputAs[TransferInput](ctx)
// original contains the input from the first call
}
// ...
}
```