LLM Node
The LLM Node processes incoming text prompts and generates responses using a local or specified language model. It enables integration of AI-powered conversational or generative text capabilities within your application, without relying on external APIs.
| Number of inputs | 0 |
| Number of outputs | 0 |
Configuration​
prompt (Optional)​
Type: String
A text prompt that guides the behavior of the LLM when generating responses. This can be used to set the context, style, or instructions for the AI's replies. If omitted, a default prompt will be used.
modelPath (Optional)​
Type: String
The initial model to load in the LLM node. Only used when initializeModel is set to true.
initializeModel (Optional)​
Type: Bool
When set the LLM not will initialize the model when it is created. Default value is false.
contextSize (Optional)​
Type: Int
The context size, in tokens, the model is loaded with. This bounds how much of the conversation the node can keep — see Conversation history. Larger values remember more but cost memory: the KV cache grows linearly with the context size.
temperature (Optional)​
Type: Float
Sampling temperature; defaults to 0.8. Higher values make replies more varied, lower values more
predictable. Set it to 0 to always take the most likely token, which makes generation
deterministic — useful for tests, where the same prompts must produce the same replies.
seed (Optional)​
Type: Int
Sampling seed. Set it to make generation reproducible at a non-zero temperature; omit it for a
random seed on each run. Ignored when temperature is 0, since that path does not sample.
maxTokens (Optional)​
Type: Int
How many tokens one reply may run to; defaults to 0, which leaves the reply bounded only by the
model stopping on its own or by the context filling. Useful where a long reply is a problem in
itself — a spoken reply, or a fixed latency budget.
The ceiling stops the reply wherever it falls, which is usually mid-sentence: nothing here knows where a sentence ends. So treat it as a backstop against a runaway reply rather than a way to ask for a short one — ask for that in the prompt, and trim the trailing fragment in your own code if you cut the reply off. It is logged when it bites:
[LLMNode] Reply stopped at the maxTokens ceiling of 120
Conversation history​
The node keeps the whole conversation so the model can refer back to earlier turns. When the context
fills up, the node evicts the oldest exchanges — one user message and its reply at a time — until
the history fits again, then continues. The system prompt (prompt / the instructions property) is
never evicted, and neither is the most recent exchange.
This means long conversations lose their oldest context gradually instead of being reset. Eviction is logged:
[LLMNode] Evicted 3 oldest exchange(s), 12 retained
Two details worth knowing:
- Some of the context is held back for the reply itself, so eviction starts before the context is completely full. A reply that runs longer than that reserve is truncated rather than allowed to overrun the context, and a warning is logged.
- A single prompt too large to fit the context on its own cannot be answered. The node leaves the conversation untouched and replies asking for a shorter message.
Cancelling a reply keeps the conversation and the message being answered, dropping only the reply itself — so the retained history is the conversation with that turn left unanswered.
Writing the instructions property resets the conversation history — the system prompt changes what
the retained turns mean, so they are discarded deliberately. It also abandons a reply in flight, for
the same reason, and blocks the writing thread until the node is between turns — for an RPC client,
whatever thread wrote the property. That is within a token in practice. If the node does not get
there within two seconds the write is refused: the prompt and the history are left as they were, an
error is logged, and reading the property back returns the prompt still in use.
Properties​
Inherits properties from its parent, Node.
Actions​
Inherits actions from its parent, Node.
loadModel​
Loads an LLM model.
Parameters​
| Name | Type | Description |
|---|---|---|
| modelPath | String | The file path of the model file. |
Return Value​
None.
prompt​
Sends a request to the LLM model with the given text.
Parameters​
| Name | Type | Description |
|---|---|---|
| text | String | The text to send to the LLM model. |
Return Value​
None.
cancel​
Abandons the reply being generated. The node stops within a token, discards the partial reply and keeps the conversation, leaving the turn it was answering unanswered — so a caller tracking the conversation itself needs no resynchronisation afterwards.
A generationCancelled event is emitted instead of responseReceived. Unlike every other way a
reply can end, no final tokenReceived is emitted: the reply is discarded rather than closed, so a
consumer following the token stream has to treat generationCancelled as an ending of its own. A
reply stopped by maxTokens does emit one, because that reply is kept.
A cancel is only taken while there is a reply it can still abandon — from the point the prompt starts
being decoded to the point the reply is committed to the conversation. Outside that window, including
in the moment the reply is being committed, the call does nothing and says so. So a cancel that
reports success is always followed by generationCancelled and never by responseReceived.
Parameters​
None.
Return Value​
| Type | Description |
|---|---|
Bool | Whether a reply was abandoned. false when there was nothing to cancel. |
Events​
Inherits events from its parent, Node.
responseReceived​
Emitted when the LLM model responds.
Data​
| Name | Type | Description |
|---|---|---|
| text | String | The text received from the LLM model. |
| processingTime | Int | Time taken to process the prompt in milliseconds. |
tokenReceived​
Emitted for each token as a reply is generated, and once more with isFinal set when the reply ends —
except when it was cancelled, which emits generationCancelled instead.
Data​
| Name | Type | Description |
|---|---|---|
| token | String | The generated token text. |
| tokenIndex | Int | The 0-based index of this token in the response. |
| cumulativeText | String | The reply so far, including this token. |
| isFinal | Bool | Whether this is the last event for this reply. |
generationCancelled​
Emitted when a reply was abandoned before completion, in response to the cancel action.
No responseReceived and no final tokenReceived follow it.
Data​
| Name | Type | Description |
|---|---|---|
| tokensGenerated | Int | How many tokens of the discarded reply had been generated. |