Engineering guides / AI and realtime voice integration
By Arte Soft engineering · 24 October 2026
AI tool calling safety: let the server authorize every consequential action
A model can propose an action, but it should not become the authority that decides whether the user may perform it. Treat tool execution as application code with familiar permission and transaction requirements.
Separate proposing from committing
Define whether a tool reads data, prepares a draft or changes a system. Reading a public manual and sending a customer message do not deserve the same treatment. For consequential actions, show the intended effect and collect confirmation at the appropriate point. A vague consent at the start of a conversation does not explain which recipient, amount or record the next action will affect.
Keep tool interfaces narrow
Prefer a small operation with explicit fields over a general command executor. Validate field types, ranges and ownership on the server even if the model output follows the expected schema. Do not accept arbitrary URLs, database queries or file paths without a specific boundary. The application should decide which tools are available for the current person and task rather than exposing the whole backend to every conversation.
Protect against content that looks like instructions
Retrieved documents, messages and attachments may contain text asking the assistant to ignore rules or reveal information. Treat that material as input to interpret, not authority to change permissions. Test adversarial examples inside realistic documents. A prompt can help describe expected behavior, but execution controls must still prevent an unauthorized action when the model proposes one.
Design retries around side effects
A network timeout after an operation may leave the client unsure whether it succeeded. Use stable operation identities, an execution status and idempotent handling where suitable. Do not let the assistant blindly repeat a payment, message or deletion because it did not receive the first response. When uncertainty remains, report it and offer a status check instead of inventing success or creating a second action.
Keep an audit that supports investigation
Record the authorized user context, proposed operation, confirmation and final outcome with appropriate redaction. Most investigations do not require retaining every word of an entire voice session. Include rejected attempts and tool errors in acceptance tests, not only successful calls. Operations staff should be able to distinguish model mistakes from server rejection, permission failures and external service downtime.
Tool execution acceptance sequence
- Ask for an action the current user is not authorized to perform and confirm the server rejects it regardless of model wording.
- Interrupt the connection immediately after a permitted action and check that a retry reports status rather than repeating the side effect.
- Change an important proposed field before confirmation and verify that the committed operation matches the approved version.
Choose a deliberately limited first workflow
Begin with a read-only task or reversible draft before enabling broad changes. Arte Soft’s GPT Realtime Assistant provides a published AI interaction reference; project-specific authorization needs separate engineering. A useful review delivers the tool inventory, permission matrix, failure examples and a testable approval path. It should explicitly state which actions remain unavailable until their confirmation, rollback and support behavior have been agreed with the business owner.
Relevant Arte Soft work
GPT Realtime Assistant — our own published product is our own product or engineering project. It provides context for the work we discuss here; it is not a claim of identical results for every client deployment.
Sources and further reading
Need help with AI action workflow review?
Send the current app, system or brief, the behaviour you need and your constraints. We can agree a discovery, implementation or repair scope with concrete acceptance criteria.
Discuss this with Arte Soft