On-device or cloud AI: choose by task, device coverage and data flow

Begin with a capability matrix

List the operation, required languages, input type and minimum supported device conditions. A text translation model is not automatically a speech recognizer, and an on-device language model is not a universal replacement for a cloud assistant. Check the relevant framework’s availability at runtime. Product planning should distinguish unavailable hardware, missing models and unsupported language combinations because each requires a different user explanation.

Include setup in the experience

A local mode may need a model download before it can work without internet. Make storage, preparation and cancellation visible. Test what happens if setup is interrupted or the user has little free space. A cloud mode avoids some local preparation but depends on connectivity and service access. Neither should claim to work instantly everywhere without testing the actual first-use path.

Draw the data boundary for each mode

Show where microphone audio becomes text, where translation happens and where results are stored. A local speech stage followed by a cloud translation is not an entirely offline workflow. Keep mode descriptions specific enough that users can make an informed choice. If the app changes modes automatically, disclose the rule and preserve any user decision not to send content to an external processor.

Compare operational behavior rather than slogans

Measure task quality, startup delay, steady-state response time, resource use and failure handling. Run on a lower-powered supported device as well as a recent one. An on-device feature has no per-request external API charge, but it still creates engineering, download and support work. A cloud feature may simplify model delivery while introducing metering, account limits and provider changes that your backend must handle.

Design fallback as a product decision

A fallback can ask the user to select another language, prepare a model or explicitly switch to cloud processing. It should not silently charge credits after a person chooses a local mode. When local output is unsuitable, explain that limitation without pretending the application can detect all translation mistakes automatically. Confirm which modes a subscription unlocks separately from whether usage is metered.

Local-versus-cloud decision inputs

• Provide the oldest supported device, priority languages and the amount of setup storage the experience may reasonably require.
• Test the local pathway after disabling internet, then separately test the cloud pathway with a deliberately unavailable connection.
• Check that the selected privacy and billing behavior survives a failed local model preparation, instead of automatically routing content through a different processor.

Accept the modes independently

Require a device-and-language test matrix, a data-flow description and an offline test performed after preparation. Arte Soft’s GPT Realtime Assistant offers a relevant published voice application reference; it does not establish that every future mode is available in every public version. For your project, the handover should identify exactly which pathway passed, what remains unsupported and how the interface communicates those boundaries without interrupting a useful task unnecessarily.

Relevant Arte Soft work

GPT Realtime Assistant — our own published product is our own product or engineering project. It provides context for the work we discuss here; it is not a claim of identical results for every client deployment.

Sources and further reading

• Apple: Translation ↗
• Apple: Foundation Models ↗