AI

OneStarter sets up a comprehensive database with caching and tracking mechanics for LLM-based AI requests. This is completely agnostic to what LLM models you choose to use. By default, we set you up with Cloudflare's AI in the backend.

Our architecture allows you to define prompts in the database which you can reference by label keys.

Regardless of which provider you use, the key API from the database is as follows:

• api.llm_call_lookup(_prompt_label text, _prompt_input text, _model_override_key text default null): searches for a cached LLM call
• api.insert_llm_call_log(_prompt_id bigint, _status text, _input text, _output text, _model_params jsonb: inserts a new LLM call log

Importantly, this api can only be accessed using the service_role in the backend. This means you cannot access it in the frontend directly.

API requests can be made to the /api/v1/ai/:tenantId/text/:promptKey. These will, by default attempt to check your LLM cache first, before creating and saving a cached request.

Model and Prompt Management

Prompt and model definitions/configurations can be managed directly in the database. For example, here are the demo prompts used to start:

insert into private.llm_models (key, model_params)
values ('@cf/meta/llama-3.1-8b-instruct-fast',
        '{
          "stream": false,
          "temperature": 0.5,
          "top_p": 0.8
        }')
on conflict (key) do update set model_params = excluded.model_params;

insert into private.llm_prompts (label, prompt, default_model_key)
values ('test',
        E'<INPUT>%s</INPUT>\n\n{INPUT} can be found the beginning of the prompt and is of the form { input: string; locale: string;}. Ensure your answer matches the language of locale, which is four characters separated by a hyphen, e.g. ar-AR = Arabic, zh-CN = Mandarin, it-IT = Italian, etc. Please answer what is the capital of {INPUT.input}. \n\nENSURE YOU USE THE PROVIDED INPUT.locale AS THE LANGUAGE OF YOUR OUTPUT ALWAYS. ENSURE YOUR OUTPUT IS IN THE FORM OF "The capital of {COUNTRY} is {ANSWER}.". For example, "The capital of Canada is Ottawa." for en-GB, or "La capitale du Canada est Ottawa." for fr-FR. ONLY RETURN THE FORMATTED OUTPUT, NO PREAMBLE.',
        '@cf/meta/llama-3.1-8b-instruct-fast')
on conflict (label) do update
    set prompt            = excluded.prompt,
        default_model_key = excluded.default_model_key;