🗨️Chat Models

Chat models take a list of messages as input and return a model-generated message as output.

1)AWS ChatBedrock

Wrapper around AWS Bedrock large language models that use the Chat endpoint.

2)Azure ChatOpenAI

Prerequisite

2. Create your Azure OpenAI and wait for approval approximately 10 business days

3. Your API key will be available at Azure OpenAI > click name_azure_openai > click Click here to manage keys

Setup

1. Click Go to Azure OpenaAI Studio

2. Click Deployments

3. Click Create new deployment

4. Select as shown below and click Create

5. Successfully created Azure ChatOpenAI

· Deployment name: gpt-35-turbo

· Instance name: top right conner

1.Chat Models in Thub > drag Azure ChatOpenAI node

2. Copy & Paste each details (API Key, Instance & Deployment name, API Version) into Azure ChatOpenAI credential

3)NIBittensorChat

Wrapper around Bittensor subnet 1 large language models.

4)ChatAnthropic

Wrapper around ChatAnthropic large language models that use the Chat endpoint.

5)ChatCohere

Wrapper around Cohere Chat Endpoints.

6)ChatGoogleGenerativeAI

Prerequisite

1. Register a Google account

2. Create an API key

Chat Models > drag ChatGoogleGenerativeAI node

1) Connect Credential > click Create New

2) Fill in the Google AI credential

3) Voila 🎉, you can now use ChatGoogleGenerativeAI node in Thub

Safety Attributes Configuration

· Click Additonal Parameters

· When configuring Safety Attributes, the amount of selection in Harm Category & Harm Block Threshold should be the same amount. If not it will throw an error Harm Category & Harm Block Threshold are not the same length

· The combination of Safety Attributes below will result in Dangerous is set to Low and Above and Harassment is set to Medium and Above

7)ChatGooglePaLM

Wrapper around Google MakerSuite PaLM large language models using the Chat endpoint.

8)Google VertexAI

Prerequisites

1. Start your GCP

2. Install the Google Cloud CLI

Setup

Enable vertex AI API

· Go to Vertex AI on GCP and click "ENABLE ALL RECOMMENDED API"

Create credential file (Optional)

There are 2 ways to create credential file

No. 1: Use GCP CLI

1) Open terminal and run the following command

Gcloud_authapplication_default_login.

2) Login to your GCP account

3) Check your credential file. You can find your credential file in

No. 2: Use GCP console

1. Go to GCP console and click "CREATE CREDENTIALS"

1. Fill in the form of Service account details and click "CREATE AND CONTINUE"

2. Select proper role (for example Vertex AI User) and click "DONE"

Click service account that you created and click "ADD KEY" -> "Create new key"

6. Select JSON and click "CREATE" then you can download your credential file

Without credential file

If you are using a GCP service like Cloud Run, or if you have installed default credentials on your local machine, you do not need to set this credential.

With credential file

1. Go to Credential page on THub and click "Add credential"

2. Click Google Vertex Auth

· Option 1: Enter path of your credential file

If you have credential file on your machine, you can enter the path of your credential file into Google Application Credential File Path

· Option 2: Paste text of your credential file

Or you can copy all text in the credential file and paste it into Google Credential JSON Object

1. Finally, click "Add" button.

2. You can now use ChatGoogleVertexAI with the credential in THub now!

9)ChatHuggingFace

Wrapper around HuggingFace large language models.

10)LocalAI Setup

LocalAI is a drop-in replacement REST API that’s compatible with OpenAI API specifications for local inferencing. It allows you to run LLMs (and not only) locally or on-prem with consumer grade hardware, supporting multiple model families that are compatible with the ggml format.

• To use ChatLocalAI within THub, follow the steps below:

For example:

· Download one of the models from gpt4all.io

# Download gpt4all-j to models/

wget https://gpt4all.io/models/ggml-gpt4all-j.bin -O models/ggml-gpt4all-j

· In the /models folder, you should be able to see the downloaded model in there:

· Refer here for list of supported models.

THub Setup

· Drag and drop a new ChatLocalAI component to canvas

Fill in the fields:

· Base Path: The base url from LocalAI such as http://localhost:8080/v1

· Model Name: The model you want to use. Note that it must be inside /models folder of LocalAI directory. For instance: ggml-gpt4all-j.bin.

11)ChatMistralAI

• Register a Mistral AI account

• Create an API key

Setup

• Chat Models > drag ChatMistralAI node

1) Connect Credential > click Create New

2) Fill in the MistralAI credential

3) Provide the details of Model name and temperature

4) Voila 🎉, you can now use MistralAI node in Thub

12)ChatOllama

• Download Ollama or run it on Docker.

• For example, you can use the following command to spin up a Docker instance with llama2

docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollamadocker

exec -it ollama ollama run llama3

Setup

• Chat Models > drag ChatOllama node

• Fill in the model that is running on Ollama. For example: llama2. You can also use additional parameters.

• you can now use ChatOllama node in THub

Additional

• If you are running both THub and Ollama on docker. You'll have to change the Base URL for ChatOllama.

• For Windows and MacOS Operating Systems specify http://host.docker.internal:8000. For Linux based systems the default docker gateway should be used since host.docker.internal is not available: http://172.17.0.1:8000

13)ChatOllama Function

ChatOllama function which is also known as Function Based Chat Ollama (FBCO), is a wrapper for the langchain OllamaFunctions class that allows for easy function creation and integration using the BaseFunction template. FBCO is designed to run locally on your machine using Ollama. It is not yet recommended to use FBCO in a production environment

Fill in the fields:

• Base Path: The base url from LocalAI such as http://localhost:8080/v1

• Model Name: The model you want to use .

• Temperature Details.

14)ChatOpenAI

Prerequisite

• An OpenAI account

• Create an API key

Setup

• Chat Models > drag ChatOpenAI node

• Connect Credential > click Create New

• Fill in the ChatOpenAI credential

• you can now use ChatOpenAI node in THub

Custom base URL and headers

THub supports using custom base URL and headers for Chat OpenAI. Users can easily use integrations like OpenRouter, TogetherAI and others that support OpenAI API compatibility.

TogetherAI

• Refer to official docs from TogetherAI

• Create a new credential with TogetherAI API key

• Click Additional Parameters on ChatOpenAI node.

• Change the Base Path.

Open Router

• Refer to official docs from OpenRouter

• Create a new credential with OpenRouter API key

• Click Additional Parameters on ChatOpenAI node

• Change the Base Path and Base Options.

Custom Model

For models that are not supported on ChatOpenAI node, you can use ChatOpenAI Custom for that. This allow users to fill in model name such as mistralai/Mixtral-8x7B-Instruct-v0.1

Image Upload

• You can also allow images to be uploaded and analyzed by LLM. Under the hood, Flowise will use OpenAI Vison model to process the image.

• From the chat interface, you will now see a new image upload button