How to Use Ollama for Local LLMs

Ollama is a platform designed for running large language models (LLMs) locally on your computer, offering a simple way to interact with advanced AI models without relying on cloud services. It emphasizes privacy, control, and local inference while supporting a variety of LLMs.

Photo by Simon Wiedensohler on Unsplash

Key Features of Ollama

Local Model Hosting: Models run directly on your device, ensuring that sensitive data stays private and is not sent to external servers.

Ease of Use: Offers a streamlined process for downloading, managing, and using LLMs locally. Models can be accessed via a command-line interface (CLI) or programmatically through HTTP requests.

Model Compatibility: Supports popular open-source models such as LLaMA 3, GPT-3.5 variants, and other fine-tuned versions. Provides a curated list of models optimized for local use.

Privacy and Security: Since all computations happen locally, it eliminates the risk of exposing sensitive data to external servers or third-party APIs.

Performance: Optimized to run efficiently on modern hardware (e.g., laptops with Apple Silicon or powerful GPUs). Some models are designed to be lightweight for use on consumer-grade devices.

Integration Options: Exposes an HTTP API for seamless integration with applications, scripts, or tools like Jupyter Notebook, making it versatile for developers.

Typical Use Cases

• Data Privacy: Organizations or individuals needing AI capabilities without exposing sensitive data.
• Offline Access: Using LLMs in environments without internet connectivity.
• Custom Applications: Integrating LLMs into software, tools, or workflows via its API.
• Educational and Experimental: Running AI experiments or training models locally.

Getting Started with Ollama

Download and install Ollama from its official website.

Use the CLI to manage models:

• Download a model: ollama pull <model_name>.
• Start the server: ollama run <model_name>.

In Docker

docker run -d  -v /home/master/ollamamodels:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

docker exec -it ollama ollama run llama2

Query the model via CLI or integrate it into your applications using its API.

By offering local inference capabilities, Ollama caters to users who prioritize data privacy, cost control, and the flexibility to run AI models on their terms.

Test the API Directly

You can test the API manually using a tool like curl to confirm its behavior:

curl -X POST http://localhost:11434/api/chat -H "Content-Type: application/json" -d '{
  "model": "llama2",
  "messages": [{"role": "user", "content": "Explain machine learning."}],
  "stream": false
}'

Run in Jupyter Notebook

Ensure you have the necessary Python libraries installed to interface with Ollama. Common ones include:

pip install ollama notebook pandas

Open Jupyter Notebook and run

import ollama
MODEL = "llama2"
messages = [
    {"role": "user", "content": "Describe some of the business applications of Generative AI"}
]
response = ollama.chat(model=MODEL, messages=messages)
print(response['message']['content'])

Chat with CSV Files

Chatting with CSV files using Ollama involves combining the capabilities of the Ollama language model with a method to parse, understand, and query the CSV data.

Approach Overview

1. Load the CSV file into a DataFrame using Pandas.
2. Preprocess the data for understanding and querying.
3. Formulate user queries and map them to CSV operations.
4. Use Ollama to generate natural language responses based on the data.

Load the Uploaded CSV

First, load the file into a Pandas DataFrame and preview the contents.

import ollama
import pandas as pd

# Load your CSV data
csv_file_path = "items.csv"
df = pd.read_csv(csv_file_path)
# Convert the first few rows of the CSV to text (or format it as needed)
csv_text = df.head().to_string()
MODEL = "llama2"
# Define the messages to send to the model
messages = [
    {"role": "user", "content": f"Here is the data from my CSV file:\n{csv_text}\nCan you analyze this?"}
]
# Send the request to the local Ollama model
response = ollama.chat(model=MODEL, messages=messages)
# Print the response from the model
print(response['message']['content'])

Run Queries

Here are some example queries you can try:

“What are the columns in the CSV file?” This will list all column names in your CSV.

“Give me a summary of the data.” This will provide a statistical summary or overview of the data.

“Filter rows where ‘Category’ is ‘Agriculture’.” This will filter rows based on the column and value specified.

OpenAI python library to connect to Ollama

There’s actually an alternative approach that some people might prefer:

#!pip install openai

from openai import OpenAI
ollama_via_openai = OpenAI(base_url='http://localhost:11434/v1', api_key='ollama')

response = ollama_via_openai.chat.completions.create(
    model=MODEL,
    messages=messages
)

print(response.choices[0].message.content)

AI — LLM, AI, DL, NLP.

Jupyter Ai Ollama for Developers | Restackio

If you liked the article, you can support the author by clapping below 👏🏻 Thanks for reading!

Oleh Dubetsky|Linkedin