Building a Telegram Bot for News Digest on Go, PostgreSQL and Ollama

In today’s fast-paced world, staying updated with the latest news can be overwhelming. With so many sources and articles, it’s easy to miss important information. To solve this problem, we can build a Telegram bot that curates a personalized news digest for users. This bot will be built using Go (Golang), PostgreSQL for data storage, and Ollama for natural language processing (NLP) tasks like summarization. In this article, we’ll walk through the steps to create a Telegram bot that fetches news articles, processes them using Ollama, and delivers a concise news digest to users.

Photo by Fujiphilm on Unsplash

Prerequisites

Before we start, ensure you have the following installed:

1. Go: Install Go from the official website.
2. PostgreSQL: Install PostgreSQL from the official website.
3. Ollama: Set up Ollama for NLP tasks from the official website.
4. Telegram Bot Token: Create a bot on Telegram using BotFather and obtain the bot token.

Set Up the Project

Create a new directory for your project and initialize a Go module:

mkdir news-digest-bot
cd news-digest-bot
go mod init news-digest-bot

Install the required Go packages:

go get -u github.com/go-telegram-bot-api/telegram-bot-api/v5
go get -u github.com/jmoiron/sqlx
go get -u github.com/lib/pq

Project Structure

Organize your Go project as follows:

/news-digest-bot
├── cmd
│   └── main.go
├── internal
│   ├── bot
│   ├── config
│   ├── storage
│   ├── handlers
│   ├── models
│   └── services
├── configs
│   └── config.yaml
└── go.mod

• cmd/main.go: Entry point of the application.
• internal/bot: Telegram bot initialization and configuration.
• internal/config: Configuration handling.
• internal/storage: Database connection and queries.
• internal/handlers: Functions to handle bot commands and messages.
• internal/models: Data models representing database tables.
• internal/services: Business logic, including news fetching and summarization.

Database Design (PostgreSQL)

Goose is a great database migration tool for Go projects. It helps manage database schema changes in a structured and versioned way, which is essential for any serious application. First, install the Goose CLI and library:

go install github.com/pressly/goose/v3/cmd/goose@latest

export PATH=$PATH:$(go env GOPATH)/bin

This will install the goose binary to your $GOPATH/bin directory.

goose

Create a migrations directory in folder internal/storage of project to store migration files.

mkdir -p internal/storage/migrations 
goose -dir internal/storage/migrations create user_table sql

Edit the newly created file to define the behavior of your migration.

-- +goose Up
-- +goose StatementBegin
CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    chat_id BIGINT UNIQUE NOT NULL,
    preferences JSONB
);
-- +goose StatementEnd

-- +goose Down
-- +goose StatementBegin
DROP TABLE IF EXITS users;
-- +goose StatementEnd

goose -dir internal/storage/migrations create article_table sql

-- +goose Up
-- +goose StatementBegin
CREATE TABLE articles (
    id SERIAL PRIMARY KEY,
    category TEXT NOT NULL,
    title TEXT NOT NULL,
    link TEXT UNIQUE NOT NULL,
    summary TEXT NOT NULL,
    source TEXT NOT NULL,
    published_at TIMESTAMP  NOT NULL,
    created_at TIMESTAMP  NOT NULL DEFAULT NOW(),
    posted_at TIMESTAMP 
);
-- +goose StatementEnd

-- +goose Down
-- +goose StatementBegin
DROP TABLE IF EXITS articles;
-- +goose StatementEnd

Now let’s apply the migrations using the same utility. First, let’s check that the connection is created.

goose -dir internal/storage/migrations postgres "host=localhost user=postgres database=news_bot password=postgres sslmode=disable" status

We see a list of migrations and their status — both are Pending. Let’s apply them, for this instead of the status command we need to execute the up command.

goose -dir internal/storage/migrations postgres "host=localhost user=postgres database=news_bot password=postgres sslmode=disable" up

To run the migration in your CI/CD, you need to run the following commands during the deploy step:

export GOOSE_DBSTRING=<link to your production database>
goose -dir internal/storage/migrations postgres up

Create structs (models)

We can start writing code and start by defining the Go structs (models) for our news items and any other relevant data structures. Create a model.go file in folder internal/model of project.

package model

import (
 "time"
)

// Article represents a news article in the database.
type Article struct {
 ID          int64      `db:"id"`           // Unique ID of the article
 Category    string     `db:"category"`     // Category of the article (e.g., "technology", "sports")
 Title       string     `db:"title"`        // Title of the article
 Link        string     `db:"link"`         // URL of the article
 Summary     string     `db:"summary"`      // Summary of the article
 Source      string     `db:"source"`       // Source of the article (e.g., news website)
 PublishedAt time.Time  `db:"published_at"` // Publication timestamp
 CreatedAt   time.Time  `db:"created_at"`   // Timestamp when the article was created in the database
 PostedAt    *time.Time `db:"posted_at"`    // Timestamp when the article was posted to Telegram (nullable)
}

// User represents a user in the database.
type User struct {
 ID          int64                  `db:"id"`          // Unique ID of the user
 ChatID      int64                  `db:"chat_id"`     // Telegram chat ID (unique for each user)
 Preferences map[string]interface{} `db:"preferences"` // User preferences (stored as JSONB)
}

Connecting to PostgreSQL

Use the SQLX library to connect to PostgreSQL. Define your database schema to store user information and news articles. Golang SQLX is a library used to facilitate database operations in Go programming language. SQLX is built on top of the standard Go database/sql package and provides more features and convenience.

Begin by installing the sqlx library and the PostgreSQL driver:

go get github.com/jmoiron/sqlx
go get github.com/lib/pq

Create a database connection file (storage/storage.go):

package storage

import (
 "news-digest-bot/internal/model"

 "github.com/jmoiron/sqlx"
 _ "github.com/lib/pq"
)

type Storage struct {
 DB *sqlx.DB
}

func NewStorage(dataSourceName string) (*Storage, error) {
 db, err := sqlx.Connect("postgres", dataSourceName)
 if err != nil {
  return nil, err
 }
 return &Storage{DB: db}, nil
}

func (s *Storage) Close() error {
 return s.DB.Close()
}

func (s *Storage) Ping() error {
 return s.DB.Ping()
}

func (s *Storage) CreateArticle(article *model.Article) error {
 query := `INSERT INTO articles (category, title, link, summary, source, published_at, created_at, posted_at)
              VALUES (:category, :title, :link, :summary, :source, :published_at, :created_at, :posted_at)`
 _, err := s.DB.NamedExec(query, article)
 return err
}

func (s *Storage) GetArticles() ([]model.Article, error) {
 var articles []model.Article
 err := s.DB.Select(&articles, "SELECT * FROM articles")
 return articles, err
}

func (s *Storage) CreateUser(user *model.User) error {
 query := `INSERT INTO users (chat_id, preferences) VALUES (:chat_id, :preferences)`
 _, err := s.DB.NamedExec(query, user)
 return err
}

func (s *Storage) GetUserByChatID(chatID int64) (*model.User, error) {
 var user model.User
 err := s.DB.Get(&user, "SELECT * FROM users WHERE chat_id=$1", chatID)
 return &user, err
}

Fetching News Articles

Implement a service that retrieves news articles from various sources. You can use APIs from news providers or scrape websites as needed. Store the fetched articles in your PostgreSQL database. Create a api connection file (services/newsapi.go):

package newsapi

import (
 "encoding/json"
 "fmt"
 "log"
 "net/http"
 "time"
)

const newsAPIURL = "https://newsapi.org/v2/everything"

// Article represents a news article fetched from the NewsAPI.
type Article struct {
 Source      Source    `json:"source"`
 Author      string    `json:"author"`
 Title       string    `json:"title"`
 Description string    `json:"description"`
 URL         string    `json:"url"`
 URLToImage  string    `json:"urlToImage"`
 PublishedAt time.Time `json:"publishedAt"`
 Content     string    `json:"content"`
}

// Source represents the source of the news article.
type Source struct {
 ID   string `json:"id"`
 Name string `json:"name"`
}

// NewsResponse represents the response from the NewsAPI.
type NewsResponse struct {
 Status       string    `json:"status"`
 TotalResults int       `json:"totalResults"`
 Articles     []Article `json:"articles"`
}

// FetchNews fetches news articles from the NewsAPI based on the specified category.
func FetchNews(apiKey, category string) ([]Article, error) {
 req, err := http.NewRequest("GET", newsAPIURL, nil)
 if err != nil {
  return nil, err
 }

 q := req.URL.Query()
 q.Add("q", category)
 q.Add("sortBy", "publishedAt")
 q.Add("apiKey", apiKey)
 req.URL.RawQuery = q.Encode()

 //log.Print("Query: ", req.URL.RawQuery)

 client := &http.Client{}
 resp, err := client.Do(req)
 if err != nil {
  return nil, err
 }
 defer resp.Body.Close()

 if resp.StatusCode != http.StatusOK {
  return nil, fmt.Errorf("failed to fetch news: %s", resp.Status)
 }

 var newsResponse NewsResponse
 if err := json.NewDecoder(resp.Body).Decode(&newsResponse); err != nil {
  return nil, err
 }

 return newsResponse.Articles, nil
}

Summarizing Articles with Ollama

Ollama provides a REST API for running and managing language models. Start the Ollama server using the ollama serve command. In your Go application, interact with this API to summarize news articles. For example, send the article content to Ollama's API and receive a concise summary in return.

First, I’ve added a GetContent function that fetches full content from a given URL.

package content

import (
 "fmt"
 "io"

 "net/http"
 "regexp"
 "strings"

 "golang.org/x/net/html"
)

// GetContent fetches content from a URL, cleans it from HTML tags, and returns the plain text.
func GetContent(URL string) (string, error) {
 resp, err := http.Get(URL)
 if err != nil {
  return "", fmt.Errorf("failed to fetch URL: %w", err)
 }
 defer resp.Body.Close()

 if resp.StatusCode != http.StatusOK {
  return "", fmt.Errorf("HTTP error: %s", resp.Status)
 }

 body, err := io.ReadAll(resp.Body)
 if err != nil {
  return "", fmt.Errorf("failed to read response body: %w", err)
 }

 // Parse HTML
 doc, err := html.Parse(strings.NewReader(string(body)))
 if err != nil {
  return "", fmt.Errorf("failed to parse HTML: %w", err)
 }

 var text string
 var f func(*html.Node)
 f = func(n *html.Node) {
  if n.Type == html.TextNode {
   text += n.Data
  }
  if n.FirstChild != nil {
   for c := n.FirstChild; c != nil; c = c.NextSibling {
    f(c)
   }
  }
 }
 f(doc)

 // Clean up whitespace and special characters
 text = cleanText(text)

 return text, nil
}

func cleanText(text string) string {
 // Remove HTML entities
 text = strings.ReplaceAll(text, " ", " ") // Example, add more as needed
 text = strings.ReplaceAll(text, "&", "&")
 text = strings.ReplaceAll(text, "&lt;", "<")
 text = strings.ReplaceAll(text, "&gt;", ">")

 // Remove excessive whitespace (multiple spaces, tabs, newlines)
 re := regexp.MustCompile(`\s+`)
 text = re.ReplaceAllString(text, " ")

 // Trim leading/trailing whitespace
 text = strings.TrimSpace(text)

 return text
}

Next, add the Summarize function, which is a new function that encapsulates the logic for creating an Ollama request and retrieving a response. It takes the text and model name as arguments and returns a summary (or error).

package ollama

import (
 "bytes"
 "encoding/json"
 "errors"
 "io/ioutil"
 "net/http"
)

const ollamaURL = "http://localhost:11434/api/chat"

type requestPayload struct {
 Model    string    `json:"model"`
 Messages []message `json:"messages"`
 Stream   bool      `json:"stream"`
}

type message struct {
 Role    string `json:"role"`
 Content string `json:"content"`
}

type responsePayload struct {
 Message struct {
  Role    string `json:"role"`
  Content string `json:"content"`
 } `json:"message"`
}

// Summarize sends a request to the local Ollama model to summarize the given text.
func Summarize(text string) (string, error) {
 payload := requestPayload{
  Model:    "llama3",
  Messages: []message{{Role: "user", Content: "Summarize the following: " + text}},
  Stream:   false,
 }

 data, err := json.Marshal(payload)
 if err != nil {
  return "", err
 }

 resp, err := http.Post(ollamaURL, "application/json", bytes.NewBuffer(data))
 if err != nil {
  return "", err
 }
 defer resp.Body.Close()

 if resp.StatusCode != http.StatusOK {
  return "", errors.New("failed to get a valid response from Ollama API")
 }

 body, err := ioutil.ReadAll(resp.Body)
 if err != nil {
  return "", err
 }

 var response responsePayload
 if err := json.Unmarshal(body, &response); err != nil {
  return "", err
 }

 return response.Message.Content, nil
}

Handling User Interactions

Use a Telegram bot library for Go, such as github.com/go-telegram-bot-api/telegram-bot-api/v5, to manage user interactions. Set up handlers for commands like /start, /subscribe, and /news. When a user requests news, fetch the latest articles from your database, summarize them using Ollama, and send the summaries back to the user.

Simplify Database Operations in Golang with SQLX

Go + Ollama = Simple Local AI

Telegram bot in Go app

If you found this article insightful and want to explore how these technologies can benefit your specific case, don’t hesitate to seek expert advice. Whether you need consultation or hands-on solutions, taking the right approach can make all the difference. You can support the author by clapping below 👏🏻 Thanks for reading!

Oleh Dubetsky|Linkedin