The short version

No technical background needed for this part.

  • Two jobs, two tools. A product like TelegramInformer does two very different things: it quietly collects data all day, and it shows that data to people on a website. I used a different tool for each, instead of forcing one to do both.
  • The collector is the heart of it. It is written in Go, a language built to do thousands of things at once without needing a big machine. One small server tracks more than 23,000 channels and finishes about 55,000 tasks a day, while sitting mostly idle.
  • The website was quick to build. Laravel and Filament come with most of what a website and an admin panel need, ready to use. The whole admin, with its charts and editing screens, took days rather than weeks.
  • One database, kept safe. Both sides meet in a single database, which also keeps the to-do list of tasks, so there is one less system to pay for and look after. The website can only read the data, so a bug there can't damage it.
  • Private by design. The two servers talk through a private, encrypted tunnel. The database is never open to the internet.
  • What it means for the business: low running costs, a small team can operate it, and there is room to grow to about 100,000 channels without a bigger server (my estimate, explained below).

If your product collects information from other places and turns it into something people use, this setup is worth a look. The rest of the article explains how it works and why.

The longer version

A data aggregation product looks like one system from the outside. Inside, it is two systems that want opposite things. One collects data all day: thousands of slow network calls, retries, rate limits, and a database that never stops growing. The other serves pages to people and search engines, and gives the team an admin to run the catalog.

I built TelegramInformer, an analytics site for public Telegram channels, around that split. A Go service owns the data. A Laravel site, with Filament for the admin, serves the web. PostgreSQL sits between them and is the only thing they share. This article explains why each piece is there, what the setup handles on modest hardware, and when I would choose it again.

Two workloads, one product

Before choosing any technology, it helps to describe the two jobs honestly.

Collecting the data Serving the website
Runs Around the clock, on a schedule When a visitor or crawler asks
Spends its time Waiting on other people's servers Rendering pages from data we already have
Main constraint Outside rate limits, failures, retries Response time, SEO, editorial workflow
Needs Cheap concurrency, careful pacing, idempotent writes Templates, routing, forms, auth, caching, two languages
Who uses it Nobody directly Visitors, search engines, the admin team

Trying to make one stack excellent at both columns is where projects get expensive. So I split by workload: each side uses the tool that makes its own column easy, and the database is the contract between them.

Why Go owns the data side

The data service is the core of the product. Everything the website shows is only as good, as fresh and as complete as what the collector brings in. That is why it gets the language best suited to it.

Concurrency that costs almost nothing. Collection is mostly waiting: a request goes out, and the answer comes back hundreds of milliseconds later. Go's goroutines are built for exactly this. In the words of the Go FAQ, a new goroutine "is given a few kilobytes, which is almost always enough", and "it is practical to create hundreds of thousands of goroutines in the same address space." Thousands of channels in flight do not need a big server.

One binary, no runtime to babysit. Each service builds into a single static binary. Deploying means copying it to a new release folder, pointing a current symlink at it, and restarting a systemd unit. Rolling back means pointing the symlink at the previous release. There is no Docker on the data server and no interpreter version to keep in sync.

Libraries that fit the job. pgx for PostgreSQL, River for the job queue, goquery for parsing public web pages, and gotd/td, a pure-Go client for Telegram's own protocol that comes with rate-limit and flood-wait middleware. Pacing uses Go's standard rate package.

The collector is a set of small, single-purpose jobs. Each one does one thing and can fail and retry on its own:

Job What it does How often
Schedule Finds what is due and queues the work Every minute
Profile Reads a channel's title, description and subscriber count Daily (more often for large channels)
Posts Reads new posts and samples the views of recent ones Every 8 hours to weekly, depending on how active the channel is
Discovery Turns reposts and mentions into new channels to check Every 6 hours
Language Detects each channel's language from its posts Hourly, in batches
Metrics Recomputes growth, reach, engagement and rankings Every 12 hours

That last point about activity matters more than it looks. A channel that posts ten times a day is checked every 8 hours. A channel that has gone quiet is checked weekly. The tiers are recalculated from the last 30 days of posts, so the collector spends its request budget where the news actually is.

The whole data service, including the admin command-line tool and a small internal API, is about 7,200 lines of Go plus 1,800 lines of tests.

PostgreSQL is the backbone, not just storage

In this design PostgreSQL does three jobs: it stores the data, it runs the job queue, and it defines what each side is allowed to touch.

The job queue lives in the database

The collector does not use Redis or a separate message broker. Jobs are rows in PostgreSQL, managed by River. The main benefit is what River's documentation calls transactional enqueueing: a job can be inserted in the same transaction as the data it depends on, so the two commit or fail together. There is no window where the data exists but the job was lost, or the job runs before its data is visible.

Underneath, queues like this rely on PostgreSQL's SKIP LOCKED. The PostgreSQL documentation says it "can be used to avoid lock contention with multiple consumers accessing a queue-like table": each worker takes the next free job without waiting for the others. Brandur Leach, one of River's authors, explains the design and its history in the River launch post.

Uniqueness is declared on the job itself. While a channel's job is waiting or running, the same job can't be queued twice:

var uniqueWhilePending = river.UniqueOpts{
    ByArgs: true,
    ByState: []rivertype.JobState{
        rivertype.JobStateAvailable, rivertype.JobStatePending,
        rivertype.JobStateRunning, rivertype.JobStateRetryable,
        rivertype.JobStateScheduled,
    },
}

type PostsArgs struct {
    ChannelID int64  `json:"channel_id" river:"unique"`
    Username  string `json:"username"`
}

func (PostsArgs) Kind() string { return "posts_web" }

func (PostsArgs) InsertOpts() river.InsertOpts {
    return river.InsertOpts{Queue: "web", MaxAttempts: 4, UniqueOpts: uniqueWhilePending}
}

One less service to run, monitor and back up is a real saving for a small team.

Time series without a second database

Subscriber counts, post views and posting activity are time series. TimescaleDB adds three things on top of plain PostgreSQL that matter here:

  • Hypertables split each large table into time-based chunks automatically.
  • Compression shrinks older chunks. On real data, posts compress 2.2 times and subscriber history 1.6 times. Post text is the bulk of it and doesn't compress as well as numbers do.
  • Continuous aggregates keep daily series (subscribers per day, views per day) up to date, so the website reads ready-made numbers instead of scanning raw samples.

Retention is a policy, not a script: raw view samples are dropped after 180 days, while the daily aggregates stay.

Database roles are the contract

The website never writes collected data. It connects with a read-only role. The admin connects with a second role that can read everything but write only to editorial fields: a channel's category, country, language, status, and its own description. The collector owns the schema and the migrations. If a bug ever slipped into the website, the database itself would refuse to let it damage the data.

In Laravel this is just two connections in config/database.php, and the admin models opt into the second one:

'admin' => [
    'driver' => 'pgsql',
    'host' => env('DB_ADMIN_HOST', env('DB_HOST', '127.0.0.1')),
    'database' => env('DB_ADMIN_DATABASE', env('DB_DATABASE')),
    'username' => env('DB_ADMIN_USERNAME', 'tginfo_admin'),
    'password' => env('DB_ADMIN_PASSWORD', ''),
    // ...
],
class Category extends Model
{
    protected $connection = 'admin';
}

What one data server handles

This is the part people usually ask about. These are the production numbers for one week in November 2026.

The data server is a single virtual machine with 2 vCPUs and 4 GB of RAM.

Measure Value
Active channels tracked 23,363 (7,614 discovered in the last week)
Posts stored 1.35 million, about 170,000 new per day
View samples 2.46 million, about 456,000 new per day
Subscriber snapshots 592,000, about 29,000 per day
Jobs completed about 55,000 per day
Jobs that failed for good 79 out of about 274,000 in seven days (under 0.03%)
CPU 16% on average, 35% at the 95th percentile
Memory 46% used on average
Database 2.9 GB, disk 23% used

The admin dashboard: data server load, database size, jobs in the last hour and service status

The admin dashboard reads one sample of the data server every five minutes: CPU, memory, disk, database size, jobs and the state of each service.

Server load and jobs completed over 24 hours, with a breakdown by job type

24 hours of load and finished jobs. The CPU line stays near 10% while the collector completes around 600 jobs every 15 minutes.

The real limit isn't the server

The most useful thing these numbers show is where the ceiling is. It is not CPU or memory. It is how many requests the outside world will accept.

Every source has rate limits. For public web pages, one outgoing IP address is good for roughly 30,000 requests a day at a polite pace. The collector is designed around that: each outgoing address has its own rate limiter, a retry always goes out through a different address, and the scheduler only queues as much work as the budget allows.

When I sized the system for 100,000 channels, the budget came to about 160,000 to 170,000 requests a day: daily profiles for every channel, posts by activity tier, plus discovery and re-checks. That averages about two requests per second. For a Go service that spends most of its time waiting, two requests per second is nothing. Growing from 23,000 to 100,000 channels means adding outgoing capacity and letting the scheduler use it, not buying a bigger server.

To be precise about what is measured and what is estimated: the table above is measured. The 100,000 figure is the design target and my estimate, based on the request budget and the current headroom. The architecture doesn't need to change to get there.

Why not Go for the website too

Go would make a fast website. It would also make a slow project.

A content site needs templates, layouts, routing, form validation, CSRF protection, authentication, caching, localisation in two languages, sitemaps, and an admin. Laravel ships all of it, documented and maintained together. In Go, each of those is a library to pick and glue together, or code to write. For a product where the data is the hard part, that time is better spent on the collector.

Speed isn't a reason to move either. Public pages are served from nginx's page cache in about 2 milliseconds, and the data behind them changes a few times a day, so caching is safe. The website runs on a separate server with 1 vCPU and 2 GB of RAM.

I'm not the only one who lands here. Mohamed Said, a PHP developer for more than a decade, argues for the same split in The polyglot stack: keep most of the application in PHP for development speed, and move the parts that are expensive to run into Go. Geocodio describes a similar arrangement in production in Supercharging Laravel with Go: a Laravel application that hands performance-critical work to Go services, where operations that took hundreds of milliseconds in PHP finish in under 50.

My rule for when to revisit this: if TelegramInformer ever sells a public data API, that API should be a Go service reading the same PostgreSQL database. It could be added without touching the website.

Filament: a working admin in days

The admin is where Laravel pays off most. Filament describes itself as "an open-source framework for building polished, maintainable business software with Laravel." In practice, you describe a table and a form in PHP, and you get a complete, consistent admin screen with search, filters, sorting and validation.

The TelegramInformer admin has six resources (channels, categories, submissions, reports, comments, contact messages) and six dashboard widgets. All of it, including the server and job charts above, is about 1,260 lines of PHP.

The Channels list in the admin, with filters, categories, country, language and status

The channel catalog: 23,000+ rows with search, filters and sortable columns, straight from a Filament table definition.

Editing a channel: the collected data on the left, the editorial fields on the right

Collected data is read-only on the left. The editorial fields on the right are the only things the admin's database role can change. "Decided by" shows whether a category came from a person or from the classification model, and how confident it was.

The categories list with English and Russian names and channel counts

Categories in both site languages, with live channel counts.

The dashboard also works as the team's to-do list: new submissions, reports and comments sit at the top, and the catalog section shows what still needs a human decision.

The catalog section of the dashboard: active channels, missing categories and countries, candidates and freshness

What needs attention: channels without a category or country, new candidates waiting for their first check, and how fresh the metrics and posts are.

Two servers, one private network

The data server and the web server live in different DigitalOcean teams, so they can't share a private network. Opening PostgreSQL to the internet was never an option. The connection between them is a WireGuard tunnel.

WireGuard was presented by Jason Donenfeld at NDSS 2017 as a small, modern tunnel built into the Linux kernel. For this setup, what matters is that it is simple to configure and nearly invisible in practice: a query from the web server to the database adds about 1.5 milliseconds.

Three layers keep it tight:

  1. The firewall on the data server accepts WireGuard traffic from the web server's address only.
  2. PostgreSQL listens on the tunnel address, not the public one.
  3. pg_hba.conf allows only the web and admin roles, only from the web server's tunnel address, and only over TLS, so the database connection is encrypted even inside the tunnel.
# /etc/wireguard/wg0.conf on the data server (keys omitted)
[Interface]
Address = 10.200.0.1/24
ListenPort = 51820

[Peer]
# the web server
AllowedIPs = 10.200.0.2/32
PersistentKeepalive = 25
# pg_hba.conf: web roles, through the tunnel, over TLS only
hostssl  tginfo  tginfo_web,tginfo_admin  10.200.0.2/32  scram-sha-256

Keeping it boring to operate

A data product earns trust by being quietly reliable. A few habits make that cheap:

  • Health checks with email alerts. Every 10 minutes a timer checks that the collector is running and finishing jobs, that metrics are fresh, that failures stay low, that the last backup succeeded and that the disk isn't filling up. It emails when something changes, not every 10 minutes.
  • Encrypted off-site backups with a tested restore. A nightly database dump goes into an encrypted repository off the server. A restore was tested end to end: every table in the restored copy matched production row for row.
  • One-step deploys and rollbacks on both servers, as described above.

When this combination fits

I'd choose Go, Laravel with Filament, and PostgreSQL again when:

  • the product depends on collecting data continuously from outside sources, with rate limits and failures as normal events;
  • there is a real website in front of it: SEO pages, rankings, search, more than one language;
  • the team needs an admin to curate, moderate and fix data by hand;
  • the team is small, so one database and two servers beat a fleet of services.

I wouldn't use it for:

  • a simple web application with no heavy background work. Laravel alone, with its own queues, is enough;
  • a real-time API product where the API is the product. Then Go on both sides makes sense;
  • a team with no Go experience and no time to learn it. Laravel's queues will carry a smaller collector a long way.

The split itself is the idea worth keeping: put each workload on the tool that makes it easy, and let the database be the contract between them.