OPQ — "Oh, Please Queue"

A lightweight, distributed message queue broker written in Go with zero external dependencies.

Features a simple TCP protocol with human-readable telnet commands, role-based authentication with queue permissions, optional file-based persistence, peer-to-peer replication for high availability, and HTTP monitoring endpoints. Designed as a development/testing alternative to Kafka or RabbitMQ for small to medium applications.

Architecture Diagram

graph TB subgraph "Client Layer" PUB[Publisher Client] SUB[Subscriber Client] HTTP[HTTP Client] end subgraph "OPQ Broker Core" AUTH[Authentication Module] QM[Queue Manager] PM[Persistence Manager] RM[Replication Manager] HTTPAPI[HTTP API Server] end subgraph "Storage Layer" MEM[In-Memory Queues] FS[File System Storage] end subgraph "Network Layer" TCP[TCP Server] PEER[Peer Connections] end subgraph "Configuration" CONFIG[Config File] USERS[User Management] end PUB --> TCP SUB --> TCP HTTP --> HTTPAPI TCP --> AUTH AUTH --> QM QM --> MEM QM --> PM QM --> RM MEM --> FS PM --> FS RM --> PEER CONFIG --> AUTH USERS --> AUTH CONFIG --> QM CONFIG --> PM CONFIG --> RM HTTPAPI --> QM

Protocol & Commands

Human-readable TCP protocol — debug with telnet, nc, or echo | nc:

AUTH bob password123
PUB myqueue "Hello, world!"
GET myqueue
ACK myqueue

Command	Description	Required Role
`AUTH <user> <pass>`	Authenticate connection	Any
`PUB <queue> <msg>`	Publish message	Publisher
`SUB <queue>`	Subscribe to queue	Subscriber
`GET <queue>`	Get next message	Subscriber
`ACK <queue>`	Acknowledge message	Subscriber

Features

Simple TCP Protocol — Human-readable commands you can type with telnet
Role-Based Security — Publishers and subscribers with queue permissions
Optional Persistence — File-based storage for durability
Peer Replication — Basic clustering for high availability
HTTP Monitoring — REST endpoints for status and queue info
Zero Dependencies — Just Go standard library

Challenges Solved

Race Conditions: sync.RWMutex for thread-safe queue operations across goroutines
Message Loss: Optional file persistence with atomic writes preventing crash data loss
Security: Role-based authentication with per-queue permissions
Single Point of Failure: Peer-to-peer replication for broker failover
Monitoring: HTTP endpoints for real-time queue status and system health
Message Acknowledgment: Pending message tracking with ACK mechanism

Configuration

{
  "broker_id": "broker1",
  "port": "8080",
  "http_port": "8081",
  "peers": ["localhost:8082"],
  "savequeuepath": "./queues",
  "enable_persistence": true,
  "users": [
    {
      "username": "publisher",
      "password": "pub123",
      "role": "publisher",
      "allowed_queues": ["orders", "notifications"]
    },
    {
      "username": "subscriber",
      "password": "sub123",
      "role": "subscriber",
      "allowed_queues": ["read_all"]
    }
  ]
}

Performance

Throughput: ~10,000 messages/second (single broker)
Memory: Minimal — just the messages in queues
Disk: Optional, linear growth with message count
Network: TCP with simple text protocol

← Back to Projects