Skip to content

Latest commit

 

History

History

surfsense_backend

README.md

Surf Backend

Technology Stack Overview

This application is a modern AI-powered search and knowledge management platform built with the following technology stack:

Core Framework and Environment

  • Python 3.12+: The application requires Python 3.12 or newer
  • FastAPI: Modern, fast web framework for building APIs with Python
  • Uvicorn: ASGI server implementation, running the FastAPI application
  • PostgreSQL with pgvector: Database with vector search capabilities for similarity searches
  • SQLAlchemy: SQL toolkit and ORM (Object-Relational Mapping) for database interactions
  • FastAPI Users: Authentication and user management with JWT and OAuth support

Key Features and Components

Authentication and User Management

  • JWT-based authentication
  • OAuth integration (Google)
  • User registration, login, and password reset flows

Search and Retrieval System

  • Hybrid Search: Combines vector similarity and full-text search for optimal results using Reciprocal Rank Fusion (RRF)
  • Vector Embeddings: Document and text embeddings for semantic search
  • pgvector: PostgreSQL extension for efficient vector similarity operations
  • Chonkie: Advanced document chunking and embedding library
    • Uses AutoEmbeddings for flexible embedding model selection
    • LateChunker for optimized document chunking based on embedding model's max sequence length

AI and NLP Capabilities

  • LangChain: Framework for developing AI-powered applications
    • Used for document processing, research, and response generation
    • Integration with various LLM models through LiteLLM
    • Document conversion utilities for standardized processing
  • GPT Integration: Integration with LLM models through LiteLLM
    • Multiple LLM configurations for different use cases:
      • Fast LLM: Quick responses (default: gpt-4o-mini)
      • Smart LLM: More comprehensive analysis (default: gpt-4o-mini)
      • Strategic LLM: Complex reasoning (default: gpt-4o-mini)
      • Long Context LLM: For processing large documents (default: gemini-2.0-flash-thinking)
  • Rerankers with FlashRank: Advanced result ranking for improved search relevance
    • Configurable reranking models (default: ms-marco-MiniLM-L-12-v2)
    • Supports multiple reranking backends (FlashRank, Cohere, etc.)
    • Improves search result quality by reordering based on semantic relevance
  • GPT-Researcher: Advanced research capabilities
    • Multiple research modes (GENERAL, DEEP, DEEPER)
    • Customizable report formats with proper citations
    • Streaming research results for real-time updates

External Integrations

  • Slack Connector: Integration with Slack for data retrieval and notifications
  • Notion Connector: Integration with Notion for document retrieval
  • Search APIs: Integration with Tavily and Serper API for web search
  • Firecrawl: Web crawling and data extraction capabilities

Data Processing

  • Unstructured: Tools for processing unstructured data
  • Markdownify: Converting HTML to Markdown
  • Playwright: Web automation and scraping capabilities

Main Modules

  • Search Spaces: Isolated search environments for different contexts or projects
  • Documents: Storage and retrieval of various document types
  • Chunks: Document fragments for more precise retrieval
  • Chats: Conversation management with different depth levels (GENERAL, DEEP)
  • Podcasts: Audio content management with generation capabilities
  • Search Source Connectors: Integration with various data sources

Development Tools

  • Poetry: Python dependency management (indicated by pyproject.toml)
  • CORS support: Cross-Origin Resource Sharing enabled for API access
  • Environment Variables: Configuration through .env files

Database Schema

The application uses a relational database with the following main entities:

  • Users: Authentication and user management
  • SearchSpaces: Isolated search environments owned by users
  • Documents: Various document types with content and embeddings
  • Chunks: Smaller pieces of documents for granular retrieval
  • Chats: Conversation tracking with different depth levels
  • Podcasts: Audio content with generation capabilities
  • SearchSourceConnectors: External data source integrations

API Endpoints

The API is structured with the following main route groups:

  • /auth/*: Authentication endpoints (JWT, OAuth)
  • /users/*: User management
  • /api/v1/search-spaces/*: Search space management
  • /api/v1/documents/*: Document management
  • /api/v1/podcasts/*: Podcast functionality
  • /api/v1/chats/*: Chat and conversation endpoints
  • /api/v1/search-source-connectors/*: External data source management

Deployment

The application is configured to run with Uvicorn and can be deployed with:

python main.py

This will start the server on all interfaces (0.0.0.0) with info-level logging.

Requirements

See pyproject.toml for detailed dependency information. Key dependencies include:

  • asyncpg: Asynchronous PostgreSQL client
  • chonkie: Document chunking and embedding library
  • fastapi and related packages
  • fastapi-users: Authentication and user management
  • firecrawl-py: Web crawling capabilities
  • langchain components for AI workflows
  • litellm: LLM model integration
  • pgvector: Vector similarity search in PostgreSQL
  • rerankers with FlashRank: Advanced result ranking
  • Various AI and NLP libraries
  • Integration clients for Slack, Notion, etc.