Lab Notebook

"From raw Pb to pure Au" — Developer's Notes

📓 Notebook #1 🧪 Breaking Leads Research Division
Pb → Hg → Au ✓
TODO: More gold!
⚗️ + 🔥 = 💰
82 → 79
E = mc² ... wait
C₆H₁₂O₆ + 6O₂
ΔG = ΔH - TΔS
PV = nRT
IMPORTANT!
Au₂O₃ → 2Au + 3/2O₂

What’s New

January 20, 2026 - Changelog Lab Notebook Redesign

📓 Card-Style Lab Notebook Pages

  • Each entry is now a distinct “page” - Changelog entries display as individual cards with clear visual separation
  • Gold accent bars - Chemistry-themed gold gradient at the top of each card
  • Paper texture backgrounds - Warm off-white in light mode, dark parchment in dark mode
  • Rounded corners and shadows - Subtle depth makes each entry feel like a physical notebook page

🧪 Technical Details

  • JavaScript DOM wrapping - Each H2 date section is dynamically wrapped in a .changelog-card div on page load
  • Why not pure CSS? - CSS sibling selectors (~) can’t create bounded card regions since Markdown renders flat HTML without wrapper divs
  • Graph paper grid preserved - The background grid pattern shows through between cards

January 19, 2026 - Sales-First Lead Details Redesign

🎯 Redesigned Lead Details for Sales Workflow

  • Contact info at the top - Name, phone, and email are now immediately visible when you open a lead
  • Bold CALL/TEXT/EMAIL buttons - One-click actions to contact the customer right away
  • Notes section moved up - Take notes during calls without scrolling down the page
  • Traffic-light status colors - Green for new, blue for contacted, yellow for interested, red for dead

📂 Collapsible Technical Sections

  • Permit & Project Details - Collapsed by default, click to expand
  • Property Details (Rentcast) - Enrichment data tucked away until needed
  • Septic Permits (Idaho) - Historical permit data in accordion
  • Less scrolling - Focus on what matters for your sales call

🎨 Improved Dark Mode Contrast

  • Status dropdown now readable - Fixed contrast issues with white text on colored backgrounds
  • Custom dropdown styling - Consistent appearance across browsers
  • All status colors pop - Green, blue, yellow, orange, cyan, and red all clearly visible

🎯 New “Converted” Status

  • For leads assigned to jobs - Cyan badge indicates the lead has been converted to an active job
  • Complete status lifecycle - New → Contacted → Interested → Qualified → Converted → Closed

January 18, 2026 - Public Landing Page

🏠 New Marketing Landing Page

  • Public homepage - New landing page at / for unauthenticated visitors
  • Leads dashboard moved - Property leads list now lives at /leads
  • Auto-redirect - Logged-in users automatically redirect to /leads
  • Feature highlights - Landing showcases Lead Tracking, Property Enrichment, and Team Collaboration

🔗 Navigation Updates

  • All internal navigation links updated to point to /leads
  • Auth callback now redirects to /leads after login
  • Logout returns users to the public landing page

January 18, 2026 - AI CSV Parsing Fix

🔧 Major Fix: AI CSV Import Now Extracts All Rows

  • 9.7x improvement - AI CSV parsing now extracts ~97% of rows (up from ~10%)
  • Root cause fixed - OpenAI’s json_object format was forcing single-object output, preventing the LLM from returning multiple leads per batch
  • Batch processing restored - Each 10-row batch now correctly extracts all valid leads instead of just one

🤖 Technical Details

  • Problem: The response_format: {"type": "json_object"} constraint forces LLM output to start with {, which prevented returning JSON arrays [...]
  • Solution: Changed prompts to request wrapped format {"leads": [{...}, {...}, ...]} and updated parsing to extract the inner array
  • Model: Using qwen2.5 via LiteLLM proxy (no “thinking mode” overhead)

📊 Test Results

BatchExtractedErrorsSuccess Rate
Batch 19/10190%
Batch 210/100100%
Batch 310/100100%
Batch 48/80100%

January 17, 2026 - Analytics Dashboard & Enrichment Queue

📊 Analytics Dashboard (NEW)

  • KPI cards - Track total jobs, active jobs, revenue, and conversion rate at a glance
  • Trend charts - Visualize jobs and revenue over time with interactive Chart.js graphs
  • Contractor performance - See which subcontractors are completing the most jobs
  • Status distribution - Pie chart showing job status breakdown
  • Conversion funnel - Track leads through New → Contacted → Qualified → Converted stages
  • Geographic insights - Bar chart of jobs by county to identify hot markets
  • Time period filter - Switch between 30, 60, 90 days, or 6 months of data
  • Dark mode support - Charts adapt colors for light and dark themes

🔄 Enrichment Queue Status UI (NEW)

  • Real-time queue monitoring - See pending and processing enrichment jobs right on the lead detail page
  • Status badges - Color-coded indicators show pending (yellow), processing (blue/pulsing), completed (green), and failed (red) states
  • Retry tracking - View attempt counts and error messages for failed enrichments
  • Cancel jobs - Abort pending enrichment requests with one click
  • Auto-polling - Queue status automatically refreshes every 5 seconds while jobs are processing
  • Smart visibility - Section only appears when there are queue items for the lead

🤖 AI-Powered CSV Parsing (NEW)

  • Intelligent field mapping - Upload any CSV format and let AI figure out which columns map to lead fields
  • No header requirements - Works with any column naming convention (e.g., “applicationNumber” → “project_number”)
  • Smart inference - AI infers work types from descriptions (e.g., “OSSF Alteration” → “repair”)
  • Toggle on upload - Enable “Use AI Parsing” checkbox when uploading CSV files
  • Handles messy data - Cleans and normalizes inconsistent data automatically

📡 Real-Time CSV Upload Progress (NEW)

  • Live progress updates - See row-by-row extraction results as they happen via WebSocket
  • Batch progress - Watch as each batch of rows is processed with running totals
  • Cancel mid-upload - Abort large uploads without losing already-imported leads
  • Error visibility - See exactly which rows failed and why, in real-time
  • No waiting - Start reviewing imported leads before the upload finishes

🔧 Enrichment Worker Improvements

  • Health endpoint - Worker exposes /health for Docker healthchecks and monitoring
  • Queue metrics API - New endpoints for queue statistics and management
  • Retry with backoff - Failed enrichments automatically retry with exponential backoff

📡 New API Endpoints

  • GET /api/v1/usage/queue/lead/{lead_id} - Get queue status for a specific lead
  • POST /api/v1/usage/queue/{id}/retry - Retry a failed queue item
  • POST /api/v1/usage/queue/{id}/cancel - Cancel a pending queue item
  • GET /api/v1/usage/queue/stats - Overall queue statistics

January 14, 2026 - Semantic Search with AI Embeddings & UI

🎨 Semantic Search Interface (NEW)

  • Smart search bar - Toggle between keyword and AI semantic search modes with one click
  • Visual mode indicator - See at a glance whether you’re using traditional or AI search
  • Similarity scores - Each result shows a percentage match (80%+ = excellent, 60-80% = good)
  • Find Similar button - Click the search icon on any lead to find related properties instantly
  • Color-coded results - Green badges for high matches, yellow for medium, gray for low
  • Search by meaning, not keywords - Find leads using natural language like “septic repair flooding basement Travis County”
  • AI-powered matching - Uses mxbai-embed-large embeddings to understand what you’re looking for
  • Find similar leads - Discover related properties based on project type, location, and description
  • Smart ranking - Results sorted by semantic similarity, not just text matching

🧠 How It Works

  • Every lead is automatically converted into a vector embedding capturing its meaning
  • Search queries are converted to embeddings and compared against all leads
  • Higher similarity scores mean closer semantic matches
  • Works even when exact words don’t match (e.g., “sewage backup” finds “septic overflow”)
  • Toggle back to keyword search anytime for exact phrase matching

📡 Search API

  • POST /api/v1/search/leads - Semantic search with filters for status, county, work type
  • GET /api/v1/search/similar/{lead_id} - Find leads similar to a specific lead
  • GET /api/v1/search/status - Check embedding generation progress

🔧 Infrastructure

  • TimescaleDB - Upgraded to timescale/timescaledb-ha for AI-native PostgreSQL
  • pgai Extension - Vector storage with HNSW indexing for fast similarity search
  • Automatic embeddings - Background worker generates embeddings for new leads

January 14, 2026 - Relationship Tracking Dashboards

📊 Buyer Management Dashboard (/buyers)

  • Purchase history - View all lead sales with payment status tracking
  • Buyer performance - Total revenue, lead counts, and feedback stats per buyer
  • Payment filtering - Filter by pending, paid, disputed, or waived status
  • Feedback tracking - Monitor buyer feedback (good_lead, bad_lead, converted)

🤝 Referral Network Visualization (/referrals)

  • Business-to-business relationships - Create and manage referral partnerships
  • Commission tracking - Set percentage or flat-fee commission structures
  • Conversion analytics - Track referral counts and conversion rates
  • Relationship status - Active, inactive, and pending relationship management

💰 Revenue & Commission Tracking (/revenue)

  • Revenue overview - Total income from lead sales across all buyers
  • Commission analysis - Payables and receivables by referral partner
  • Payment aging - Track overdue payments and collection status
  • Breakdown views - Revenue segmented by buyer, time period, and referrer

📡 New API Endpoints

  • POST /api/v1/purchases/ - Record lead sales to buyers
  • GET /api/v1/purchases/stats - Purchase revenue statistics
  • POST /api/v1/referrals/ - Create referral relationships
  • GET /api/v1/referrals/stats - Network and commission statistics

January 13, 2026 - Lead Transfer System with Billing

🔄 Transfer Leads Between Users

  • User-to-user transfers - Move lead ownership to any team member with one click
  • Transfer fees - Set a fee when transferring to track billing between users
  • Full history - Complete audit trail shows every transfer with timestamps
  • Ownership display - Lead detail page shows current owner and transfer count

💰 Billing & Payment Tracking

  • Payment status - Track whether transfer fees are pending, paid, disputed, or waived
  • Billing statistics - See who owes you money and who you owe
  • Detailed breakdown - View all transfers by direction (sent/received)
  • Mark as paid - Update payment status when fees are settled

📋 New Transfer API

  • POST /api/leads/{id}/transfer - Transfer a lead to another user
  • GET /api/leads/{id}/transfers - View transfer history for a lead
  • GET /api/transfers/ - List your transfers (sent or received)
  • PATCH /api/transfers/{id}/payment - Update payment status
  • GET /api/transfers/billing/stats - Your billing summary

🎨 Frontend Updates

  • Transfer button - Opens modal with recipient dropdown and fee input
  • Ownership section - Shows current owner on lead detail page
  • Transfer history - View all previous transfers for a lead

December 7, 2025 - HSP Integration for Qualified Leads

🏢 Export Qualified Leads to HSP CRM

  • Dedicated API endpoint - /api/v1/hsp/qualified-leads for HSP system integration
  • Qualified leads only - Only exports leads marked as qualified
  • Incremental sync - Use since parameter for hourly polling without duplicates
  • Export tracking - Each lead tracks when it was exported and how many times

🔐 Permission-Based Access

  • HSP API key - Create dedicated API keys with can_export_to_hsp permission
  • Rate limiting - Configurable limits prevent API abuse
  • Audit logging - All exports are logged for compliance

📊 Export Format

  • HSP-compatible schema - Pre-formatted for HSP CRM import
  • Source tracking - source_lead_id field enables deduplication on HSP side
  • Full lead details - Contact info, property details, project description

January 5, 2026 - LLM-Powered Lead Parsing

➕ Add Leads Directly from the Interface

  • Add Lead button - New button in the filter bar opens a modal for creating leads
  • Three input modes - Choose between Form, Text, or File Upload
  • Form entry - Fill out structured fields for address, contact info, property details
  • Text entry - Describe leads naturally and let our system extract the data automatically
  • Bulk upload - Import multiple leads from CSV or text files in one go
  • Preview extraction - See what data was extracted from text before creating the lead
  • Skip duplicates - Option to automatically skip addresses that already exist

🧠 Smarter Lead Creation from Text

  • Natural language understanding - The create_lead_from_text MCP tool now uses qwen2.5:7b via Ollama for intelligent extraction
  • Handles variations - Works with informal language, typos, and different phrasings that regex can’t match
  • Better accuracy - Correctly extracts names, addresses, phone numbers, and project details from conversational text
  • Graceful fallback - If Ollama is unavailable, automatically falls back to regex-based parsing

🤖 Multi-AI MCP Support

  • Connect any AI - The connect page now supports Claude, ChatGPT, Grok, and more
  • Same MCP server - One endpoint works with all AI providers
  • ChatGPT instructions - Developer Mode + Connectors setup guide added
  • Grok instructions - xAI SDK Remote MCP Tools setup guide added

🦙 Ollama Integration

  • qwen2.5:7b - Primary model for NLP tasks (7.6B parameters, excellent at entity extraction)
  • llama3.2:3b - Lightweight alternative for quick tasks
  • mxbai-embed-large - Embedding model for future semantic search features
  • Self-hosted - All LLM processing runs on our own infrastructure

December 19, 2025 - Conversation Type Improvements

📞 All Conversation Types Now Supported

  • Calls, SMS, Email & more - See all GHL conversation types with correct icons
  • Type-specific icons - 📞 for Calls, 📱 for SMS, 📧 for Email, 💬 for WhatsApp/Live Chat
  • Updated stats - Stats card now shows “Calls” count (replacing SMS) reflecting actual data
  • Type filter expanded - Filter dropdown includes Call, SMS, Email, WhatsApp, Live Chat

🖥️ Better Viewport Handling

  • Scrollable conversations list - Conversations panel now stays within the viewport
  • No more page overflow - The interface is fully usable without scrolling the entire page
  • Improved mobile experience - Works better on all screen sizes

⚡ Faster Load Times with Caching

  • Instant page loads - Conversations cached in your browser for 2 minutes
  • “Last refreshed” indicator - See exactly when data was last fetched
  • Smart cache invalidation - Cache clears automatically when you send a message
  • Force refresh - Click the Refresh button to bypass cache and get fresh data
  • Two-tier caching - Both frontend (localStorage) and backend caching reduce API load

December 19, 2025 - GHL Conversations Dashboard

📬 Unified Inbox for All Conversations

  • Conversations tab - New “Conversations” link in main navigation opens the unified inbox
  • All conversations in one place - See every GHL conversation across all synced leads
  • Split-view layout - Conversation list on the left, message thread on the right
  • Stats at a glance - See total conversations, unread count, calls count, and synced leads
  • Search and filter - Find conversations by name or message content, filter by type
  • Unread filter - Toggle to show only unread conversations

💬 View and Reply to Messages

  • Full message threads - See complete conversation history with timestamps
  • Direction indicators - Messages styled by direction (inbound left-aligned, outbound right-aligned)
  • Reply directly - Send SMS or Email responses without leaving the inbox
  • Type selector - Choose between SMS, Email, or WhatsApp when replying
  • Real-time updates - Refresh to see latest messages

🔗 Lead Context

  • Lead info on every conversation - See the lead’s name, address, and phone number
  • Click to view lead - Jump directly to the full lead detail page
  • Empty state guidance - Helpful message when no leads are synced to GHL yet

December 18, 2024 - Personal GoHighLevel Settings

🔐 Your Own GHL Connection

  • Personal credentials - Each user can now configure their own GoHighLevel Private Integration Token
  • Profile settings - Access GHL settings from your profile page
  • Secure storage - Tokens are stored securely and never displayed after saving
  • Test connection - Verify your GHL credentials work before importing contacts
  • Fallback support - System-wide credentials still work if you don’t set personal ones

📥 What You Can Do With GHL Integration

  • Import your contacts - Pull contacts from your own GHL location into the leads database
  • Sync customer data - Bring over names, phone numbers, emails, and addresses
  • Preview before importing - See exactly what will be imported before committing
  • Avoid duplicates - System automatically skips contacts that already exist (matched by email or phone)
  • Quick setup - Just paste your Private Integration Token from GHL settings

🔄 Bi-Directional GHL Sync (NEW)

  • Push leads to GHL - Send any lead directly to GoHighLevel as a contact with one click
  • View conversations - See all SMS/Email conversations from GHL right on the lead page
  • Send messages - Reply to customers via SMS or Email without leaving Leads Viewer
  • Appointment visibility - View scheduled appointments for each lead from GHL calendar
  • Pipeline sync - Job status automatically syncs to GHL opportunity stages
  • Payment history - See GHL payment transactions on job detail pages
  • Auto-tagging - Leads pushed to GHL are automatically tagged for easy identification

December 17, 2024 - GoHighLevel Integration & Dark Mode

🔗 GoHighLevel Contact Import

  • Import contacts from GHL - Pull your GoHighLevel contacts directly into the leads database
  • MCP-powered - Uses GoHighLevel’s official MCP server for reliable, authenticated access
  • Preview before import - See what data will be imported before committing
  • Skip duplicates - Automatically skips contacts that already exist (matched by email or phone)
  • Field mapping - GHL contact fields are intelligently mapped to lead fields (names, phones, emails, addresses)
  • Setup instructions - Built-in guidance for obtaining your Private Integration Token (PIT)

🌙 Dark Mode Improvements

  • Changelog dark mode - The What’s New page now fully supports dark mode
  • No flash on load - Smart detection prevents white flash when loading in dark mode

December 13, 2024 (Evening) - Jobs Map View & Navigation Updates

🗺️ Jobs Map View

  • Interactive map - See all jobs on a map with clustered markers
  • Status-colored pins - Each job marker reflects its status (yellow=pending, blue=accepted, purple=in progress, green=completed, etc.)
  • Click for details - Tap any marker to see job number, customer, phone, contractor, and value
  • One-click call - Tap the phone number in the popup to call the customer directly
  • Filter sync - Map updates when you change status, county, contractor, or job type filters
  • Remember preference - Map visibility is saved, so it opens automatically next time

🧭 Improved Navigation

  • Jobs link everywhere - Access Jobs from any page in the app
  • Consistent order - Navigation now follows: Property Leads → Jobs → SubContractors → Businesses
  • Mobile-friendly - Horizontal scrolling nav on mobile with abbreviated labels

December 13, 2024 - Jobs Dashboard for Sales & Project Managers

📊 Jobs Overview Page

  • New /jobs page - Dedicated view for salespeople and project managers to manage all jobs
  • Dashboard stats - See total jobs, active count, completed count, total value, and profit at a glance
  • Status pipeline - Click any status pill to quickly filter (Pending, Accepted, In Progress, etc.)
  • Smart filtering - Filter by status, county, contractor, or job type

📋 Job Detail Page

  • Full job management - View complete job details at /jobs/[id]
  • Progress tracker - Visual 6-step workflow from Pending to Paid
  • Property & customer info - See address, coordinates, customer contact all in one place
  • Contractor card - Quick access to assigned contractor’s phone and profile
  • Financials section - Edit job value, materials, and costs with auto-calculated profit
  • Key dates timeline - Track when each milestone was reached
  • Activity log - See all changes and who made them

🔄 Convert Lead to Job

  • Streamlined conversion - Convert any lead to a job right from the lead detail page
  • Subcontractor selection - Choose from available TCEQ-licensed contractors
  • Job type options - Installation, Repair, Inspection, Pump Out, or Maintenance
  • Set initial value - Enter job value with automatic 50/50 profit split calculation
  • Instant link - After conversion, click “View Job Details” to see the new job

🔧 Bug Fixes

  • Auth fix - DEV_MODE now properly bypasses authentication (fixed 503 errors)
  • Contractor display - Service counties now display correctly in dropdowns

December 12, 2024 - SubContractor Portal & Jobs

🏗️ SubContractor Portal

  • Dedicated login - SubContractors now have their own portal at /portal/login
  • Dashboard overview - See pending, active, and completed job counts at a glance
  • Financial summary - Track total earnings, pending payments, and completed job values
  • Accept or decline jobs - Contractors can review and respond to job assignments

📋 Job Management System

  • Lead-to-job conversion - Convert qualified leads to active jobs from the lead detail page
  • Full job lifecycle - Track jobs from pending → accepted → in progress → inspection → completed → paid
  • Job assignment - Assign jobs to specific subcontractors with job type and value
  • Status tracking - Every status change is logged with timestamps and user info
  • Financial tracking - Automatic profit calculation with 50% subcontractor split

💰 Payments Tracking

  • Payment dashboard - SubContractors can see all their completed jobs and payment status
  • Earnings history - View payment history and pending amounts
  • Transparent financials - Clear breakdown of job value, costs, and contractor payment

🔧 Behind the Scenes

  • New API routes - Full REST API for jobs and portal functionality
  • Activity logging - Complete audit trail for all job changes
  • JWT authentication - Secure portal access with token-based auth

December 11, 2024 - Business Relationship Tracking

🏢 Lead Source Businesses

  • Track lead buyers - Keep records of plumbers, HVAC companies, and contractors who want to buy leads
  • Manage lead sources - Track real estate agents and property managers who send you leads
  • Partner networks - Build relationships with related service providers for referrals
  • Import from discovery tool - Bulk import businesses found via Google Places searches

💰 Lead Purchase Tracking

  • Sales history - Track every lead sold with price, payment status, and delivery method
  • Quality feedback - Record buyer feedback to identify your best lead sources
  • Revenue analytics - See total revenue per buyer and track spending patterns
  • Refund management - Handle refund requests and track approval status

🤝 Referral Network Management

  • Partner referrals - Track who refers business to whom in your network
  • Commission tracking - Manage referral fees with percentage or flat-rate options
  • Conversion metrics - See how many referrals actually convert to business
  • Agreement management - Track partnership terms and renewal dates

🔧 Behind the Scenes

  • Clearer naming - Contractors renamed to SubContractors for clarity (these are TCEQ-licensed septic pros)
  • Database improvements - New tables for tracking business relationships
  • Better performance - Fixed async database connection pooling

December 7, 2024 - Major Upgrade

🚀 Performance & Reliability

  • Faster database - Upgraded to PostgreSQL for better speed and reliability
  • Handle more users - Multiple people can use the system at the same time without slowdowns

✨ Install on Your Devices

  • Works like a native app - Install on your phone or computer for quick access
  • Works offline - View recently accessed leads even without internet

🎯 Smart Lead Filtering

  • Remember your location - Automatically shows leads near you
  • Set preferred regions - Only see leads in your assigned counties
  • Distance sorting - Leads closest to you appear first

📱 Better Mobile Experience

  • Optimized for phones - Everything works great on mobile devices
  • Second screen support - View lead details on your phone while working on computer

📊 For Administrators

  • Activity tracking - See who’s doing what in the system
  • Assign regions - Control which counties each team member can see
  • User preferences - Each person can customize their view

December 7, 2024 (Evening) - Latest Updates

📋 What’s New Page

  • See all updates - New changelog page shows what’s been added to the app
  • User-friendly - Written in plain language, not technical jargon
  • Always updated - Check back to see new features as they’re released

🔔 Push Notifications

  • Get notified - Backend infrastructure ready for browser notifications
  • VAPID keys generated - Secure push notification support configured
  • Coming to your device soon - Frontend integration in progress

🛠️ Developer Improvements

  • Faster deployments - Updates roll out more smoothly
  • Better performance - Behind-the-scenes improvements for speed
  • More reliable - Enhanced error handling and monitoring

November 29, 2024 - Initial Launch

🏠 Lead Management

  • View all property leads in one place
  • See owner information and property details
  • Track which leads you’ve contacted
  • Add notes to remember important details

🗺️ Map Views

  • See all leads on an interactive map
  • Click any lead pin to see details
  • Visual overview of your territory

👷 Contractor Management

  • Keep track of all your contractors
  • Store contact information
  • Assign contractors to specific leads

🔐 Secure Access

  • Secure login protects your data
  • Different access levels for team members
  • Automatic logout for security

📤 Export Data

  • Download lead lists for offline work
  • Create reports for your team
  • Share data with other tools

Have ideas for new features? Talk to your administrator about what would help you most!

💡 Got an idea for the lab?

We're always experimenting! Let your lab administrator know what formulas would help you transmute leads faster.

— Page ∞ of Lab Notebook #1 —
Last updated: January 2026