Browser Automation - SpiderIQ by Di-Atomic

Overview

SpiderBrowser is an anti-detect browser profile management system using Camoufox for C++-level fingerprint spoofing. It maintains persistent authenticated sessions at scale (200+ accounts) for platforms requiring login.

SpiderBrowser is an Admin-only feature. All endpoints require X-Admin-Key authentication.

Primary Use Cases

YouTube Cookie Export

Export authenticated cookies in Netscape format for yt-dlp video downloads

Email Warmup

Manage 200+ SmartLead.ai accounts for email deliverability warmup

Platform Authentication

Maintain LinkedIn, Instagram sessions with realistic fingerprints

Google Maps Quality

Warmed profiles get full data vs compact format from Google Maps

Key Features

Camoufox Integration - C++-level fingerprint spoofing (not JavaScript-based)
VNC Web Access - Manual login, CAPTCHA solving, 2FA via noVNC
Cookie Export - Netscape format compatible with yt-dlp
SpiderProxy Integration - Mobile IP rotation for realistic sessions
Profile Warmup - Automated browsing to build profile history
Batch Operations - Start/stop/export multiple profiles simultaneously

How It Works

Architecture

API Gateway (/api/v1/admin/browser/*) - Profile management, browser control
Browser Worker (workers/SpiderBrowser/) - Camoufox instances with VNC
noVNC - Web-based VNC access on port 6080
SpiderProxy - Mobile proxy assignment for each profile

Quick Start

1. Create a Profile

curl -X POST "https://spideriq.ai/api/v1/admin/browser/profiles" \
  -H "X-Admin-Key: YOUR_ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "profile_name": "youtube-prod-01",
    "platform": "youtube",
    "account_email": "user@example.com",
    "timezone": "Europe/London"
  }'

Response:

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "profile_name": "youtube-prod-01",
  "platform": "youtube",
  "status": "new",
  "created_at": "2025-01-21T10:00:00Z"
}

2. Start Browser Session

curl -X POST "https://spideriq.ai/api/v1/admin/browser/profiles/{profile_id}/start" \
  -H "X-Admin-Key: YOUR_ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "headless": false,
    "navigate_to": "https://youtube.com"
  }'

Response:

{
  "success": true,
  "vnc_url": "http://localhost:6080/vnc.html?autoconnect=true",
  "browser_status": "running"
}

Open the VNC URL in your browser to access the Camoufox instance:

http://localhost:6080/vnc.html?autoconnect=true

Use VNC to:

Complete manual login
Solve CAPTCHAs
Handle 2FA verification
Verify session is active

4. Export Cookies

After successful login, export cookies:

curl -X POST "https://spideriq.ai/api/v1/admin/browser/profiles/{profile_id}/cookies/export" \
  -H "X-Admin-Key: YOUR_ADMIN_KEY"

Response:

{
  "profile_id": "550e8400-e29b-41d4-a716-446655440000",
  "platform": "youtube",
  "cookies_json": [...],
  "netscape_format": "# Netscape HTTP Cookie File\n.youtube.com\tTRUE\t/\tTRUE\t...",
  "expires_at": "2025-02-21T10:00:00Z"
}

5. Stop Browser

curl -X POST "https://spideriq.ai/api/v1/admin/browser/profiles/{profile_id}/stop" \
  -H "X-Admin-Key: YOUR_ADMIN_KEY"

Profile Management

Create Profile

POST /api/v1/admin/browser/profiles

Request Body:

{
  "profile_name": "linkedin-sales-01",
  "platform": "linkedin",
  "account_email": "sales@company.com",
  "account_username": "john.smith",
  "password": "secure_password",
  "timezone": "America/New_York",
  "locale": "en-US",
  "notes": "Sales team account for outreach"
}

Parameters:

Field	Type	Required	Description
`profile_name`	string	Yes	Unique profile identifier
`platform`	string	Yes	`youtube`, `linkedin`, `instagram`, `smartlead`, `googlemaps`
`account_email`	string	No	Account email for the platform
`account_username`	string	No	Account username
`password`	string	No	Encrypted password (Fernet)
`timezone`	string	No	Profile timezone (default: `Europe/Kiev`)
`locale`	string	No	Browser locale (default: `en-US`)
`notes`	string	No	Internal notes

List Profiles

GET /api/v1/admin/browser/profiles?platform=youtube&status=active

Query Parameters:

Parameter	Description
`platform`	Filter by platform
`status`	Filter by status (`new`, `warming_up`, `active`, `needs_login`, `banned`, `cooldown`)
`limit`	Results per page (default: 50)
`offset`	Pagination offset

Update Profile

PUT /api/v1/admin/browser/profiles/{profile_id}

{
  "status": "active",
  "notes": "Updated notes"
}

Delete Profile

DELETE /api/v1/admin/browser/profiles/{profile_id}

Profile Status Lifecycle

new

Profile just created, not yet used

warming_up

In warmup phase (2-3 weeks of gradual usage)

active

Ready for production use

needs_login

Session expired, requires re-authentication via VNC

banned

Account banned on platform (requires new account)

cooldown

Temporary rate-limit cooldown (auto-recovers)

VNC Access

SpiderBrowser provides web-based VNC access via noVNC for manual browser operations.

Access Methods

noVNC Web Interface:

http://localhost:6080/vnc.html?autoconnect=true

Direct VNC (port 5900):

vnc://localhost:5900
Password: spiderbrowser (configurable via VNC_PASSWORD)

Common VNC Operations

Initial Login - Complete first-time platform login
CAPTCHA Solving - Manually solve CAPTCHA challenges
2FA Verification - Enter SMS/authenticator codes
Session Recovery - Re-login when session expires
Manual Warmup - Browse naturally to build profile history

Keep VNC sessions short. Close the browser when done to free resources. The profile state is automatically saved.

Export Single Profile

POST /api/v1/admin/browser/profiles/{profile_id}/cookies/export

Returns cookies in both JSON and Netscape format.

Export All YouTube Cookies

Aggregate cookies from all active YouTube profiles:

POST /api/v1/admin/browser/cookies/youtube

Response:

{
  "profiles_count": 15,
  "netscape_format": "# Netscape HTTP Cookie File\n...",
  "exported_at": "2025-01-21T10:00:00Z"
}

Using Cookies with yt-dlp

Save the Netscape format to a file and use with yt-dlp:

# Save cookies to file
echo "$NETSCAPE_COOKIES" > youtube_cookies.txt

# Use with yt-dlp
yt-dlp --cookies youtube_cookies.txt "https://www.youtube.com/watch?v=VIDEO_ID"

Warmup & Automation

Profile Warmup

New profiles should be warmed up before production use to build browsing history and avoid detection. Start Warmup:

POST /api/v1/admin/browser/profiles/{profile_id}/warmup/start

Warmup Activities:

Random Google searches
Visit popular websites
Watch YouTube videos
Scroll and click naturally
Build cookie history

Recommended Warmup Schedule:

Week 1: 15-30 minutes/day, light browsing
Week 2: 30-60 minutes/day, normal activity
Week 3: Full production usage

Automated login for supported platforms:

POST /api/v1/admin/browser/profiles/{profile_id}/login

Automated login may trigger security challenges. Use VNC for manual verification if needed.

Navigate to URL

POST /api/v1/admin/browser/profiles/{profile_id}/navigate

{
  "url": "https://youtube.com/feed/subscriptions"
}

Take Screenshot

POST /api/v1/admin/browser/profiles/{profile_id}/screenshot

Returns base64-encoded PNG screenshot.

SpiderProxy Integration

Each browser profile can be assigned a sticky mobile proxy from SpiderProxy for realistic IP rotation.

Automatic Assignment

When starting a browser, SpiderProxy automatically assigns an available modem:

POST /api/v1/admin/browser/profiles/{profile_id}/start

The proxy assignment is saved to the profile and reused for subsequent sessions.

Manual Assignment

Assign a specific modem to a profile:

PUT /api/v1/admin/browser/profiles/{profile_id}

{
  "proxy_modem_id": "modem-uuid-here"
}

Proxy Benefits

Mobile IPs - Carrier-grade residential IPs
Sticky Sessions - Same IP across multiple sessions
Auto-Rotation - Scheduled IP rotation (configurable)
Health Monitoring - Automatic fallback on modem issues

Batch Operations

Start Multiple Browsers

POST /api/v1/admin/browser/batch/start

{
  "profile_ids": [
    "uuid-1",
    "uuid-2",
    "uuid-3"
  ]
}

Stop Multiple Browsers

POST /api/v1/admin/browser/batch/stop

{
  "profile_ids": ["uuid-1", "uuid-2", "uuid-3"]
}

POST /api/v1/admin/browser/batch/cookies

{
  "profile_ids": ["uuid-1", "uuid-2", "uuid-3"]
}

Pool Status

Get status of all running browsers:

GET /api/v1/admin/browser/pool/status

Response:

{
  "active_browsers": 5,
  "max_browsers": 10,
  "memory_used_mb": 2048,
  "memory_limit_mb": 8192,
  "browsers": [
    {
      "profile_id": "uuid-1",
      "platform": "youtube",
      "status": "running",
      "uptime_seconds": 3600
    }
  ]
}

Best Practices

Profile Naming Convention

Use a consistent naming scheme:

{platform}-{purpose}-{number}

Examples:

youtube-media-01 - YouTube for SpiderMedia
linkedin-sales-01 - LinkedIn for sales outreach
smartlead-warmup-15 - SmartLead email warmup

Warmup Guidelines

New profiles need 2-3 weeks of warmup before production use. Rushing warmup increases ban risk.

Start Slow - Light activity in week 1
Vary Behavior - Don’t repeat exact same actions
Use Real Content - Watch actual videos, read posts
Respect Timing - Mimic human usage patterns (not 24/7)
Monitor Health - Check for security challenges

Security Considerations

Never commit credentials to version control. Use environment variables or the encrypted password field.

Store passwords encrypted (Fernet)
Rotate VNC passwords regularly
Limit admin key access
Monitor for unusual activity
Delete banned profiles promptly

Resource Management

Max 10 concurrent browsers per worker
512MB memory per browser instance
4GB shared memory (shm_size) for Chrome
Close idle browsers to free resources

Troubleshooting

Browser Won’t Start

Check worker health

curl http://localhost:3010/health

Ensure the SpiderBrowser worker is running and healthy.

Check pool capacity

GET /api/v1/admin/browser/pool/status

If active_browsers >= max_browsers, stop some browsers first.

Check memory

If memory is exhausted, the worker may reject new browsers. Restart the worker or reduce concurrent browsers.

VNC Black Screen

Browser not started

Ensure the browser is running:

GET /api/v1/admin/browser/profiles/{id}/status

Display issue

The Xvfb virtual display may have crashed. Restart the browser worker.

Not logged in

Access VNC and verify the session is authenticated before exporting.

Cookies expired

Re-login via VNC to refresh the session.

Profile Banned

Immediate ban

The fingerprint may be detected. Try a different proxy/modem.

Gradual ban

Behavior patterns triggered detection. Use longer warmup and more natural browsing.

API Reference

Profile Endpoints

Method	Endpoint	Description
POST	`/admin/browser/profiles`	Create profile
GET	`/admin/browser/profiles`	List profiles
GET	`/admin/browser/profiles/{id}`	Get profile
PUT	`/admin/browser/profiles/{id}`	Update profile
DELETE	`/admin/browser/profiles/{id}`	Delete profile

Browser Control

Method	Endpoint	Description
POST	`/admin/browser/profiles/{id}/start`	Start browser
POST	`/admin/browser/profiles/{id}/stop`	Stop browser
GET	`/admin/browser/profiles/{id}/status`	Get status
POST	`/admin/browser/profiles/{id}/navigate`	Navigate to URL
POST	`/admin/browser/profiles/{id}/screenshot`	Take screenshot

Method	Endpoint	Description
POST	`/admin/browser/profiles/{id}/cookies/export`	Export cookies
GET	`/admin/browser/profiles/{id}/cookies/latest`	Get latest export
POST	`/admin/browser/cookies/youtube`	Export all YouTube cookies

Automation

Method	Endpoint	Description
POST	`/admin/browser/profiles/{id}/login`	Auto-login
POST	`/admin/browser/profiles/{id}/warmup/start`	Start warmup
POST	`/admin/browser/profiles/{id}/warmup/stop`	Stop warmup

Batch & Stats

Method	Endpoint	Description
POST	`/admin/browser/batch/start`	Batch start
POST	`/admin/browser/batch/stop`	Batch stop
POST	`/admin/browser/batch/cookies`	Batch export
GET	`/admin/browser/pool/status`	Pool status
GET	`/admin/browser/stats`	Statistics
GET	`/admin/browser/health`	Worker health

Need Help?

API Reference

Complete API documentation

SpiderProxy Guide

Mobile proxy setup for browsers

Support

Contact support team

System Status

Check API health

Guides

​Overview

​Primary Use Cases

YouTube Cookie Export

Email Warmup

Platform Authentication

Google Maps Quality

​Key Features

​How It Works

​Architecture

​Quick Start

​1. Create a Profile

​2. Start Browser Session

​3. Access VNC for Manual Login

​4. Export Cookies

​5. Stop Browser

​Profile Management

​Create Profile

​List Profiles

​Update Profile

​Delete Profile

​Profile Status Lifecycle

​VNC Access

​Access Methods

​Common VNC Operations

​Cookie Export

​Export Single Profile

​Export All YouTube Cookies

​Using Cookies with yt-dlp

​Warmup & Automation

​Profile Warmup

​Platform-Specific Login

​Navigate to URL

​Take Screenshot

​SpiderProxy Integration

​Automatic Assignment

​Manual Assignment

​Proxy Benefits

​Batch Operations

​Start Multiple Browsers

​Stop Multiple Browsers

​Batch Cookie Export

​Pool Status

​Best Practices

​Profile Naming Convention

​Warmup Guidelines

​Security Considerations

​Resource Management

​Troubleshooting

​Browser Won’t Start

​VNC Black Screen

​Cookie Export Empty

​Profile Banned

​API Reference

​Profile Endpoints

​Browser Control

​Cookie Management

​Automation

​Batch & Stats

​Need Help?

API Reference

SpiderProxy Guide

Support

System Status

Overview

Primary Use Cases

Key Features

How It Works

Architecture

Quick Start

1. Create a Profile

2. Start Browser Session

3. Access VNC for Manual Login

4. Export Cookies

5. Stop Browser

Profile Management

Create Profile

List Profiles

Update Profile

Delete Profile

Profile Status Lifecycle

VNC Access

Access Methods

Common VNC Operations

Cookie Export

Export Single Profile

Export All YouTube Cookies

Using Cookies with yt-dlp

Warmup & Automation

Profile Warmup

Platform-Specific Login

Navigate to URL

Take Screenshot

SpiderProxy Integration

Automatic Assignment

Manual Assignment

Proxy Benefits

Batch Operations

Start Multiple Browsers

Stop Multiple Browsers

Batch Cookie Export

Pool Status

Best Practices

Profile Naming Convention

Warmup Guidelines

Security Considerations

Resource Management

Troubleshooting

Browser Won’t Start

VNC Black Screen

Cookie Export Empty

Profile Banned

API Reference

Profile Endpoints

Browser Control

Cookie Management

Automation

Batch & Stats

Need Help?