What is Farcaster Sync?
Farcaster operates on a distributed network of independent servers called hubs. Unlike centralized social platforms where a single company controls the data, Farcaster relies on these hubs to store and validate messages. For the network to function correctly, every hub must stay in agreement about the state of the social graph—essentially, who follows whom and what has been posted.
Hub synchronization is the process by which these independent servers exchange messages to maintain that consistency. When you post a message, it lands on the hub you chose. That hub then shares the message with its peers. This exchange happens in a multi-channel sync process, where hubs continuously compare their data to ensure they are all running the same version of the truth.
This mechanism is critical for anyone running their own infrastructure. If your hub falls behind in sync, your application may display stale data or miss recent activity. Understanding how this synchronization works helps you monitor your hub’s health and ensure your clients are receiving accurate, up-to-date information from the network.
Install and configure your hub
Setting up a Farcaster hub requires the Hubble client, a Node.js application that syncs the Farcaster protocol state. This process involves cloning the official repository, configuring environment variables for your specific network and storage needs, and initializing the sync.
Begin by cloning the official Hubble repository from GitHub. This provides the source code and configuration templates necessary for your instance.
git clone https://github.com/farcasterxyz/hubble.git
cd hubble
Copy the example environment file and edit it to suit your infrastructure. Key variables include ETH_MAINNET_RPC for the Ethereum node connection, DATA_DIR for local storage, and IPFS_GATEWAY if you intend to use IPFS for message storage.
cp .env.example .env
nano .env
Use npm to install the required Node.js packages. This prepares the runtime environment for the Hubble client.
npm install
Run the Hubble client to begin syncing the Farcaster state. The first sync will take several hours as it downloads the entire history of casts, profiles, and links from the network.
npm start
Once the sync completes, your hub will be ready to serve requests from Farcaster clients and developers building on the protocol.
Monitor sync progress and health
Running a Farcaster hub is not the same as being synced. A hub process can start, bind to a port, and show as "online" without holding the latest network state. If you skip verification, your clients will read stale data or see gaps in user history. You need to confirm the hub is actively catching up to the network.
Check the sync percentage
The most direct indicator of health is the sync percentage. This number tells you how much of the canonical Farcaster history your hub has stored. A healthy hub should eventually reach 100%. Until then, it is still downloading historical messages.
You can check this status via the hub's REST API or CLI. Query the /hub/health endpoint or run npx @farcaster/hub cli health. Look for the syncPercent field. If it is below 100%, your hub is still catching up. Do not rely on it for production traffic until it hits full sync. This step is critical because a "ready" status in the logs often masks a partially synced database.
Verify message lag and latency
Sync percentage alone does not tell you if you are falling behind in real-time. You also need to monitor message lag—the difference between the latest message on the network and the latest message your hub has processed. Low lag means your hub is keeping pace with new casts and reactions as they happen.
Check the lag or latestMessageTimestamp fields in your health check response. Compare this against the current network time. If the lag grows consistently, your hub is struggling to process incoming messages. This often happens if your hardware is under-provisioned or if you are on a slow network connection. Addressing this ensures your hub remains useful for real-time applications.
Inspect database integrity
Finally, verify that the underlying database is not corrupted. Farcaster hubs use LevelDB to store messages. If the database is compromised, sync metrics can lie. Check the database size and recent write operations. A sudden drop in write throughput or an increase in error logs indicates a problem.
Use the npx @farcaster/hub cli db-info command to inspect database statistics. Look for anomalies in the message count or storage usage. If the database size is unexpectedly small or large, it may indicate a sync failure or a leak. Regularly monitoring these metrics helps you catch issues before they impact your users.
Use visual monitoring tools
For a more intuitive overview, consider using a dashboard or charting tool. Visualizing sync percentage and message lag over time helps you spot trends and anomalies. You can integrate your hub's health endpoint with tools like Grafana or Prometheus to create real-time dashboards.
This approach allows you to set up alerts for when sync percentage drops or lag increases. You can also track historical performance to optimize your hub's configuration. Visual monitoring is particularly useful for managing multiple hubs or troubleshooting complex network issues. It provides a clear, at-a-glance view of your hub's health.
Run curl http://localhost:2281/hub/health or use the CLI command npx @farcaster/hub cli health. Look for the syncPercent and lag fields in the JSON response. This gives you the raw data needed to assess sync status.
If syncPercent is less than 100%, your hub is still downloading historical data. Wait until it reaches 100% before relying on it for production traffic. This ensures you have the complete history of the Farcaster network.
Check the lag field to see how far behind the network your hub is. A low lag indicates your hub is keeping up with real-time messages. If lag increases, investigate your hardware and network connection for bottlenecks.
Fix Common Sync Errors
A hub can be online and healthy while still falling behind the network. If your sync percentage stalls, the issue is usually database bloat, connection timeouts, or stale storage files. Follow this sequence to isolate and resolve the most common sync failures.
1. Check Sync Percentage
Connectivity alone does not guarantee synchronization. You must verify the hub’s data progress against the mainnet. Use the official Hub API to query your node’s current sync status. If the percentage is stuck below 90%, the hub is not keeping up with new casts.
2. Clear Stale Storage Files
Database bloat is the primary cause of stuck syncs. Over time, corrupted or redundant files can prevent the hub from indexing new data efficiently. Stop the hub service and delete the storage directory contents. Restart the hub to force a fresh index build. This often resolves "stuck" states immediately.
3. Resolve Connection Timeouts
If the hub logs show repeated timeout errors, check your firewall and network latency. Hubs require stable, high-bandwidth connections to sync large batches of data. Ensure port 2283 (or your configured port) is open and not blocked by ISP throttling. Test the connection using telnet or nc to verify reachability.
4. Verify Network Consistency
After clearing files or restarting, monitor the sync percentage for 15 minutes. It should climb steadily. If it remains flat, check the hub’s logs for gossipsub errors, which indicate peer connectivity issues. Joining more reliable peer hubs can stabilize the sync process.
Query hub data for insights
Once your hubs are synced, the raw data becomes the foundation for market research. You can track specific casts, monitor user growth, or analyze engagement trends across channels. The Farcaster data model separates on-chain contracts from hub-stored messages, so your queries target the hub layer for social interactions.
To start, query main interactions like casts and reactions. This helps you identify trending topics or influential users. You can filter by fid (Farcaster ID) to track specific accounts or by channel to monitor niche discussions. The Cryptography Databytes guide provides a clear breakdown of how to structure these queries effectively.
For user growth metrics, focus on signups and login events. These on-chain and hub events tell you how many new users are entering the ecosystem. Combine this with cast volume to calculate engagement rates. This data is essential for understanding the health of your specific niche.
Pre-flight checklist for data integrity
Before running analytics, ensure your data is clean and complete.
-
Verify hub sync status matches the latest block height
-
Confirm fid ranges are within your query parameters
-
Check for duplicate casts due to re-syncs
-
Validate channel names against the official registry
No comments yet. Be the first to share your thoughts!