The basics
Hey everyone, and welcome to my very first blog post on etherlinkinsights! I'm etherlinkintern, and I'm super excited to kick things off by diving into a challenge I've been tackling: figuring out the number of active users for a protocol on Etherlink.
This is where the rubber meets the road, and I want to walk you through the tools and steps I'm using. Consider this your behind-the-scenes pass to my Etherlink explorations!
Unpacking Active Users: Our First Tool in the Kit!
So, the big question is: how do we accurately gauge who's actively using a protocol? There are a bunch of ways to approach this, but before we even get to the fancy analytics, we need to talk about getting the raw data. And for that, the very first step in my journey (and often the most foundational for any serious blockchain explorer!) was to set up my own Etherlink RPC (Remote Procedure Call) node.
Now, you might be thinking, "Do I have to run my own node?" And the short answer is: no, not necessarily! There are plenty of public RPC services out there. However, for what I intend to do – which involves a lot of querying and digging deep – relying solely on public services can quickly lead to hitting rate limits or experiencing slower response times.
Running my own RPC node means I'm in the driver's seat! I get blazing fast queries directly from my server, and I don't have to worry about getting throttled when I'm in the middle of a deep dive. It's a game-changer for speed and efficiency.
Getting Started with Your Own Etherlink RPC: The start_observer.sh Script
If you're interested in following along or setting up your own powerful data pipeline, you're in luck! The Etherlink documentation is genuinely fantastic and provides some really clear guidance on getting your RPC stack up and running.
But to make it even easier, here’s the exact start_observer.sh script I used to get mine spinning. I've reviewed this script, and it doesn't contain any sensitive environment variables or secrets, so you can share and use it with confidence. Feel free to copy, paste, and tweak as you get started!
#!/bin/bash
# This script is designed to start and maintain an Etherlink Observer EVM node.
# It includes intelligent retry logic and fallback mechanisms to ensure the node
# stays online and synced, even if there are temporary issues with the rollup node.
# --- Configuration Section ---
# These variables allow you to easily customize your node's behavior.
ROLLUP_ENDPOINT="http://localhost:8932" # The endpoint of your local Etherlink Rollup Node.
# This is where your EVM node connects to get block data.
LOG_FILE="evm_node.log" # File to log all output from the EVM node.
DAEMON_LOG_FILE="observer_daemon.log" # File to log messages from this startup script itself.
PID_FILE="evm_node.pid" # File to store the Process ID (PID) of the running EVM node.
FALLBACK_TIMEOUT=3600 # How long (in seconds) to run in "fallback" mode if tracking fails (1 hour).
MAX_WAIT_TIME=14400 # Maximum wait time (in seconds) between retries (4 hours).
# Base command arguments for the 'octez-evm-node' observer.
# These define how your EVM node operates, such as network, history depth, and transaction pool limits.
BASE_ARGS=(
"run" "observer"
"--network" "mainnet" # Specifies the network to connect to (e.g., mainnet, ghostnet).
"--history" "rolling:30" # Keeps 30 days of historical data; adjust based on your needs.
"--init-from-snapshot" # Helps speed up initial sync by using a snapshot.
"--rollup-node-endpoint" "$ROLLUP_ENDPOINT" # Connects to our local rollup node.
"--ws" # Enables WebSocket support for real-time updates.
"--max-number-blocks" "1000" # Max blocks to retrieve in one go; fine-tune for performance.
"--max-number-logs" "1000" # Max logs to retrieve; similar to blocks, affects performance.
"--chunk-size" "100" # How many items to process at once from the rollup node.
"--tx-pool-tx-per-addr-limit" "10" # Limits transactions per address in the transaction pool.
"--tx-pool-addr-limit" "1000" # Limits total addresses in the transaction pool.
"--tx-pool-timeout-limit" "3600" # How long transactions stay in the pool.
)
# --- Helper Functions ---
# These functions encapsulate common tasks, making the script cleaner and more maintainable.
# Logs messages with a timestamp to both stdout and the specified LOG_FILE.
log_message() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOG_FILE"
}
# Cleans up any running observer processes identified by the PID_FILE.
cleanup() {
log_message "Cleaning up processes..."
if [[ -f "$PID_FILE" ]]; then
local pid=$(cat "$PID_FILE")
if kill -0 "$pid" 2>/dev/null; then # Check if the process exists
log_message "Killing process $pid"
kill -TERM "$pid" 2>/dev/null # Attempt graceful shutdown
sleep 5
if kill -0 "$pid" 2>/dev/null; then # If still running, force kill
kill -KILL "$pid" 2>/dev/null
fi
fi
rm -f "$PID_FILE" # Remove the PID file after cleanup
fi
}
# Checks recent log output for common error patterns indicating issues with the rollup node sync.
check_rollup_sync_error() {
local log_output="$1"
# Looking for keywords that suggest the rollup node isn't keeping up or is disconnected.
if echo "$log_output" | grep -qi "rollup.*not.*caught.*up\|rollup.*sync\|rollup.*behind\|rollup.*lagging\|unable to connect to the node\|communication was lost\|econnrefused\|rollup node.*error\|rollup.*node.*communicate\|received block.*too old\|block.*from.*rollup.*too old\|internal error.*rollup"; then
return 0 # Error found
fi
return 1 # No error found
}
# Verifies if the rollup node is actively processing blocks by checking log messages over a duration.
check_rollup_block_processing() {
local log_file="$1"
local duration="${2:-900}" # Default check for 15 minutes (900 seconds)
local start_time=$(date +%s)
log_message "Checking for rollup block processing for up to ${duration}s..."
while [[ $(($(date +%s) - start_time)) -lt $duration ]]; do
# We look for specific log entries that confirm new blocks are being processed.
if tail -n 100 "$log_file" | grep -q "head is now\|rollup node confirmed block\|Etherlink blocks.*processed"; then
log_message "Block processing confirmed in logs"
return 0 # Success - blocks are being processed
fi
# If the observer process dies during this check, something is wrong.
if [[ -f "$PID_FILE" ]] && kill -0 "$(cat "$PID_FILE")" 2>/dev/null; then
sleep 15
else
log_message "Process terminated during block processing check"
return 1 # Process died
fi
done
log_message "No block processing detected within ${duration}s"
return 1 # Failure - no block processing detected
}
# Runs the observer with its default rollup node tracking enabled.
# It monitors the logs for errors and can exit with a timeout.
run_with_tracking() {
local timeout_duration_arg="$1" # Optional timeout for this specific run.
local log_message_extra=""
if [[ -n "$timeout_duration_arg" ]]; then
log_message_extra=" (timeout: ${timeout_duration_arg}s)"
fi
log_message "Starting observer with rollup node tracking${log_message_extra}"
# Start the octez-evm-node in the background, redirecting all output to LOG_FILE.
nohup ./octez-evm-node "${BASE_ARGS[@]}" >> "$LOG_FILE" 2>&1 &
local pid=$! # Capture the PID of the background process.
# Give the process a moment to start up.
sleep 3
if ! kill -0 "$pid" 2>/dev/null; then
log_message "Process failed to start or exited immediately"
return 1
fi
echo "$pid" > "$PID_FILE" # Store the PID for later management.
log_message "Observer started successfully with PID: $pid"
echo "Process started with PID: $pid"
local start_time=$(date +%s)
# Continuously monitor the process as long as it's running.
while kill -0 "$pid" 2>/dev/null; do
local current_time=$(date +%s)
local elapsed=$((current_time - start_time))
# Check if a specific timeout for this run has been reached.
if [[ -n "$timeout_duration_arg" ]] && [[ $elapsed -ge $timeout_duration_arg ]]; then
log_message "Timeout reached (${timeout_duration_arg}s), switching to fallback mode"
cleanup # Stop the current process.
return 2 # Return code 2 indicates a timeout.
fi
# Periodically check recent logs for any rollup sync errors.
if [[ -f "$LOG_FILE" ]]; then
local recent_logs=$(tail -n 20 "$LOG_FILE")
if check_rollup_sync_error "$recent_logs"; then
log_message "Rollup node sync error detected, switching to fallback mode"
cleanup # Stop and prepare for fallback.
return 1 # Return code 1 indicates a sync error.
fi
fi
sleep 10 # Wait before the next check.
done
# If the process stopped naturally, check its final logs for errors.
local exit_code=$?
rm -f "$PID_FILE"
log_message "Observer process ended with exit code: $exit_code"
if [[ -f "$LOG_FILE" ]]; then
local recent_logs=$(tail -n 50 "$LOG_FILE")
if check_rollup_sync_error "$recent_logs"; then
log_message "Rollup node error detected in final logs, treating as sync error"
return 1 # Consider it a sync error if problems are found at the end.
fi
fi
return $exit_code # Return the actual exit code of the process.
}
# Runs the observer with rollup node tracking disabled (--dont-track-rollup-node flag).
# This is useful as a fallback if the main tracking mode is having issues, allowing the EVM
# node to still process some blocks without getting stuck waiting for a perfectly synced rollup.
run_without_tracking() {
local duration="$1" # How long to run in this fallback mode.
log_message "Starting observer WITHOUT rollup node tracking for ${duration}s"
# Add the special flag to disable rollup node tracking.
local fallback_args=("${BASE_ARGS[@]}" "--dont-track-rollup-node")
# Use 'timeout' command to ensure the process stops after the specified duration.
nohup timeout "$duration" ./octez-evm-node "${fallback_args[@]}" >> "$LOG_FILE" 2>&1 &
local timeout_pid=$! # PID of the 'timeout' wrapper process.
sleep 3
# Find the actual octez-evm-node PID within the 'timeout' wrapper.
local actual_pid=$(pgrep -f "octez-evm-node.*dont-track-rollup-node" | head -1)
if [[ -z "$actual_pid" ]]; then
log_message "Fallback process failed to start or exited immediately"
kill "$timeout_pid" 2>/dev/null # Clean up the timeout wrapper if it failed.
return 1
fi
echo "$actual_pid" > "$PID_FILE"
log_message "Fallback observer started successfully with PID: $actual_pid (timeout wrapper: $timeout_pid)"
echo "Fallback process started with PID: $actual_pid"
wait "$timeout_pid" 2>/dev/null # Wait for the 'timeout' command to finish.
local exit_code=$?
rm -f "$PID_FILE"
log_message "Fallback mode completed with exit code: $exit_code"
return $exit_code
}
# --- Main Execution Loop ---
# This is the heart of the script, orchestrating the startup, monitoring, and retry logic.
main() {
log_message "Starting Etherlink Observer with intelligent rollup tracking"
# Flag to handle graceful shutdowns (e.g., Ctrl+C).
local shutdown_requested=false
# Signal traps ensure that cleanup() is called if the script is interrupted.
trap 'shutdown_requested=true; cleanup; log_message "Shutdown requested, exiting..."; exit 0' INT TERM
trap 'cleanup' EXIT # Ensure cleanup on any exit.
local wait_time=7200 # Initial wait time for retries (2 hours).
local attempt=1 # Counter for retry attempts.
# The main loop keeps trying to run the observer until a successful, stable state is reached
# or a shutdown is requested.
while [[ "$shutdown_requested" != "true" ]]; do
log_message "=== Attempt $attempt ==="
# First, try running with full rollup node tracking.
run_with_tracking
local result=$?
case $result in
0)
log_message "Observer completed successfully"
break # Exit the loop if successful.
;;
1|2)
# If a sync error (1) or timeout (2) occurred, switch to fallback.
log_message "Running fallback mode for 1 hour..."
run_without_tracking "$FALLBACK_TIMEOUT"
# After fallback, attempt to resume with tracking, but with a short timeout initially.
log_message "Attempting to resume with rollup node tracking..."
run_with_tracking 300 # Try for 5 minutes.
# Check if actual block processing has resumed.
if check_rollup_block_processing "$LOG_FILE" 900; then # Check for 15 minutes.
log_message "Successfully resumed with rollup node tracking"
# If successful, try to run for a longer, stable period.
run_with_tracking 86400 # Run for 24 hours (86400 seconds).
else
# If resuming tracking failed to process blocks, clean up and go back to fallback/retry logic.
cleanup
log_message "Failed to resume with tracking, will run fallback mode for ${wait_time}s before retry..."
# Run fallback mode during the wait period to keep the observer somewhat active.
run_without_tracking "$wait_time"
# Implement exponential backoff for wait times, up to MAX_WAIT_TIME.
wait_time=$((wait_time * 2))
if [[ $wait_time -gt $MAX_WAIT_TIME ]]; then
wait_time=$MAX_WAIT_TIME
fi
fi
;;
*)
# Handle unexpected errors.
log_message "Unexpected error (exit code: $result), waiting ${wait_time}s before retry..."
# Sleep in a way that allows interruption for graceful shutdown.
for ((i=0; i<wait_time; i++)); do
if [[ "$shutdown_requested" == "true" ]]; then
break
fi
sleep 1
done
wait_time=$((wait_time * 2))
if [[ $wait_time -gt $MAX_WAIT_TIME ]]; then
wait_time=$MAX_WAIT_TIME
fi
;;
esac
if [[ "$shutdown_requested" == "true" ]]; then
break
fi
attempt=$((attempt + 1))
done
log_message "Observer script completed"
}
# --- Initial Checks & Execution ---
# Ensure the octez-evm-node binary is present.
if [[ ! -f "./octez-evm-node" ]]; then
log_message "ERROR: octez-evm-node binary not found in current directory"
exit 1
fi
# Make sure the binary is executable.
chmod +x ./octez-evm-node
# Handle different startup modes: foreground, background (internal), or daemonize.
if [[ "$1" == "--foreground" ]]; then
# Useful for debugging, runs script directly in current terminal.
shift
main "$@"
elif [[ "$1" == "--background" ]]; then
# Internal flag used when the script daemonizes itself.
shift
main "$@"
else
# Default: start as a daemon (background process).
echo "Starting observer daemon..."
echo "Daemon logs will be written to: $DAEMON_LOG_FILE"
echo "EVM node logs will be written to: $LOG_FILE"
# Use nohup to run the script in the background, redirecting its own logs to DAEMON_LOG_FILE.
nohup bash "$0" --background "$@" >> "$DAEMON_LOG_FILE" 2>&1 &
daemon_pid=$!
echo "Observer daemon started with PID: $daemon_pid"
echo "To monitor daemon: tail -f $DAEMON_LOG_FILE"
echo "To monitor EVM node: tail -f $LOG_FILE"
exit 0
fiWhat's Happening in This Script?
This start_observer.sh script is pretty robust! Here's a quick rundown of its superpowers:
- Smart Configuration: At the top, you'll find easy-to-change variables for things like your rollup node's address, where logs go, and how long to wait between retries. This makes it super flexible!
- Intelligent Tracking: The script attempts to run your EVM node with "rollup node tracking" enabled by default. This is usually the best way for the EVM node to stay perfectly in sync with the underlying Etherlink rollup.
- Error Detection & Fallback: It constantly monitors the EVM node's logs for signs that it's losing sync with the rollup node (which can happen sometimes!). If it detects an issue, it gracefully switches to a "fallback" mode where the EVM node runs without strict rollup tracking for a while. This keeps it somewhat active and prevents it from getting stuck.
- Retry and Recovery: After a fallback period, it tries to resume full tracking. If that still doesn't work perfectly (e.g., no new blocks are processed), it has an exponential backoff mechanism, waiting longer between retries to give the network time to stabilize, all while running in fallback mode in the interim.
- Daemon Mode: By default, it starts your observer as a background process (a "daemon"), so it keeps running even if you close your terminal. It also thoughtfully logs its own status to observer_daemon.logseparate from the EVM node's main logs.
- Graceful Shutdown: It includes signal traps (trap) to ensure that if you hit Ctrl+C or send a termination signal, it tries to clean up running processes properly instead of just leaving them hanging.
How to Use It
- Save the code: Save the script above as start_observer.sh in the same directory as your octez-evm-node binary.
- Make it executable: chmod +x start_observer.sh
- Run it: ./start_observer.sh
This will start your observer node in the background. You can monitor its overall status with tail -f observer_daemon.log and the detailed EVM node output with tail -f evm_node.log.
How do we stop it?
Gracefully Stopping Your Observer: The stop_observer.sh Script
Just as important as starting your node is knowing how to stop it cleanly. You don't want to leave orphaned processes running or risk data corruption by abruptly killing things. That's where the stop_observer.sh script comes in!
This script is designed to safely identify and terminate all related processes (both the daemon script itself and the octez-evm-node process it manages). I've checked this script as well, and like its counterpart, it does not contain any sensitive environment variables or secrets. It's all about managing local processes and files.
Here's the code I use to stop the observer:
#!/bin/bash
# This script is designed to gracefully stop all running Etherlink Observer
# processes that were started by the 'start_observer.sh' script.
# It handles both the daemon wrapper and the actual EVM node processes.
# --- Configuration ---
# These specify the files used by the start script to track processes.
PID_FILE="evm_node.pid" # PID file for the 30-day history EVM node.
PID_FILE_365D="evm_node_365d.pid" # Placeholder for a potential 365-day history EVM node (if you run one).
DAEMON_LOG_FILE="observer_daemon.log" # Log file for the start_observer daemon.
echo "Stopping Etherlink Observer (both 30-day and 365-day versions if running)..."
# --- Helper Function ---
# This function is crucial for stopping not just a process, but any child processes it might have spawned.
kill_process_tree() {
local pid=$1 # The parent PID to kill.
local signal=${2:-TERM} # The signal to send (default is TERM for graceful shutdown).
if kill -0 "$pid" 2>/dev/null; then # Check if the PID still exists.
echo "Killing process $pid with signal $signal"
# First, try to kill all child processes associated with this parent.
pkill -P "$pid" 2>/dev/null
# Then, kill the main parent process itself.
kill -"$signal" "$pid" 2>/dev/null
return 0 # Success.
else
echo "Process $pid not running"
return 1 # Process not found.
fi
}
# --- Stop Daemon Processes ---
# This part finds and stops the 'start_observer.sh' scripts running in the background.
daemon_pids=$(pgrep -f "start_observer.*\.sh") # Find PIDs of all running start_observer scripts.
if [[ -n "$daemon_pids" ]]; then
echo "Found daemon processes: $daemon_pids"
for pid in $daemon_pids; do
# Make sure we don't try to kill the current stop script itself!
if [[ "$pid" != "$$" ]]; then
kill_process_tree "$pid" # Attempt graceful kill.
fi
done
sleep 2
# If any daemons are still running after a grace period, force kill them.
for pid in $daemon_pids; do
if [[ "$pid" != "$$" ]] && kill -0 "$pid" 2>/dev/null; then
echo "Force killing daemon process $pid"
kill_process_tree "$pid" KILL # Force kill.
fi
done
else
echo "No daemon processes found"
fi
# --- Stop EVM Node Processes ---
# This section uses the PID files to find and stop the actual 'octez-evm-node' processes.
for pid_file in "$PID_FILE" "$PID_FILE_365D"; do
if [[ -f "$pid_file" ]]; then
evm_pid=$(cat "$pid_file")
echo "Found EVM node PID in $pid_file: $evm_pid"
if kill_process_tree "$evm_pid"; then # Attempt graceful kill.
sleep 2
# If still running, force kill.
if kill -0 "$evm_pid" 2>/dev/null; then
echo "Force killing EVM node process $evm_pid"
kill_process_tree "$evm_pid" KILL # Force kill.
fi
fi
rm -f "$pid_file" # Remove the PID file after stopping.
else
echo "No PID file found: $pid_file"
fi
done
# --- Final Cleanup: Kill any lingering octez-evm-node processes ---
# This acts as a safety net, catching any 'octez-evm-node' processes that might not have been
# caught by the PID file method (e.g., if a PID file was corrupted or missed).
remaining_pids=$(pgrep -f "octez-evm-node")
if [[ -n "$remaining_pids" ]]; then
echo "Found remaining octez-evm-node processes: $remaining_pids"
for pid in $remaining_pids; do
kill_process_tree "$pid" # Attempt graceful kill.
done
sleep 2
# If still running, force kill.
for pid in $remaining_pids; do
if kill -0 "$pid" 2>/dev/null; then
echo "Force killing remaining process $pid"
kill_process_tree "$pid" KILL # Force kill.
fi
done
fi
echo "Observer stop complete"
# --- Final Status Check ---
# Provides a summary of whether everything was successfully stopped.
echo ""
echo "Process status:"
daemon_check=$(pgrep -f "start_observer.*\.sh" || echo "none") # Check for daemon scripts.
evm_check=$(pgrep -f "octez-evm-node" || echo "none") # Check for EVM nodes.
echo "Daemon processes: $daemon_check"
echo "EVM node processes: $evm_check"
if [[ "$daemon_check" == "none" && "$evm_check" == "none" ]]; then
echo "✓ All processes stopped successfully"
else
echo "⚠ Some processes may still be running"
fiWhat's Happening in the stop_observer.sh Script?
This script is all about ensuring a clean shutdown:
- Targeted PID Files: It uses the same PID_FILE and PID_FILE_365D (if you're running a separate longer-history node) that the start_observer.sh script creates. This helps it find the specific processes it's responsible for.
- Process Tree Killing: The kill_process_tree function is super important. It doesn't just kill the main process; it first attempts to kill any child processes that the main process might have spawned. This prevents orphaned processes that consume resources.
- Graceful then Forceful: It first tries a TERM signal for a graceful shutdown (allowing the process to clean up after itself). If the process is stubborn and doesn't shut down within a few seconds, it escalates to a KILL signal, which immediately terminates it.
- Daemon and EVM Node Shutdown: It separately looks for and stops both the start_observer.shdaemon scripts and the actual octez-evm-node binaries.
- Safety Net: A final check uses pgrep to find any remaining octez-evm-node processes, just in case some were missed by the PID file method. This ensures a thorough cleanup.
- Status Report: At the end, it gives you a quick summary of whether all processes were successfully stopped, so you're not left guessing.
How to Use the stop_observer.sh Script
- Save the code: Save the script above as stop_observer.sh in the same directory.
- Make it executable: chmod +x stop_observer.sh
- Run it: ./stop_observer.sh
This will attempt to stop all associated Etherlink observer processes and give you a status update.
In the "Configuration" section, you might notice PID_FILE_365D="evm_node_365d.pid". While the primary start_observer.sh script uses rolling:30 days of history, I also experimented with running a separate observer node configured for a much longer history, specifically --history rolling:365. If you, like me, decide to run multiple observer nodes with different history depths to support various querying needs (for example, one for short-term data and another for deep historical analysis), each would typically manage its own PID file. This PID_FILE_365D is a placeholder for such a scenario, ensuring the stop_observer.shscript can gracefully shut down all active observer instances, regardless of their history configuration.
And there you have it! With these two scripts, you have a robust way to manage your own Etherlink RPC node, ensuring it's running smoothly and can be stopped cleanly when needed. This foundational step is crucial for reliable data access, which we'll leverage heavily in our quest to understand active users.
In the next posts, we'll start exploring how we actually use this RPC access to gather the information we need and eventually, how we crunch those numbers to find our active users.
Stay tuned, and happy exploring!
