Common Issues¶

This guide covers the most frequently encountered issues when deploying and managing Waveshift with wavectl.

Deployment Issues¶

AWS Permission Errors¶

Problem: Access denied or UnauthorizedOperation errors during deployment

Symptoms:

Error: failed to create VPC: UnauthorizedOperation: You are not authorized to perform this operation
Error: failed to create IAM role: AccessDenied: User is not authorized to perform: iam:CreateRole

Solutions:

Verify AWS Credentials:
```
aws sts get-caller-identity
```
Check IAM Permissions: Ensure your AWS user/role has the required permissions:
EC2 Full Access
IAM Full Access
Route53 Full Access
S3 Full Access
SSM Full Access
VPC Full Access

Use Administrator Access (for initial setup):

aws iam attach-user-policy --user-name your-username --policy-arn arn:aws:iam::aws:policy/AdministratorAccess

Docker Image Access Issues¶

Problem: Cannot access Docker images during deployment

Symptoms:

Error: failed to start containers: access denied for Docker registry

Solutions:

Verify AWS Account Authorisation:
```
aws sts get-caller-identity
```
Ensure your AWS account has been pre-authorized by Blackfire Technology.
Check Identity Provider: Verify you're using the correct identity provider string provided by Blackfire Technology.
Contact Support: If issues persist, contact Blackfire Technology support with your AWS account ID.

Instance Launch Failures¶

Problem: EC2 instances fail to launch

Symptoms:

Error: failed to launch instance: InsufficientInstanceCapacity
Error: failed to launch instance: InstanceLimitExceeded

Solutions:

Check Service Limits:

aws service-quotas get-service-quota --service-code ec2 --quota-code L-1216C47A

Try Different Instance Types:

wavectl deploy --kasm-instance-type m6a.xlarge --region us-east-1 --identity-provider your-provider

Use Different Availability Zone:

aws ec2 describe-availability-zones --region us-east-1

Health Check Issues¶

Services Not Ready¶

Problem: Services are not ready after deployment

Symptoms: - wavectl sites command fails or times out - Cannot access Waveshift console - Services starting up

Solutions:

Wait for Services to Start:
Infrastructure takes 10-15 minutes after wavectl deploy
The wavectl sites --add command includes automatic health checking
Wait the full time before attempting site registration

Check Instance Status:

aws ec2 describe-instances --instance-ids $(cat wavectl.json | jq -r '.waveshift_instance_id')

Verify Security Groups:

aws ec2 describe-security-groups --group-ids $(cat wavectl.json | jq -r '.private_sg_id')

DNS Resolution Issues¶

Problem: Cannot resolve waveshift.internal or kasm.internal

Symptoms:

Error: failed to connect to waveshift.internal: no such host

Solutions:

Check Route53 Private Zone:

aws route53 list-hosted-zones
aws route53 list-resource-record-sets --hosted-zone-id $(cat wavectl.json | jq -r '.hosted_zone_id')

Verify VPC DNS Settings:

aws ec2 describe-vpcs --vpc-ids $(cat wavectl.json | jq -r '.vpc_id')

Test from Within VPC:

# SSH to instance and test DNS
nslookup waveshift.internal

Site Management Issues¶

Site Registration Failures¶

Problem: wavectl sites --add N fails to register sites

Symptoms:

Error: API authentication failed
Error: Cannot connect to orchestration service

Solutions:

Verify Waveshift Connectivity:

wavectl proxy
# Then test: curl http://localhost:8080/health

Check API Key:
```
cat wavectl.json | jq '.api_key'
```

Test API Connectivity:

curl -k https://waveshift.internal/health

WireGuard Key Generation Issues¶

Problem: Sites created without proper WireGuard keys

Symptoms:

Error: failed to generate WireGuard keys
Site created but missing wg_private_key

Solutions:

Check System Entropy:

cat /proc/sys/kernel/random/entropy_avail

Install WireGuard Tools:

# Ubuntu/Debian
sudo apt install wireguard-tools

# macOS
brew install wireguard-tools

Retry Site Creation:
```
wavectl sites --add 1
```

Router Configuration Issues¶

SSH Connection Failures¶

Problem: Cannot connect to GL.iNet router via SSH

Symptoms:

Error: failed to connect to router: connection refused
Error: SSH authentication failed

Solutions:

Verify Router IP:
```
ping 10.34.0.1
nmap -p 22 10.34.0.1
```

Check SSH Service:

# Connect manually to test
ssh root@10.34.0.1

Try Different Credentials:

# Use custom SSH credentials
wavectl configure-hub --site 0 --ssh-user root --ssh-password your-custom-password

Router Configuration Failures¶

Problem: Router configuration script fails

Symptoms:

Error: failed to update DHCP configuration
Error: WireGuard setup failed
Error: device scanner installation failed

Solutions:

Check Router Model Compatibility:

ssh root@10.34.0.1 "cat /etc/openwrt_release"

Verify WireGuard Support:

ssh root@10.34.0.1 "opkg list-installed | grep wireguard"

Manual Configuration Check:

ssh root@10.34.0.1 "uci show network"
ssh root@10.34.0.1 "uci show dhcp"

Router Reboot Issues¶

Problem: Router doesn't come back online after configuration

Symptoms:

Router configured but not responding after reboot
Cannot ping router after configuration

Solutions:

Wait for Reboot Completion:

# Wait 2-3 minutes for full reboot
sleep 180
ping 10.34.0.1

Check Network Configuration:

# Router may have changed IP after configuration
nmap -sn 10.8.0.0/24

Physical Reset if Needed:
Press and hold reset button for 10 seconds
Reconfigure with original settings

Network Connectivity Issues¶

VPN Connection Problems¶

Problem: Cannot establish WireGuard VPN connection

Symptoms:

WireGuard tunnel not connecting
Cannot reach internal services through VPN

Solutions:

Check WireGuard Configuration:
```
wg show
```

Verify Endpoint Connectivity:

# Test UDP connectivity to WireGuard port
nc -u waveshift-public-ip 51820

Check Firewall Rules:

# Ensure UDP 51820 is allowed
sudo ufw status

Internal Service Access Issues¶

Problem: Cannot access Waveshift or Kasm services

Symptoms:

Connection timeout to waveshift.internal
Cannot load Kasm interface

Solutions:

Verify VPN Connection:

ping 10.0.1.10  # Waveshift internal IP

Test Direct IP Access:

curl -k https://$(cat wavectl.json | jq -r '.waveshift_private_ip')

Configuration File Issues¶

Corrupted Configuration¶

Problem: wavectl.json file is corrupted or missing data

Symptoms:

Error: failed to parse config
Error: required field missing

Solutions:

Validate JSON Syntax:
```
cat wavectl.json | jq '.'
```
Check RequiredFields:
```
cat wavectl.json | jq 'keys'
```
Restore from Backup:
```
cp wavectl.json.backup wavectl.json
```

Multiple Configuration Files¶

Problem: Managing multiple environments with different configs

Solution:

Environment-Specific Deployments: - Use separate directories for each environment - Each directory has its own wavectl.json

# Production directory
cd ~/waveshift-production
wavectl sites --add 5

# Staging directory
cd ~/waveshift-staging
wavectl sites --add 2

Performance Issues¶

Slow Deployment Times¶

Problem: Deployment takes longer than expected

Solutions:

Use Larger Instance Types:

wavectl deploy --kasm-instance-type m6a.4xlarge --region us-east-1 --identity-provider your-provider

Choose Optimal Region:

# Use region closest to your location
wavectl deploy --region us-west-2 --identity-provider your-provider

Monitor AWS Service Health:
Check AWS Service Health Dashboard
Try different availability zones

Slow Service Startup¶

Problem: Services take too long to become healthy

Solutions:

Increase Health Check Timeout:

wavectl health-check --wait --timeout 900

Check Instance Resources:

# Monitor CPU and memory usage
aws cloudwatch get-metric-statistics --namespace AWS/EC2 --metric-name CPUUtilisation

Hardware Setup Issues¶

Nodes Not Appearing¶

Problem: Nodes don't show up in Unassigned Nodes list

Solutions:

Check Power:
Verify node power LED is on
Try different power adapter (minimum 3A USB-C required)

Check Cabling:

# Node must have TWO cables:
# - LAN2 to Control Hub (management)
# - WAN to Bearer Router (internet)

Check Control Hub Connectivity:
Verify Control Hub can reach internet
From tablet browser: http://10.1.0.1 should load
Wait for Registration:
Nodes can take up to 2 minutes to register
Refresh Nodes page in console
Check Node Firmware:
Nodes must have Waveshift firmware
Pre-flashed in demo kit

Cannot Access Console¶

Problem: http://10.1.0.1 doesn't load

Solutions:

Check Tablet Connection:
USB-Ethernet adapter properly connected
Ethernet cable from adapter to Control Hub LAN
Check tablet network settings (should auto-configure)
Verify Tablet IP:
Tablet should have IP in 10.1.0.x range
Check Network Settings on tablet
Restart Control Hub:
Unplug power from Control Hub
Wait 10 seconds
Reconnect power
Wait 1 minute for boot
Try Different LAN Port:
Control Hub has multiple LAN ports
Try connecting tablet to different port

No Internet on Nodes¶

Problem: Nodes online but PoPs fail to build

Solutions:

Verify Bearer Internet:
Connect laptop to Bearer
Test: ping 8.8.8.8
Check Node WAN Connection:
Node WAN port must connect to Bearer LAN
Verify cable is seated properly
Check Bearer DHCP:
Bearer must assign IPs to node WAN ports
Check Bearer admin panel for connected devices

Debugging Commands¶

Comprehensive System Check¶

#!/bin/bash
# Complete Waveshift system diagnostic

echo "=== AWS Connectivity ==="
aws sts get-caller-identity

echo "=== Configuration File ==="
cat wavectl.json | jq '.'

echo "=== EC2 Instance Status ==="
aws ec2 describe-instances --instance-ids \
  $(cat wavectl.json | jq -r '.waveshift_instance_id') \
  $(cat wavectl.json | jq -r '.kasm_instance_id')

echo "=== DNS Resolution ==="
nslookup waveshift.internal
nslookup kasm.internal

echo "=== VPC Configuration ==="
aws ec2 describe-vpcs --vpc-ids $(cat wavectl.json | jq -r '.vpc_id')

echo "=== Security Groups ==="
aws ec2 describe-security-groups --filters "Name=vpc-id,Values=$(cat wavectl.json | jq -r '.vpc_id')"

echo "=== Route53 Records ==="
aws route53 list-resource-record-sets --hosted-zone-id $(cat wavectl.json | jq -r '.hosted_zone_id')

### Router Diagnostic

```bash
{{ ... }}
# GL.iNet router diagnostic script

ROUTER_IP="10.34.0.1"

echo "=== Router Connectivity ==="
ping -c 3 $ROUTER_IP

echo "=== SSH Test ==="
ssh -o ConnectTimeout=5 root@$ROUTER_IP "echo 'SSH OK'"

echo "=== Router Info ==="
ssh root@$ROUTER_IP "cat /etc/openwrt_release"

echo "=== Network Configuration ==="
ssh root@$ROUTER_IP "uci show network | grep -E '(interface|proto|ipaddr)'"

echo "=== WireGuard Status ==="
ssh root@$ROUTER_IP "wg show"

echo "=== DHCP Configuration ==="
ssh root@$ROUTER_IP "uci show dhcp"

echo "=== Device Scanner ==="
ssh root@$ROUTER_IP "crontab -l | grep device"

Getting Help¶

Log Collection¶

When reporting issues, collect these logs:

wavectl Output:

wavectl deploy --region us-east-1 --identity-provider your-provider > deployment.log 2>&1

AWS CloudTrail Events:

aws logs describe-log-groups --log-group-name-prefix /aws/

Instance Logs:

# Via SSH to instances
sudo journalctl -u waveshift.service
sudo journalctl -u kasm.service

Support Information¶

Include this information when seeking support:

wavectl version info
AWS region and account ID
Configuration file (sanitized)
Error messages and logs
Steps to reproduce the issue

For additional help, check the CLI Reference or review the Workflow Examples.