By Hafiz Ali | Linux System Administrator with 8+ years experience managing Ubuntu servers and VPN infrastructure. Certified RHCE and Ubuntu Server Specialist
🕒 Last updated: December 2024 | Tested on Ubuntu 22.04 LTS and 24.04 LTS
`Service startup failures can cripple your Ubuntu server’s functionality. This comprehensive guide covers systemd service troubleshooting from basic diagnostics to advanced recovery techniques, helping you resolve startup issues quickly and prevent future problems.
🚨 Quick Service Status Overview
| Service State | Meaning | Common Causes | Fix Priority |
|---|---|---|---|
| 🟢 Active (running) | Service is working | N/A | ✅ Good |
| 🔴 Failed | Service crashed | Configuration errors | 🚨 High |
| 🟡 Inactive (dead) | Service stopped | Not started yet | 🟡 Medium |
| 🟣 Activating | Starting up | Slow initialization | 🟡 Monitor |
| ⚫ Not-found | Service missing | Not installed | 🔧 Required |
🔍 Initial Service Diagnostics
Start with these basic commands to understand your service’s current state:
# Check all service statuses
sudo systemctl status
# Check specific service (e.g., SSH)
sudo systemctl status ssh
# List all failed services
sudo systemctl --failed
# Check service dependencies
systemctl list-dependencies ssh
# Verify service unit file location
systemctl cat ssh🛠️ Common Service Issues & Solutions
❌ Issue 1: Service Failed to Start
🔍 Error Examples:
Job for nginx.service failed because the control process exited with error code.
See "systemctl status nginx.service" and "journalctl -xe" for details.🛠️ Solution Steps:
# Step 1: Check detailed service status
sudo systemctl status service-name -l
# Step 2: View service logs
sudo journalctl -u service-name -f
# Step 3: Check for configuration errors
sudo systemctl edit service-name --full
# Step 4: Test manual startup
sudo /usr/sbin/service-name --test-config
# Step 5: Reset service if needed
sudo systemctl reset-failed service-name⏱️ Issue 2: Service Timeout During Startup
🔍 Error Examples:
nginx.service: Start operation timed out. Terminating.
Timeout was set to 90s.🛠️ Solution Steps:
# Increase service timeout
sudo systemctl edit service-name
# Add these lines:
[Service]
TimeoutStartSec=300
# Reload and restart
sudo systemctl daemon-reload
sudo systemctl restart service-name
# Alternatively, check what's blocking startup
sudo strace -p $(pgrep service-name)🔗 Issue 3: Dependency Problems
🔍 Error Examples:
nginx.service: Failed to set up special execution directory: No such file or directory
Dependency failed for Nginx HTTP Server.🛠️ Solution Steps:
# Check service dependencies
systemctl list-dependencies service-name
# Check dependency order
systemctl list-dependencies service-name --reverse
# Verify required services are running
sudo systemctl status network.target
sudo systemctl status basic.target
# Check for missing directories/files
sudo journalctl -u service-name | grep -i "no such\|failed\|error"📋 Service Management Commands Reference
🎯 Basic Service Controls
# Start a service
sudo systemctl start service-name
# Stop a service
sudo systemctl stop service-name
# Restart a service
sudo systemctl restart service-name
# Reload configuration
sudo systemctl reload service-name
# Enable auto-start on boot
sudo systemctl enable service-name
# Disable auto-start
sudo systemctl disable service-name
# Check if service is enabled
systemctl is-enabled service-name📊 Advanced Service Monitoring
# Follow service logs in real-time
sudo journalctl -u service-name -f
# Show service logs from boot
sudo journalctl -u service-name -b
# Check service resource usage
systemd-cgtop
# Analyze service startup time
systemd-analyze critical-chain service-name
# Plot service startup timeline
systemd-analyze plot > boot-analysis.svg🔧 Specific Service Troubleshooting
🌐 Web Server Issues (NGINX/Apache)
# Test NGINX configuration
sudo nginx -t
# Check for port conflicts
sudo netstat -tulpn | grep :80
# Verify virtual host configurations
sudo apache2ctl configtest # For Apache
# Check web server error logs
sudo tail -f /var/log/nginx/error.log
sudo tail -f /var/log/apache2/error.log🛡️ VPN Service Issues
# For OpenVPN (from our previous guides)
sudo systemctl status openvpn@server
sudo journalctl -u openvpn@server -f
# For WireGuard
sudo systemctl status wg-quick@wg0
sudo journalctl -u wg-quick@wg0 -f
# Check VPN interface status
ip link show wg0
ip link show tun0
# Verify routing tables
ip route show🗄️ Database Service Issues (MySQL/PostgreSQL)
# Check database service status
sudo systemctl status mysql
sudo systemctl status postgresql
# View database logs
sudo tail -f /var/log/mysql/error.log
sudo journalctl -u postgresql -f
# Check disk space for databases
df -h /var/lib/mysql
df -h /var/lib/postgresql
# Verify database connectivity
mysql -u root -p -e "SELECT 1;"
psql -U postgres -c "SELECT 1;"🚨 Emergency Recovery Procedures
🔓 Rescue Mode & Emergency Boot
# Boot into rescue mode from GRUB
# Add "systemd.unit=rescue.target" to kernel line
# Emergency mode for critical failures
# Add "systemd.unit=emergency.target" to kernel line
# Single user mode (runlevel 1)
# Add "single" to kernel line🔄 Systemd Rescue Commands
# Emergency restart of systemd
sudo systemctl daemon-reload
sudo systemctl reset-failed
# Check for corrupted units
systemctl --state=failed
systemctl list-unit-files | grep bad
# Revert service to vendor version
sudo systemctl revert service-name
# Mask a problematic service (prevent all starts)
sudo systemctl mask service-name
# Unmask a service
sudo systemctl unmask service-name📝 Service Configuration Management
⚙️ Creating Custom Service Files
# Create a custom service
sudo nano /etc/systemd/system/custom-service.serviceExample service file:
[Unit]
Description=My Custom Service
After=network.target
[Service]
Type=simple
User=myuser
ExecStart=/usr/local/bin/my-service
Restart=on-failure
RestartSec=5s
[Install]
WantedBy=multi-user.target🔧 Service Override Files
# Create service overrides without modifying original
sudo systemctl edit service-name
# This creates: /etc/systemd/system/service-name.service.d/override.conf
# Example override to increase memory limits:
[Service]
MemoryMax=2G
CPUQuota=150%🔍 Advanced Diagnostics Tools
📊 Systemd Analysis Commands
# Analyze boot performance
systemd-analyze time
systemd-analyze blame
systemd-analyze critical-chain
# Check for failed units
systemctl list-units --state=failed
# Verify unit file syntax
systemd-analyze verify /etc/systemd/system/service-name.service
# Check service security settings
systemd-analyze security service-name🐛 Debugging with Strace & Ltrace
# Trace system calls during service start
sudo strace -f -o /tmp/service-trace.log systemctl start service-name
# Trace library calls
sudo ltrace -f -o /tmp/service-ltrace.log systemctl start service-name
# Analyze file access during startup
sudo strace -e trace=file systemctl start service-name🛡️ Prevention & Best Practices
✅ Service Health Monitoring
# Set up service monitoring script
#!/bin/bash
SERVICES=("nginx" "mysql" "ssh")
for service in "${SERVICES[@]}"; do
if ! systemctl is-active --quiet $service; then
echo "ALERT: $service is down"
systemctl restart $service
fi
done
# Add to crontab for regular checks
# */5 * * * * /usr/local/bin/service-monitor.sh📋 Startup Optimization
# Parallelize service startup
sudo systemctl enable systemd-udev-trigger.service
# Disable unnecessary services
sudo systemctl disable bluetooth.service
sudo systemctl disable cups.service # If no printer
# Optimize service dependencies
systemd-analyze critical-chain❓ Frequently Asked Questions
🔧 How do I find why a service failed?
Use this diagnostic sequence:
sudo systemctl status service-name -l
sudo journalctl -u service-name -f --since "1 hour ago"
sudo systemctl cat service-name
systemctl list-dependencies service-name⚠️ What’s the difference between restart and reload?
Restart: Stops and starts the service (downtime)
Reload: Reloads configuration without stopping service (no downtime)
Use reload when possible to maintain service availability.
🔄 My service keeps crashing. How to auto-restart?
# Edit service to add auto-restart
sudo systemctl edit service-name
# Add these lines:
[Service]
Restart=on-failure
RestartSec=5s
StartLimitInterval=100
StartLimitBurst=5🔗 Related Ubuntu Guides
- 🌐 Ubuntu Network Configuration Guide
- 🔧 Common APT Errors and Fixes
- ⚡ OpenVPN Server Setup Guide
- 🛠️ Browse All Ubuntu Troubleshooting Guides
🔄 Master Ubuntu Service Management
Our complete Ubuntu Troubleshooting category has everything you need for enterprise-grade server administration.