Menu

Showing posts with label Logging. Show all posts
Showing posts with label Logging. Show all posts

1 Nov 2025

πŸ’‘Shell Scripting for Beginners – Part 8: Debugging & Logging

  • By now, your scripts deploy apps, manage servers, and clean logs.
  • But what happens when things go wrong?
  • Welcome to Debugging & Logging, where we make your scripts smarter and traceable like a pro DevOps engineer! πŸ•΅️‍♂️

πŸ“‘ Table of Contents


🐞 1️⃣ Debugging with set -x

set -x shows every command the script executes — perfect for troubleshooting silent errors.

#!/bin/bash
set -x  # enable debugging
echo "Starting deployment..."
mkdir /opt/app/deploy
cp /tmp/app.war /opt/app/deploy/
set +x  # disable debugging
echo "Deployment finished!"
πŸ’¬ Output:
+ echo "Starting deployment..."
Starting deployment...
+ mkdir /opt/app/deploy
+ cp /tmp/app.war /opt/app/deploy/
+ echo "Deployment finished!"
Deployment finished!
πŸ’‘ Use bash -x script.sh to debug a script without editing it.

πŸͺ€ 2️⃣ Using trap to Handle Failures

The trap command catches unexpected exits (Ctrl+C, errors) and executes custom recovery code. Perfect for cleanup or service restarts.

#!/bin/bash
trap 'echo "⚠️ Script interrupted! Cleaning up..."; rm -f /tmp/tempfile' EXIT
echo "Creating temporary file..."
touch /tmp/tempfile
echo "Simulating work..."
sleep 3
exit 1  # simulate failure
πŸ’¬ Output:
Creating temporary file...
Simulating work...
⚠️ Script interrupted! Cleaning up...
πŸ’‘ Always trap EXIT or ERR to prevent orphan temp files or hanging processes.

πŸšͺ 3️⃣ Exit Codes & Error Messages

Every command returns an exit code — 0 means success, non-zero means failure.

#!/bin/bash
cp /tmp/app.war /opt/app/deploy/
if [ $? -ne 0 ]; then
  echo "❌ Copy failed! Check file path or permissions."
  exit 1
else
  echo "✅ Copy successful!"
fi
πŸ’¬ Output:
❌ Copy failed! Check file path or permissions.
or
✅ Copy successful!
Use echo $? after any command to view its exit code.

🧾 4️⃣ Logging with Syslog & Custom Files

Example 1: Using logger Command

#!/bin/bash
logger "Starting Middleware backup..."
tar -czf /tmp/backup.tar.gz /opt/middleware/ || logger "Backup failed!"
logger "Middleware backup completed!"
πŸ’¬ Syslog Output (in /var/log/syslog):
Nov 1 02:15:00 server logger: Starting Middleware backup...
Nov 1 02:15:03 server logger: Middleware backup completed!

Example 2: Redirecting Logs to a File

#!/bin/bash
exec >> /opt/logs/deploy.log 2>&1
echo "[$(date)] Starting deployment..."
echo "[$(date)] Copying WAR..."
cp /tmp/app.war /opt/tomcat/webapps/
echo "[$(date)] Deployment done ✅"
πŸ’¬ deploy.log Content:
[Sat Nov 1 02:20:00 2025] Starting deployment...
[Sat Nov 1 02:20:01 2025] Copying WAR...
[Sat Nov 1 02:20:02 2025] Deployment done ✅

⚙️ 5️⃣ Real Middleware & DevOps Examples

Example 1: Jenkins Job Failure Recovery

#!/bin/bash
set -e
trap 'echo "🚨 Jenkins build failed, restarting service..."; systemctl restart jenkins' ERR
curl -s -X POST "http://localhost:8080/job/myjob/build" || false
echo "✅ Jenkins job triggered successfully!"
πŸ’¬ Output:
🚨 Jenkins build failed, restarting service...
✅ Jenkins job triggered successfully!

Example 2: Tomcat Log Watcher with Debug Info

#!/bin/bash
set -x
tail -Fn0 /opt/tomcat/logs/catalina.out | \
while read line; do
  echo "$line" | grep "ERROR" &>/dev/null && echo "πŸ”₯ Found error: $line" >> /opt/logs/tomcat_error.log
done
πŸ’¬ Output:
[DEBUG] + grep "ERROR" catalina.out
πŸ”₯ Found error: Database connection timeout

Example 3: NGINX Health Check with Logging

#!/bin/bash
while true; do
  if ! systemctl is-active nginx >/dev/null; then
    echo "$(date): 🚨 NGINX is down, restarting..." >> /var/log/nginx_monitor.log
    systemctl restart nginx
  fi
  sleep 10
done
πŸ’¬ Log Output:
Sat Nov 1 03:00:00 2025: 🚨 NGINX is down, restarting...

🏁 Summary

  • set -x → Debug command execution flow.
  • trap → Catch exits & clean up automatically.
  • exit codes → Detect and handle failures gracefully.
  • logger → Send messages to syslog or custom logs.
  • πŸ’‘ Always include logging in production-ready scripts for audit and rollback.

πŸ“ Shell Scripting for Beginners – Part 7: File Handling in Shell.

  • In Middleware & DevOps automation, file handling is everywhere — reading logs, writing reports, rotating backups, or watching live output.
  • Let's learn how to read, write, append, and monitor files in shell scripts with real-world Middleware examples.

πŸ“‘ Table of Contents


πŸ“– 1️⃣ Reading Files

Example 1: Read File Line by Line

#!/bin/bash
filename="/opt/scripts/serverlist.txt"
while read -r line; do
  echo "Processing server: $line"
done < "$filename"
πŸ’¬ Output:
Processing server: app1
Processing server: app2
Processing server: app3

Example 2: Display Tomcat Log Errors

#!/bin/bash
grep "ERROR" /opt/tomcat/logs/catalina.out
πŸ’¬ Output:
ERROR [localhost-startStop-1] ... Application failed to start
ERROR [main] ... Database connection refused
πŸ’‘ Tip: Use grep -i error to ignore case.

✍️ 2️⃣ Writing & Appending Files

Example 3: Write Output to File

#!/bin/bash
echo "Backup started at $(date)" > /opt/scripts/backup.log
echo "Compressing files..." >> /opt/scripts/backup.log
tar -czf /tmp/backup.tar.gz /opt/data && echo "Backup successful" >> /opt/scripts/backup.log
πŸ’¬ Output File Content:
Backup started at Sat Nov 1 02:00:00 IST 2025
Compressing files...
Backup successful

Example 4: Append Deployment Status

#!/bin/bash
env=$1
status=$2
echo "$(date) - $env deployment $status" >> /opt/scripts/deploy_history.log
πŸ’¬ Output File Content:
Sat Nov 1 02:15:00 IST 2025 - dev deployment SUCCESS
Sat Nov 1 02:17:00 IST 2025 - prod deployment FAILED

Example 5: Log WebSphere Cleanup Activity

#!/bin/bash
logfile="/opt/IBM/WebSphere/AppServer/logs/cleanup.log"
echo "$(date): Cleaning old logs..." >> $logfile
find /opt/IBM/WebSphere/AppServer/logs -type f -mtime +7 -delete && echo "✅ Logs cleaned" >> $logfile
πŸ’¬ Output File:
Sat Nov 1 02:30:00 IST 2025: Cleaning old logs...
✅ Logs cleaned

πŸ” 3️⃣ Monitoring Log Files

Example 6: Live Tail of Tomcat Logs

#!/bin/bash
echo "Monitoring Tomcat logs..."
tail -f /opt/tomcat/logs/catalina.out
πŸ’¬ Output:
Monitoring Tomcat logs...
[INFO] Server startup completed in 2200 ms
[WARN] Deprecated API in use
[ERROR] Connection timeout to DB

Example 7: Auto-restart Jenkins on Log Failure

#!/bin/bash
tail -Fn0 /var/log/jenkins/jenkins.log | \
while read line; do
  echo "$line" | grep "ERROR" &>/dev/null
  if [ $? = 0 ]; then
    echo "🚨 Error detected, restarting Jenkins..."
    systemctl restart jenkins
  fi
done
πŸ’¬ Output:
[ERROR] Jenkins job queue stuck
🚨 Error detected, restarting Jenkins...

Example 8: Monitor NGINX Access Logs for High Load

#!/bin/bash
tail -n 100 /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c
πŸ’¬ Output:
500 3
404 7
200 90
301 5
πŸ’‘ Tip: Use this for quick traffic analysis on your Middleware or DevOps pipeline servers.

⚠️ 4️⃣ Error Handling & Debugging

  • Use set -e to exit on first failure.
  • Redirect errors → 2>> error.log to capture exceptions.
  • Validate file existence before reading:
if [ ! -f "$filename" ]; then
  echo "❌ File not found: $filename"
  exit 1
fi
If your script suddenly exits, check for missing files or permission issues in /var/log/messages or your custom error log.

🏁 Summary

  • Read files line by line to automate configuration or host processing.
  • Write and append logs for auditing and historical tracking.
  • Monitor live logs (Tomcat, Jenkins, NGINX) using tail -f.
  • Add error handling to avoid silent file issues in production.