MiddlewareBox: Jenkins

Showing posts with label Jenkins. Show all posts

2 Nov 2025

🐧 Linux Shell Scripting for Beginners – Complete Tutorial Series.

A 13-part practical shell scripting course for DevOps and Middleware Engineers.

💡 How This Tutorial Series Will Help You

✔ Build a strong foundation in Linux Shell scripting — from basics to automation.
✔ Automate daily Middleware and DevOps tasks: monitoring, backups, and log management.
✔ Write scripts for Middleware platforms like Tomcat, Jenkins, and WebSphere.
✔ Debug, schedule, and manage production-ready shell scripts confidently.
✔ Perfect for Sysadmins, Middleware, and DevOps Engineers upgrading to Cloud roles.

📚 Complete Course Index

🐚 Part 1: What is a Variable?

Learn how variables store data and simplify automation scripts. Foundation of all shell logic.

📥 Part 2: Reading User Input in Shell

Make interactive scripts using read command with Jenkins and Tomcat examples.

🧩 Part 3: Conditional Statements (if, elif, else)

Use decision-making in your scripts with examples checking files, services, and network health.

🔁 Part 4: Loops in Shell (for, while, until)

Automate repetitive DevOps tasks like log cleanup, backups, and status monitoring.

⚙️ Part 5: Functions in Shell (Reusable Scripts)

Write modular, reusable functions — restart Tomcat or back up Jenkins in one click.

📦 Part 6: Arrays & Arguments

Store multiple values, manage user inputs, and process server lists in scripts.

📁 Part 7: File Handling in Shell

Learn how to read, write, append, and handle configuration or log files safely.

💡 Part 8: Debugging & Logging

Find and fix script errors, use logs, and enable debug mode for safe execution.

🕒 Part 9: Scheduling & Automation

Use cron and @reboot jobs to run scripts automatically for health checks or backups.

🎯 Part 10: Middleware Health Monitoring Project

Create a production-ready monitoring script for Tomcat, Jenkins, and NGINX with auto-restart.

🧾 Part 11: Log Management & Rotation

Automate log rotation with logrotate and custom retention scripts for Middleware systems.

💾 Part 12: Backup & Restore Automation Project

Perform automated backup and restore for Jenkins, Tomcat, and WebSphere with rollback support.

⚙️ Part 13: Subshells, Process Control & Parallel Execution

Run multiple jobs in parallel, monitor background processes, and speed up automation workflows.

🚀 Why MiddlewareBox Shell Scripting Series?

✅ Step-by-step, beginner-friendly explanations.
✅ Each topic includes real DevOps & Middleware context.
✅ Examples with Output and Error handling.
✅ Simple language for better understanding.

✨ Start Learning Now → Part 1: What is a Variable?
& build your own scripts from scratch! 💪

Pradeep V. (VK)

My Middleware journey began long before cloud, containers, or Prometheus dashboards—deep in traditional enterprise servers, late-night incident calls, SSL headaches, and log-hunting. Over the years, I’ve seen the shift from manual, ticket-driven ops to DevOps, containerization, CI/CD, and cloud. I’m here to share practical lessons from that evolution to help accelerate your journey.

⚙️ Shell Scripting for Beginners – Part 13: Subshells, Process Control & Parallel Execution.

Now let's level up and make your shell scripts multitask like a Middleware and DevOps pro.
In this part, you'll learn how to run processes in the background, manage subshells, and execute multiple tasks in parallel for faster automation.

📑 Table of Contents

1️⃣ What is a Subshell?
2️⃣ Running Commands in the Background
3️⃣ Managing Jobs (fg, bg, kill)
4️⃣ Parallel Execution in DevOps
5️⃣ nohup & disown for Long Running Tasks
6️⃣ Real-World Examples (Tomcat, Jenkins, Docker)
🧠 Pro Tips & Troubleshooting
🏁 Summary

1️⃣ What is a Subshell?

A subshell is a new child shell launched from your current shell. It allows running commands in isolation — any variable or environment change inside doesn’t affect the parent shell.

#!/bin/bash
VAR="ParentShell"
( VAR="SubShell"; echo "Inside subshell: $VAR" )
echo "Outside subshell: $VAR"

💬 Output:

Inside subshell: SubShell
Outside subshell: ParentShell

2️⃣ Running Commands in the Background

Add & at the end of a command to run it in the background. This allows your script to keep running while other tasks continue behind the scenes.

#!/bin/bash
echo "Starting Jenkins backup..."
tar -czf /opt/backups/jenkins_backup.tar.gz /var/lib/jenkins & 
echo "Backup started in background (PID: $!)"

💬 Output:

Starting Jenkins backup...
Backup started in background (PID: 2356)

3️⃣ Managing Jobs

When you run background processes, you can manage them using built-in commands:

jobs → List current background jobs
fg %1 → Bring job 1 to foreground
bg %1 → Resume job 1 in background
kill %1 → Kill job 1

#!/bin/bash
long_task() {
  sleep 30 &
}
long_task
jobs

💬 Output:

[1]+ Running sleep 30 &

4️⃣ Parallel Execution in DevOps

Let’s say you want to restart multiple Tomcat servers or back up several apps simultaneously. We can use background jobs with a simple for loop.

#!/bin/bash
servers=("devbox" "testbox" "prodbox")
for srv in "${servers[@]}"; do
  echo "Restarting Tomcat on $srv..."
  ssh $srv "systemctl restart tomcat" &
done

wait  # Wait for all background jobs to finish
echo "✅ All Tomcat servers restarted in parallel!"

💬 Output:

Restarting Tomcat on devbox...
Restarting Tomcat on testbox...
Restarting Tomcat on prodbox...
✅ All Tomcat servers restarted in parallel!

5️⃣ nohup & disown

Sometimes, you need your scripts to survive terminal disconnections — for instance, a long-running Jenkins restore or DB migration.

#!/bin/bash
nohup ./db_migration.sh > /var/log/db_migration.log 2>&1 &
disown
echo "🧠 DB migration running safely in background, even if you close terminal."

💬 Output:

nohup: ignoring input and appending output to 'nohup.out'
🧠 DB migration running safely in background, even if you close terminal.

6️⃣ Real-World DevOps Examples

🧩 Example 1: Parallel Log Collection

#!/bin/bash
servers=("dev" "test" "prod")
for s in "${servers[@]}"; do
  scp $s:/var/log/tomcat/catalina.out /opt/logs/$s.log &
done
wait
echo "✅ Logs collected from all servers."

🧰 Example 2: Monitor Jenkins & Docker in Background

#!/bin/bash
( while true; do pgrep -f jenkins >/dev/null || systemctl restart jenkins; sleep 10; done ) &
( while true; do docker ps | grep -q "webapp" || docker start webapp; sleep 15; done ) &
echo "🩺 Jenkins and Docker monitored in background."

📦 Example 3: Run Multiple CI Jobs Simultaneously

#!/bin/bash
for job in build-test build-stage build-prod; do
  curl -s -X POST "http://jenkins.local/job/$job/build" &
done
wait
echo "✅ All Jenkins build jobs triggered in parallel!"

🧠 Pro Tips & Troubleshooting

🧩 Process Control Best Practices

Use wait to synchronize parallel tasks before continuing.
Log PIDs ($!) to monitor background tasks easily.
Redirect background job output to separate log files using > file 2>&1 &.
For persistent background services, prefer systemd over manual nohup scripts.

⚙️ Debugging Issues

Run ps -ef | grep scriptname to confirm background processes.
Use set -x or bash -x script.sh for detailed command tracing.
Use trap to clean up child processes on script exit.
If parallel SSH commands hang, add -o ConnectTimeout=5 in SSH for safety.

🏁 Summary

✅ Learned to create and manage subshells.
⚙️ Mastered process control using &, jobs, fg, bg, and kill.
🚀 Ran real-world DevOps tasks in parallel — Jenkins builds, Tomcat restarts, Docker checks.
💡 Used nohup and disown for persistent automation tasks.

Pradeep V. (VK)

💾 Shell Scripting for Beginners – Part 12: Backup & Restore Automation Project (DevOps & Middleware Edition)

Here we'll automate backup and restore tasks for real middleware & DevOps systems like Jenkins, Tomcat, WebSphere, MySQL, Docker, and Apache.
Each example includes a working restore script and troubleshooting tips. ⚙️

1️⃣ Jenkins Backup & Restore

#!/bin/bash
JENKINS_HOME="/var/lib/jenkins"
BACKUP_DIR="/opt/backups/jenkins"
mkdir -p $BACKUP_DIR
tar -czf $BACKUP_DIR/jenkins_$(date +%F_%H-%M).tar.gz -C $JENKINS_HOME .
echo "✅ Jenkins backup created in $BACKUP_DIR"

♻️ Restore Jenkins

#!/bin/bash
BACKUP_FILE="/opt/backups/jenkins/jenkins_2025-11-02_02-10.tar.gz"
service jenkins stop
tar -xzf "$BACKUP_FILE" -C /var/lib/jenkins
chown -R jenkins:jenkins /var/lib/jenkins
service jenkins start
echo "♻️ Jenkins restored successfully."

2️⃣ Tomcat Backup & Restore

#!/bin/bash
TOMCAT_HOME="/opt/tomcat"
BACKUP_DIR="/opt/backups/tomcat"
mkdir -p $BACKUP_DIR
tar -czf $BACKUP_DIR/tomcat_$(date +%F).tar.gz $TOMCAT_HOME/webapps $TOMCAT_HOME/conf
echo "📦 Tomcat backup completed."

♻️ Restore Tomcat

#!/bin/bash
BACKUP_FILE="/opt/backups/tomcat/tomcat_2025-11-02.tar.gz"
service tomcat stop
tar -xzf "$BACKUP_FILE" -C /
service tomcat start
echo "♻️ Tomcat restored and restarted."

3️⃣ WebSphere Backup & Restore

#!/bin/bash
WAS_PROFILE="/opt/IBM/WebSphere/AppServer/profiles/AppSrv01"
BACKUP_DIR="/opt/backups/websphere"
mkdir -p $BACKUP_DIR
tar -czf $BACKUP_DIR/was_$(date +%F_%H-%M).tar.gz -C $WAS_PROFILE .
echo "📁 WebSphere configuration backup done."

♻️ Restore WebSphere

#!/bin/bash
BACKUP_FILE="/opt/backups/websphere/was_2025-11-02_03-00.tar.gz"
service was stop
tar -xzf "$BACKUP_FILE" -C /opt/IBM/WebSphere/AppServer/profiles/AppSrv01
service was start
echo "♻️ WebSphere restored successfully."

4️⃣ MySQL Database Backup & Restore

#!/bin/bash
DB="middlewaredb"
USER="root"
PASS="Secret123"
BACKUP_DIR="/opt/backups/mysql"
mkdir -p $BACKUP_DIR
mysqldump -u $USER -p$PASS $DB > $BACKUP_DIR/${DB}_$(date +%F).sql
gzip $BACKUP_DIR/${DB}_$(date +%F).sql
echo "🗄️ MySQL backup completed."

♻️ Restore MySQL

#!/bin/bash
BACKUP_FILE="/opt/backups/mysql/middlewaredb_2025-11-02.sql.gz"
gunzip "$BACKUP_FILE"
mysql -u root -pSecret123 middlewaredb < /opt/backups/mysql/middlewaredb_2025-11-02.sql
echo "✅ Database restored successfully."

5️⃣ Docker Containers Backup & Restore

#!/bin/bash
BACKUP_DIR="/opt/backups/docker"
mkdir -p $BACKUP_DIR
for cid in $(docker ps -q); do
  cname=$(docker inspect --format='{{.Name}}' $cid | cut -d'/' -f2)
  docker export $cid > $BACKUP_DIR/${cname}_$(date +%F).tar
done
echo "🐳 Docker containers exported successfully."

♻️ Restore Docker Container

#!/bin/bash
BACKUP_FILE="/opt/backups/docker/myapp_2025-11-02.tar"
docker import "$BACKUP_FILE" myapp_restored:latest
docker run -d --name myapp_restored myapp_restored:latest
echo "♻️ Docker container restored and running."

6️⃣ Apache Webserver Backup & Restore

#!/bin/bash
BACKUP_DIR="/opt/backups/apache"
mkdir -p $BACKUP_DIR
tar -czf $BACKUP_DIR/apache_$(date +%F).tar.gz /etc/apache2 /var/www/html /etc/ssl
echo "🌐 Apache configuration, website, and SSL backed up."

♻️ Restore Apache

#!/bin/bash
BACKUP_FILE="/opt/backups/apache/apache_2025-11-02.tar.gz"
tar -xzf "$BACKUP_FILE" -C /
service apache2 restart
echo "✅ Apache restored and running."

7️⃣ Automating Backups

Schedule automatic backups using cron so your systems stay protected 24×7.

# Run daily at 2 AM
0 2 * * * /opt/scripts/backup_all.sh >> /var/log/backup_all.log 2>&1

# Run on every reboot
@reboot /opt/scripts/backup_all.sh >> /var/log/backup_boot.log 2>&1

💬 Output:

[CRON] Scheduled backup started…
✅ Jenkins, Tomcat, WebSphere, MySQL, Docker, Apache backed up successfully.

🧠 Pro Tips & Troubleshooting

✅ General Best Practices

Use a dedicated /opt/backups partition or NFS mount to avoid filling system drives.
Compress large backups with gzip or zstd [Developed by Meta] for better space efficiency.
Store daily, weekly, and monthly copies separately (retention policy).
Automate restores in a test environment weekly to ensure recovery works.

⚙️ Troubleshooting Common Issues

❌ Permission Denied: Run backups as sudo or fix ownership using chown.
📦 Disk Full: Use df -h to check space before backup. Add cleanup logic for old files.
🧾 Log Rotation: Add logrotate entry to keep backup logs small and readable.

🏁 Summary

✅ Full backup & restore automation for all major middleware and DevOps tools.
🧩 Scripts are modular, reusable, and easy to integrate in CI/CD pipelines.
⚙️ Cron ensures zero manual intervention for daily protection.

Pradeep V. (VK)

1 Nov 2025

🧾 Shell Scripting for Beginners – Part 11: Log Management & Rotation for Middleware & DevOps Projects

Logs are the heartbeat of your servers 💓 — but if left unmanaged, they'll crash your systems.
In this part, you'll learn how to clean, compress, and rotate logs for Tomcat, Jenkins, Apache, Docker, Kubernetes, and system services using shell scripts and logrotate.

📑 Table of Contents

1️⃣ Log Cleanup Scripts
2️⃣ Apache & Middleware Examples
3️⃣ Docker & K8s Cleanup
4️⃣ logrotate Configuration
5️⃣ How to Run & Test logrotate
🏁 Summary

1️⃣ Log Cleanup Scripts

#!/bin/bash
# Delete logs older than 7 days for Tomcat & Jenkins
find /opt/tomcat/logs/ -type f -name "*.log" -mtime +7 -exec rm -f {} \;
find /var/log/jenkins/ -type f -name "*.log" -mtime +7 -exec rm -f {} \;
echo "🧹 Tomcat & Jenkins logs older than 7 days removed!"

💬 Output:

🧹 Tomcat & Jenkins logs older than 7 days removed!

2️⃣ Apache & Middleware Examples

#!/bin/bash
ARCHIVE="/opt/log_archive/apache"
mkdir -p $ARCHIVE
find /var/log/apache2/ -type f -name "*.log" -mtime +3 -exec gzip {} \;
mv /var/log/apache2/*.gz $ARCHIVE 2>/dev/null
echo "📦 Apache logs compressed and archived in $ARCHIVE"

💬 Output:

📦 Apache logs compressed and archived in /opt/log_archive/apache

3️⃣ Docker & Kubernetes Log Cleanup

#!/bin/bash
echo "🐳 Cleaning Docker & K8s logs..."
docker system prune -af
find /var/log/containers/ -type f -name "*.log" -mtime +5 -exec rm -f {} \;
echo "✅ Docker and Kubernetes logs cleaned!"

💬 Output:

✅ Docker and Kubernetes logs cleaned!

4️⃣ logrotate Configuration

# /etc/logrotate.d/devops-services
/opt/tomcat/logs/*.log
/var/log/apache2/*.log
/var/lib/jenkins/logs/*.log
/var/log/docker/*.log {
    weekly
    rotate 6
    compress
    missingok
    notifempty
    sharedscripts
    postrotate
        systemctl reload apache2 2>/dev/null || true
        systemctl restart tomcat 2>/dev/null || true
    endscript
}

💬 Result:

🔁 Logs for Apache, Tomcat, Jenkins, and Docker will rotate weekly and auto-restart services.

5️⃣ How to Run & Test logrotate

Once your config is ready, you can run logrotate manually or let cron handle it.

🧠 Step 1 – Check Syntax of Your Config

sudo logrotate -d /etc/logrotate.d/devops-services

💬 Output:

Reading config file /etc/logrotate.d/devops-services
Handling /opt/tomcat/logs/*.log
Handling /var/log/apache2/*.log
**dry-run mode (no rotation performed)**

⚙️ Step 2 – Run logrotate Manually

sudo logrotate -f /etc/logrotate.d/devops-services

💬 Output:

Rotating logs for Tomcat, Apache, Jenkins...
Compression complete.
Rotation successful ✅

⏰ Step 3 – Schedule via Cron (Daily or Weekly)

# /etc/cron.daily/logrotate
/usr/sbin/logrotate /etc/logrotate.conf

💡 By default, most Linux systems already run logrotate daily using this cron job. You can verify the last run log at /var/lib/logrotate/status.

🏁 Summary

✅ Cleaned and archived logs for Tomcat, Jenkins, Apache, Docker, and K8s.
✅ Created unified logrotate configuration.
✅ Learned to manually test (-d) and force-run (-f) logrotate.
✅ Automated rotation through cron for 24×7 log hygiene.

Pradeep V. (VK)

🎯Shell Scripting for Beginners – Part 10: Middleware Health Monitoring Project (Mixed Example).

This is our final, beginner-friendly project 🎯 — where we'll combine everything you've learned: variables, if-else, loops, functions, logging, and cron automation.
Let's create a simple Middleware Health Monitoring script that checks Tomcat, Jenkins, Docker, MySQL, and SSL expiry.

📑 Table of Contents

1️⃣ Overview
2️⃣ Full Simple Script
3️⃣ Example Output
4️⃣ Automate the Script

1️⃣ Overview

This script uses:

Variables – store names, logs, and domains.
Functions – reusable checks for each service.
If-Else – to decide if a service needs restart.
For loop – to check multiple Docker containers.
While loop – to retry connection.
Logs – write output to a file.

2️⃣ Full Middleware Health Monitoring Script

#!/bin/bash
# MiddlewareBox - Simple Health Monitor

LOG="/var/log/middleware_health.log"
domain="middlewarebox.com"
echo "===== Middleware Health Check =====" >> $LOG
date >> $LOG

# --- Function to check Tomcat ---
check_tomcat() {
  echo "Checking Tomcat..."
  if pgrep -f "org.apache.catalina.startup.Bootstrap" > /dev/null; then
     echo "✅ Tomcat is running" | tee -a $LOG
  else
     echo "🚨 Tomcat is down, restarting..." | tee -a $LOG
     /opt/tomcat/bin/startup.sh
     echo "✅ Tomcat restarted" | tee -a $LOG
  fi
}

# --- Function to check Jenkins ---
check_jenkins() {
  echo "Checking Jenkins..."
  if systemctl is-active --quiet jenkins; then
     echo "✅ Jenkins is active" | tee -a $LOG
  else
     echo "🚨 Jenkins not running, restarting..." | tee -a $LOG
     systemctl restart jenkins
     echo "✅ Jenkins restarted" | tee -a $LOG
  fi
}

# --- Function to check MySQL ---
check_db() {
  echo "Checking MySQL..."
  if mysql -u root -pSecret -e "show databases;" > /dev/null 2>&1; then
     echo "✅ MySQL connection successful" | tee -a $LOG
  else
     echo "🚨 MySQL connection failed" | tee -a $LOG
     echo "Retrying in 5 seconds..."
     sleep 5
     mysql -u root -pSecret -e "show databases;" && echo "✅ MySQL reconnected" | tee -a $LOG
  fi
}

# --- Check Docker Containers using FOR Loop ---
check_docker() {
  echo "Checking Docker containers..."
  for c in nginx webapp db
  do
    status=$(docker inspect -f '{{.State.Status}}' $c 2>/dev/null)
    if [ "$status" != "running" ]; then
      echo "🚨 Container $c is $status, restarting..." | tee -a $LOG
      docker start $c
    else
      echo "✅ Container $c is running" | tee -a $LOG
    fi
  done
}

# --- Check SSL certificate expiry ---
check_ssl() {
  echo "Checking SSL expiry..."
  expiry=$(echo | openssl s_client -servername $domain -connect $domain:443 2>/dev/null | openssl x509 -noout -dates | grep notAfter | cut -d= -f2)
  days_left=$(( ($(date -d "$expiry" +%s) - $(date +%s)) / 86400 ))
  if [ $days_left -le 15 ]; then
    echo "⚠️ SSL expires in $days_left days!" | tee -a $LOG
  else
    echo "✅ SSL valid for $days_left days" | tee -a $LOG
  fi
}

# --- MAIN EXECUTION FLOW ---
check_tomcat
check_jenkins
check_db
check_docker
check_ssl

echo "✅ Health check completed successfully!" | tee -a $LOG
echo "========================================" >> $LOG

💬 Example Output:

✅ Tomcat is running
✅ Jenkins is active
✅ MySQL connection successful
✅ Container nginx is running
✅ SSL valid for 70 days
✅ Health check completed successfully!

3️⃣ Automate the Script

# Run every 30 minutes
*/30 * * * * /opt/scripts/middleware_health.sh >> /var/log/health_cron.log 2>&1

# Run automatically on system reboot
@reboot /opt/scripts/middleware_health.sh >> /var/log/boot_health.log 2>&1

💬 Output (Cron log):

[BOOT] Health check executed
✅ All services healthy

🏁 Summary

✅ Combined all major shell scripting concepts.
✅ Checks Tomcat, Jenkins, Docker, MySQL, and SSL in one file.
✅ Uses if, for, functions, and basic while retry logic.
✅ Can run manually or via cron / reboot automation.

Pradeep V. (VK)

🕒 Shell Scripting for Beginners – Part 9: Scheduling & Automation

In DevOps and Middleware environments, automation isn't complete until it's scheduled.
Here, we master cron, at, @reboot, and Jenkins integration — running scripts automatically for backups, restarts, monitoring, and cleanups.

📑 Table of Contents

1️⃣ Automating with Cron
2️⃣ One-Time Tasks with At
3️⃣ Auto-Execute on System Reboot
4️⃣ Jenkins Scheduler Integration
5️⃣ Real Middleware Automation Examples
6️⃣ Tips & Troubleshooting
🏁 Summary

⏰ 1️⃣ Automating with Cron

cron executes scripts periodically.
Perfect for recurring jobs like backups, cleanup, and health checks.

Example 1: Daily Jenkins Backup at 3 AM

0 3 * * * /opt/scripts/jenkins_backup.sh >> /var/log/jenkins_backup.log 2>&1

💬 Output (log):

[03:00] Jenkins backup started
[03:03] ✅ Jenkins backup completed

Example 2: Weekly Tomcat Log Cleanup

0 2 * * 0 /opt/scripts/cleanup_tomcat_logs.sh >> /var/log/tomcat_cleanup.log 2>&1

💬 Output:

🧹 Deleted old logs from /opt/tomcat/logs
✅ Weekly cleanup done

🕐 2️⃣ One-Time Tasks with At

at executes a job once — ideal for maintenance windows, patches, or urgent restarts.

Example 3: Restart WebSphere at 11:30 PM

echo "/opt/IBM/WebSphere/AppServer/bin/stopServer.sh server1 && sleep 15 && /opt/IBM/WebSphere/AppServer/bin/startServer.sh server1" | at 23:30

💬 Output:

job 17 at Sat Nov 1 23:30:00 2025

Example 4: Run Backup Once in 2 Hours

echo "/opt/scripts/backup_db.sh" | at now + 2 hours

💬 Output:

job 18 at Sat Nov 1 05:30:00 2025

🔄 3️⃣ Auto-Execute on System Reboot

The @reboot keyword in crontab ensures your scripts run every time Linux boots. Perfect for restarting services, mounting drives, or initializing environments.

Example 5: Auto-Start Tomcat After Reboot

@reboot /opt/tomcat/bin/startup.sh >> /var/log/tomcat_reboot.log 2>&1

💬 Output (after reboot):

✅ Tomcat started automatically at boot.

Example 6: Health Check Script at Boot

@reboot /opt/scripts/health_check.sh >> /var/log/health_boot.log 2>&1

💬 Output (log):

[BOOT] Checking WebSphere, Jenkins, NGINX...
✅ All services active.

💡 To verify reboot jobs, check grep CRON /var/log/syslog after restart.

⚙️ 4️⃣ Jenkins Scheduler Integration

In CI/CD, Jenkins provides cron-like scheduling for build pipelines.

Example 7: Daily Jenkins Build at 2 AM

pipeline {
  triggers { cron('H 2 * * *') }
  stages {
    stage('Nightly Build') {
      steps {
        sh '/opt/scripts/build_app.sh'
      }
    }
  }
}

💬 Output:

✅ Jenkins triggered build_app.sh
Build completed successfully.

🧰 5️⃣ Real Middleware & DevOps Automation Examples

Example 8: NGINX Health Monitor Every 10 Minutes

*/10 * * * * /opt/scripts/nginx_health.sh >> /var/log/nginx_monitor.log 2>&1

💬 Output:

[02:40] 🌐 NGINX is active
[02:50] 🚨 NGINX down – restarted automatically.

Example 9: Database Backup Every 6 Hours

0 */6 * * * mysqldump -u root -pSecret middlewaredb > /backup/db_$(date +%F_%H).sql

💬 Output:

✅ Backup completed: db_2025-11-01_06.sql

Example 10: Restart Jenkins After Crash Detection

*/15 * * * * pgrep jenkins >/dev/null || systemctl restart jenkins

💬 Output:

[03:15] 🚨 Jenkins was down, restarted successfully.

Example 11: Auto-Remount Missing Volume

@reboot mount | grep /mnt/backup >/dev/null || mount /dev/sdb1 /mnt/backup

💬 Output:

✅ /mnt/backup auto-mounted on system startup.

🧠 6️⃣ Tips & Troubleshooting

📄 Check logs → grep CRON /var/log/syslog
🔒 Ensure permissions → chmod +x /opt/scripts/*.sh
⚙️ Test reboot jobs manually → sudo run-parts /etc/cron.d/
📧 Use MAILTO in crontab for notifications
🚨 For Jenkins jobs, check /var/lib/jenkins/jobs/*/builds

🏁 Summary

Cron → Schedules recurring jobs
At → Runs one-time tasks
@reboot → Executes scripts on startup
Jenkins → Manages enterprise-level schedules
💡 Combine them to create self-healing, auto-starting Middleware environments.

Pradeep V. (VK)

💡Shell Scripting for Beginners – Part 8: Debugging & Logging

By now, your scripts deploy apps, manage servers, and clean logs.
But what happens when things go wrong?
Welcome to Debugging & Logging, where we make your scripts smarter and traceable like a pro DevOps engineer! 🕵️‍♂️

📑 Table of Contents

Debugging with set -x
Using trap to Handle Failures
Exit Codes & Error Messages
Logging with Syslog & Files
Real Middleware Examples
Summary

🐞 1️⃣ Debugging with `set -x`

set -x shows every command the script executes — perfect for troubleshooting silent errors.

#!/bin/bash
set -x  # enable debugging
echo "Starting deployment..."
mkdir /opt/app/deploy
cp /tmp/app.war /opt/app/deploy/
set +x  # disable debugging
echo "Deployment finished!"

💬 Output:

+ echo "Starting deployment..."
Starting deployment...
+ mkdir /opt/app/deploy
+ cp /tmp/app.war /opt/app/deploy/
+ echo "Deployment finished!"
Deployment finished!

💡 Use bash -x script.sh to debug a script without editing it.

🪤 2️⃣ Using `trap` to Handle Failures

The trap command catches unexpected exits (Ctrl+C, errors) and executes custom recovery code. Perfect for cleanup or service restarts.

#!/bin/bash
trap 'echo "⚠️ Script interrupted! Cleaning up..."; rm -f /tmp/tempfile' EXIT
echo "Creating temporary file..."
touch /tmp/tempfile
echo "Simulating work..."
sleep 3
exit 1  # simulate failure

💬 Output:

Creating temporary file...
Simulating work...
⚠️ Script interrupted! Cleaning up...

💡 Always trap EXIT or ERR to prevent orphan temp files or hanging processes.

🚪 3️⃣ Exit Codes & Error Messages

Every command returns an exit code — 0 means success, non-zero means failure.

#!/bin/bash
cp /tmp/app.war /opt/app/deploy/
if [ $? -ne 0 ]; then
  echo "❌ Copy failed! Check file path or permissions."
  exit 1
else
  echo "✅ Copy successful!"
fi

💬 Output:

❌ Copy failed! Check file path or permissions.
or
✅ Copy successful!

Use echo $? after any command to view its exit code.

🧾 4️⃣ Logging with Syslog & Custom Files

Example 1: Using `logger` Command

#!/bin/bash
logger "Starting Middleware backup..."
tar -czf /tmp/backup.tar.gz /opt/middleware/ || logger "Backup failed!"
logger "Middleware backup completed!"

💬 Syslog Output (in /var/log/syslog):

Nov 1 02:15:00 server logger: Starting Middleware backup...
Nov 1 02:15:03 server logger: Middleware backup completed!

Example 2: Redirecting Logs to a File

#!/bin/bash
exec >> /opt/logs/deploy.log 2>&1
echo "[$(date)] Starting deployment..."
echo "[$(date)] Copying WAR..."
cp /tmp/app.war /opt/tomcat/webapps/
echo "[$(date)] Deployment done ✅"

💬 deploy.log Content:

[Sat Nov 1 02:20:00 2025] Starting deployment...
[Sat Nov 1 02:20:01 2025] Copying WAR...
[Sat Nov 1 02:20:02 2025] Deployment done ✅

⚙️ 5️⃣ Real Middleware & DevOps Examples

Example 1: Jenkins Job Failure Recovery

#!/bin/bash
set -e
trap 'echo "🚨 Jenkins build failed, restarting service..."; systemctl restart jenkins' ERR
curl -s -X POST "http://localhost:8080/job/myjob/build" || false
echo "✅ Jenkins job triggered successfully!"

💬 Output:

🚨 Jenkins build failed, restarting service...
✅ Jenkins job triggered successfully!

Example 2: Tomcat Log Watcher with Debug Info

#!/bin/bash
set -x
tail -Fn0 /opt/tomcat/logs/catalina.out | \
while read line; do
  echo "$line" | grep "ERROR" &>/dev/null && echo "🔥 Found error: $line" >> /opt/logs/tomcat_error.log
done

💬 Output:

[DEBUG] + grep "ERROR" catalina.out
🔥 Found error: Database connection timeout

Example 3: NGINX Health Check with Logging

#!/bin/bash
while true; do
  if ! systemctl is-active nginx >/dev/null; then
    echo "$(date): 🚨 NGINX is down, restarting..." >> /var/log/nginx_monitor.log
    systemctl restart nginx
  fi
  sleep 10
done

💬 Log Output:

Sat Nov 1 03:00:00 2025: 🚨 NGINX is down, restarting...

🏁 Summary

set -x → Debug command execution flow.
trap → Catch exits & clean up automatically.
exit codes → Detect and handle failures gracefully.
logger → Send messages to syslog or custom logs.
💡 Always include logging in production-ready scripts for audit and rollback.

Pradeep V. (VK)

📁 Shell Scripting for Beginners – Part 7: File Handling in Shell.

In Middleware & DevOps automation, file handling is everywhere — reading logs, writing reports, rotating backups, or watching live output.
Let's learn how to read, write, append, and monitor files in shell scripts with real-world Middleware examples.

📑 Table of Contents

Reading Files
Writing & Appending Files
Monitoring Log Files
Error Handling & Debug Tips
Summary

📖 1️⃣ Reading Files

Example 1: Read File Line by Line

#!/bin/bash
filename="/opt/scripts/serverlist.txt"
while read -r line; do
  echo "Processing server: $line"
done < "$filename"

💬 Output:

Processing server: app1
Processing server: app2
Processing server: app3

Example 2: Display Tomcat Log Errors

#!/bin/bash
grep "ERROR" /opt/tomcat/logs/catalina.out

💬 Output:

ERROR [localhost-startStop-1] ... Application failed to start
ERROR [main] ... Database connection refused

💡 Tip: Use grep -i error to ignore case.

✍️ 2️⃣ Writing & Appending Files

Example 3: Write Output to File

#!/bin/bash
echo "Backup started at $(date)" > /opt/scripts/backup.log
echo "Compressing files..." >> /opt/scripts/backup.log
tar -czf /tmp/backup.tar.gz /opt/data && echo "Backup successful" >> /opt/scripts/backup.log

💬 Output File Content:

Backup started at Sat Nov 1 02:00:00 IST 2025
Compressing files...
Backup successful

Example 4: Append Deployment Status

#!/bin/bash
env=$1
status=$2
echo "$(date) - $env deployment $status" >> /opt/scripts/deploy_history.log

💬 Output File Content:

Sat Nov 1 02:15:00 IST 2025 - dev deployment SUCCESS
Sat Nov 1 02:17:00 IST 2025 - prod deployment FAILED

Example 5: Log WebSphere Cleanup Activity

#!/bin/bash
logfile="/opt/IBM/WebSphere/AppServer/logs/cleanup.log"
echo "$(date): Cleaning old logs..." >> $logfile
find /opt/IBM/WebSphere/AppServer/logs -type f -mtime +7 -delete && echo "✅ Logs cleaned" >> $logfile

💬 Output File:

Sat Nov 1 02:30:00 IST 2025: Cleaning old logs...
✅ Logs cleaned

🔍 3️⃣ Monitoring Log Files

Example 6: Live Tail of Tomcat Logs

#!/bin/bash
echo "Monitoring Tomcat logs..."
tail -f /opt/tomcat/logs/catalina.out

💬 Output:

Monitoring Tomcat logs...
[INFO] Server startup completed in 2200 ms
[WARN] Deprecated API in use
[ERROR] Connection timeout to DB

Example 7: Auto-restart Jenkins on Log Failure

#!/bin/bash
tail -Fn0 /var/log/jenkins/jenkins.log | \
while read line; do
  echo "$line" | grep "ERROR" &>/dev/null
  if [ $? = 0 ]; then
    echo "🚨 Error detected, restarting Jenkins..."
    systemctl restart jenkins
  fi
done

💬 Output:

[ERROR] Jenkins job queue stuck
🚨 Error detected, restarting Jenkins...

Example 8: Monitor NGINX Access Logs for High Load

#!/bin/bash
tail -n 100 /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c

💬 Output:

500 3
404 7
200 90
301 5

💡 Tip: Use this for quick traffic analysis on your Middleware or DevOps pipeline servers.

⚠️ 4️⃣ Error Handling & Debugging

Use set -e to exit on first failure.
Redirect errors → 2>> error.log to capture exceptions.
Validate file existence before reading:

if [ ! -f "$filename" ]; then
  echo "❌ File not found: $filename"
  exit 1
fi

If your script suddenly exits, check for missing files or permission issues in /var/log/messages or your custom error log.

🏁 Summary

Read files line by line to automate configuration or host processing.
Write and append logs for auditing and historical tracking.
Monitor live logs (Tomcat, Jenkins, NGINX) using tail -f.
Add error handling to avoid silent file issues in production.

Pradeep V. (VK)

📦 Shell Scripting for Beginners – Part 6: Arrays & Arguments

Arrays and arguments give your scripts flexibility — perfect for automating Middleware & DevOps tasks like restarting Tomcat across servers, cleaning WebSphere logs, or passing dynamic inputs to Jenkins jobs.

📑 Table of Contents

Arrays
Command-line Arguments
Combining Arrays & Arguments
Error Handling & Debugging
Summary

1️⃣ Arrays in Shell

🎯 Example 1 – Basic Array

#!/bin/bash
servers=("dev" "stage" "prod")
echo "First server: ${servers[0]}"
echo "All servers: ${servers[@]}"

💬 Output:

First server: dev
All servers: dev stage prod

🔁 Example 2 – Loop through Array

#!/bin/bash
apps=("Tomcat" "WebSphere" "JBoss")
for app in "${apps[@]}"; do
  echo "Managing $app service..."
done

💬 Output:

Managing Tomcat service...
Managing WebSphere service...
Managing JBoss service...

🧰 Example 3 – Restart Tomcat on Multiple Servers

#!/bin/bash
servers=("app1" "app2" "app3")
for s in "${servers[@]}"; do
  echo "Restarting Tomcat on $s..."
  ssh $s "systemctl restart tomcat" || echo "⚠️ Restart failed on $s"
done
echo "✅ Restart loop completed."

💬 Output:

Restarting Tomcat on app1...
Restarting Tomcat on app2...
⚠️ Restart failed on app3
✅ Restart loop completed.

🧹 Example 4 – Cleanup WebSphere Logs

#!/bin/bash
logs=(SystemOut.log SystemErr.log activity.log)
for file in "${logs[@]}"; do
  echo "Deleting $file..."
  rm -f /opt/IBM/WebSphere/AppServer/logs/$file || echo "⚠️ Could not remove $file"
done
echo "🧹 Old WebSphere logs removed."

💬 Output:

Deleting SystemOut.log...
Deleting SystemErr.log...
Deleting activity.log...
🧹 Old WebSphere logs removed.

2️⃣ Command-line Arguments

📦 Example 5 – Access Arguments

#!/bin/bash
echo "Environment: $1"
echo "Version: $2"

💬 Output:

$ ./deploy.sh prod v2.0
Environment: prod
Version: v2.0

🚀 Example 6 – Trigger Jenkins Job

#!/bin/bash
job_name=$1
if [ -z "$job_name" ]; then
  echo "❌ Usage: $0 "
  exit 1
fi
curl -s -X POST "http://localhost:8080/job/$job_name/build?token=deploy123" || echo "⚠️ Failed to trigger job."

💬 Output:

$ ./jenkins_trigger.sh BuildApp
✅ Triggered Jenkins job BuildApp

🧠 Example 7 – Argument Validation

#!/bin/bash
if [ $# -lt 2 ]; then
  echo "Usage: $0  "
  exit 1
fi
echo "Deploying $1 version $2..."

💬 Output:

$ ./deploy.sh myapp v1.5
Deploying myapp version v1.5...

3️⃣ Combining Arrays & Arguments

🌍 Example 8 – Check NGINX on Multiple Hosts

#!/bin/bash
hosts=("$@")
for h in "${hosts[@]}"; do
  echo "Checking NGINX on $h..."
  ssh $h "systemctl is-active nginx" >/dev/null 2>&1 && echo "✅ $h OK" || echo "❌ $h DOWN"
done

💬 Output:

$ ./check_nginx.sh host1 host2 host3
✅ host1 OK
✅ host2 OK
❌ host3 DOWN

💾 Example 9 – Query Multiple Databases

#!/bin/bash
dbs=("salesdb" "hrdb" "logdb")
for db in "${dbs[@]}"; do
  echo "Checking $db connection..."
  mysql -h dbserver -u admin -psecret -e "use $db;" >/dev/null 2>&1 && echo "✅ $db reachable" || echo "⚠️ $db not reachable"
done

💬 Output:

Checking salesdb connection... ✅ salesdb reachable
Checking hrdb connection... ⚠️ hrdb not reachable
Checking logdb connection... ✅ logdb reachable

🧩 Example 10 – CI/CD Pipeline Trigger

#!/bin/bash
branches=("main" "dev" "hotfix")
for b in "${branches[@]}"; do
  echo "Triggering pipeline for $b..."
  curl -s -X POST "https://ci.middlewarebox.com/job/build-$b/build" || echo "⚠️ Failed for $b"
done

💬 Output:

Triggering pipeline for main... ✅ Done
Triggering pipeline for dev... ✅ Done
Triggering pipeline for hotfix... ⚠️ Failed for hotfix

4️⃣ Error Handling & Debugging

Use set -x to trace executed commands.
Check argument count with $#.
Use "$@" to safely pass all args to loops.
Handle SSH/curl/mysql failures with || and echo messages.
Log outputs → script.sh >> /var/log/script.log 2>&1

🏁 Summary

Arrays – store lists of servers, apps, or files.
Arguments – pass values into scripts for dynamic execution.
Combine both for multi-server automation.
Always include error handling to avoid silent failures.

Pradeep V. (VK)

⚙️ Shell Scripting for Beginners – Part 5: Functions in Shell (Reusable Scripts)

Functions let you organize and reuse code in your shell scripts like a Middleware and DevOps engineer.
Instead of repeating logic for Tomcat restarts or Jenkins backups, define it once — then call it from anywhere.
We'll also explore how to handle errors gracefully inside functions.

📑 Table of Contents

💡 What is a Function?
🧩 Function Syntax
Example 1: Basic Function
Example 2: Function with Parameters
Example 3: Return Values
Example 4: Restart Tomcat (with error handling)
Example 5: Jenkins Backup (with error handling)
Example 6: WebSphere Log Cleanup
Example 7: Database Health Check
Example 8: Modular CI/CD Wrapper Function
🏁 Summary

💡 What is a Function?

A function groups commands under one name so you can call them repeatedly. In Middleware automation, you can define restart, deploy, or monitor tasks as reusable functions.

🧩 Function Syntax

function_name() {
   commands
}
# OR
function function_name {
   commands
}

🎯 Example 1: Basic Function

#!/bin/bash
say_hello() {
  echo "Hello from MiddlewareBox!"
}
say_hello

💬 Output:

Hello from MiddlewareBox!

🧠 Example 2: Function with Parameters

#!/bin/bash
greet_user() {
  echo "Welcome $1! You are working on the $2 environment."
}
greet_user Pradeep DevOps

💬 Output:

Welcome Pradeep! You are working on the DevOps environment.

📦 Example 3: Function with Return Value

#!/bin/bash
add_numbers() {
  sum=$(( $1 + $2 ))
  echo $sum
}
result=$(add_numbers 25 30)
echo "The total is: $result"

💬 Output:

The total is: 55

🧰 Example 4: Restart Tomcat (with error handling)

#!/bin/bash
restart_tomcat() {
  echo "Stopping Tomcat..."
  /opt/tomcat/bin/shutdown.sh || { echo "❌ Failed to stop Tomcat"; return 1; }
  sleep 3
  echo "Starting Tomcat..."
  /opt/tomcat/bin/startup.sh || { echo "❌ Failed to start Tomcat"; return 2; }
  echo "✅ Tomcat restarted successfully!"
}
restart_tomcat

💬 Output:

Stopping Tomcat...
Starting Tomcat...
✅ Tomcat restarted successfully!

If Tomcat fails to start, check /opt/tomcat/logs/catalina.out.

💾 Example 5: Jenkins Backup Function (with error handling)

#!/bin/bash
backup_jenkins() {
  backup_dir="/var/jenkins_home/backup"
  mkdir -p $backup_dir || { echo "❌ Failed to create backup directory"; return 1; }
  tar -czf $backup_dir/jenkins_$(date +%F).tar.gz /var/jenkins_home || { echo "❌ Backup failed"; return 2; }
  echo "✅ Backup saved in $backup_dir"
}
backup_jenkins

If backup fails, ensure enough disk space and Jenkins home path is correct.

🧹 Example 6: WebSphere Log Cleanup

#!/bin/bash
cleanup_was_logs() {
  log_path="/opt/IBM/WebSphere/AppServer/logs"
  echo "Cleaning logs in $log_path"
  find $log_path -name "*.log" -type f -mtime +7 -exec rm -f {} \; || echo "⚠️ Cleanup error occurred"
  echo "🧹 Old WebSphere logs removed successfully."
}
cleanup_was_logs

If permissions fail, run with sudo or check AppServer/logs ownership.

🧠 Example 7: Database Health Check

#!/bin/bash
check_db() {
  if nc -zv dbserver 3306 >/dev/null 2>&1; then
    echo "✅ Database reachable"
  else
    echo "🚨 Database not reachable!"
    return 1
  fi
}
check_db || echo "⚙️ Please verify DB server or firewall."

🚀 Example 8: Modular CI/CD Wrapper

#!/bin/bash
deploy_app() {
  echo "Building app..."
  mvn clean package || { echo "❌ Build failed"; return 1; }
  echo "Deploying WAR..."
  scp target/app.war prod:/opt/tomcat/webapps/ || { echo "❌ Deploy failed"; return 2; }
  ssh prod "systemctl restart tomcat"
  echo "✅ Deployment complete!"
}
deploy_app

If deploy fails, verify SSH connectivity and WAR path.

🏁 Summary

Functions make scripts modular, reusable, and easier to maintain.
Handle errors using return and conditional checks.
Great for DevOps: automate Jenkins, Tomcat, WebSphere, and CI/CD pipelines.
Combine with loops and conditionals from previous lessons for full automation.

Pradeep V. (VK)

Menu

2 Nov 2025

💡 How This Tutorial Series Will Help You

📚 Complete Course Index

🚀 Why MiddlewareBox Shell Scripting Series?

📑 Table of Contents

1️⃣ What is a Subshell?

2️⃣ Running Commands in the Background

3️⃣ Managing Jobs

4️⃣ Parallel Execution in DevOps

5️⃣ nohup & disown

6️⃣ Real-World DevOps Examples

🧩 Example 1: Parallel Log Collection

🧰 Example 2: Monitor Jenkins & Docker in Background

📦 Example 3: Run Multiple CI Jobs Simultaneously

🧠 Pro Tips & Troubleshooting

🧩 Process Control Best Practices

⚙️ Debugging Issues

🏁 Summary

📑 Table of Contents

1️⃣ Jenkins Backup & Restore

♻️ Restore Jenkins

2️⃣ Tomcat Backup & Restore

♻️ Restore Tomcat

3️⃣ WebSphere Backup & Restore

♻️ Restore WebSphere

4️⃣ MySQL Database Backup & Restore

♻️ Restore MySQL

5️⃣ Docker Containers Backup & Restore

♻️ Restore Docker Container

6️⃣ Apache Webserver Backup & Restore

♻️ Restore Apache

7️⃣ Automating Backups

🧠 Pro Tips & Troubleshooting

✅ General Best Practices

⚙️ Troubleshooting Common Issues

🏁 Summary

1 Nov 2025

📑 Table of Contents

1️⃣ Log Cleanup Scripts

2️⃣ Apache & Middleware Examples

3️⃣ Docker & Kubernetes Log Cleanup

4️⃣ logrotate Configuration

5️⃣ How to Run & Test logrotate

🧠 Step 1 – Check Syntax of Your Config

⚙️ Step 2 – Run logrotate Manually

⏰ Step 3 – Schedule via Cron (Daily or Weekly)

🏁 Summary

📑 Table of Contents

1️⃣ Overview

2️⃣ Full Middleware Health Monitoring Script

3️⃣ Automate the Script

🏁 Summary

📑 Table of Contents

⏰ 1️⃣ Automating with Cron

Example 1: Daily Jenkins Backup at 3 AM

Example 2: Weekly Tomcat Log Cleanup

🕐 2️⃣ One-Time Tasks with At

Example 3: Restart WebSphere at 11:30 PM

Example 4: Run Backup Once in 2 Hours

🔄 3️⃣ Auto-Execute on System Reboot

Example 5: Auto-Start Tomcat After Reboot

Example 6: Health Check Script at Boot

⚙️ 4️⃣ Jenkins Scheduler Integration

Example 7: Daily Jenkins Build at 2 AM

🧰 5️⃣ Real Middleware & DevOps Automation Examples

Example 8: NGINX Health Monitor Every 10 Minutes

Example 9: Database Backup Every 6 Hours

Example 10: Restart Jenkins After Crash Detection

Example 11: Auto-Remount Missing Volume

🧠 6️⃣ Tips & Troubleshooting

🏁 Summary

📑 Table of Contents

🐞 1️⃣ Debugging with set -x

🪤 2️⃣ Using trap to Handle Failures

🚪 3️⃣ Exit Codes & Error Messages

🧾 4️⃣ Logging with Syslog & Custom Files

Example 1: Using logger Command

Example 2: Redirecting Logs to a File

⚙️ 5️⃣ Real Middleware & DevOps Examples

🐞 1️⃣ Debugging with `set -x`

🪤 2️⃣ Using `trap` to Handle Failures

Example 1: Using `logger` Command