Hermann.Brauer/miyagi-pbs-zfs
1
0
Fork 0
mirror of https://github.com/bashclub/miyagi-pbs-zfs.git synced 2025-12-06 06:28:43 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #3 from bashclub/main

Browse Source 
re
This commit is contained in:
Chriz
2024-06-22 11:49:31 +02:00
committed by GitHub
parent af7b2704d6 fcfc5dace2
commit 6914d3224d
4 changed files with 101 additions and 73 deletions
Download Patch File Download Diff File Expand all files Collapse all files

140
README.md
View File

@@ -1,70 +1,70 @@
# miyagi-pbs-zfs
Secure Proxmox PVE with Proxmox Backup Server PBS and ZFS Pull Replication with a mostly turned off System
Optimize Processes without colliding Replications, Backups, Monitorings or Scrubs
Save lot of Money with less performant Hardware
Start it with bash miyagi-pbs-zfs -c configfile
There are german payed Lessons here:
14. + 16.05.2024 (13-17h) - Replikationen und Backups Trojanersicher (V2) - on https://cloudisevil.com
You also can searh vor sysops.tv or zfs. rocks on YouTube #miyagi - an english Tutorial will follow!
What it does
Miyagi said, best defense, no be there
Usecase
Proxmox Backupserver is running unnecessarly 24/7
ZFS Replication is usually done by a zfs send, so its a push
What if our Backup/Replicaserver is turned off most the time, nobody can attack it
Consider not using a Gateway, use Routes!
Prerequisites
Proxmox with ZFS on Host to Backup/Replicate - we recommend check_mk Agent for automaticly added Tests
Proxmox with ZFS on Target Machine - it´s WOL MAC Address
Proxmox Backup Server as a VM oder better PCT on Target machine
Proxmox Backup Server Datastore has to be Setup on Source
Your contet of your Public Key of the Target Host .ssh/id_rsa.pub added to
Host to Backup .ssh/authorized_keys
Proxmox Backup Server on Target Host .ssh/authorized_keys
ssh one from your Target Host to Source Host and PBS to confirm Host Key with a yes
At all Proxmox 'apt install zfs-auto-snapshot -y'
Target Hosts needs the following tools to be installed
https://github.com/bashclub/zsync
https://github.com/bashclub/check-zfs-replication
Any Host waking up the Target Host or a daily Cronjob
What we do...
Turning on the Computer with a @reboot Cron
@reboot /root/pbs-zfs-daily.sh -c 200-ssd.conf && /root/pbs-zfs-daily.sh -c 200-hdd.conf
So Miyagi at this point can pull two ZFS-Datastores to one Target Datastore, using the full Path of ZFS for Naming.
If you have multiple Target Datasets, please run multiple Configs and disable Proxmox Backup Server!
Miyagi will tag your Source for Replication with Zsync!
Replicating by a Pull with https://github.com/bashclub-zsync
Pushing a Report to Proxmox Source Host, monitored by Check_MK unsing https://github.com/bashclub/checkzfs
The Report will be found by Check_MK´s built in Autdiscovery for new Services.
Data will be valid vor about one Day (90000s) bevore it expires.
So there nothing can go wrong!
Regarding to the Weekday doing Maintenance on Proxmox Backup Server for getting more free Space.
Triggering a Push Backup with PBS (only Way to do) Backup to PBS and checking Exitcode for Report
Reporting PBS Result with 100% certainty in compact OK/WARN State to backuped PVE Host using Check_MK
Verifying older Backups
Protecting the PBS Backups with a ZFS Snapshot
Updating the Host and PBS
TURNING OFF THE MACHINE!!!
Test Targets Tank if less than 75% free, otherwise report to Check_MK
Test Disks after PBS Maintenance, before shutdown with SmartCTL Short Test and report to Check_MK
Support multiple Sources
# miyagi-pbs-zfs
Secure Proxmox PVE with Proxmox Backup Server PBS and ZFS Pull Replication with a mostly turned off System
Optimize Processes without colliding Replications, Backups, Monitorings or Scrubs
Save lot of Money with less performant Hardware
Start it with bash miyagi-pbs-zfs -c configfile
There are german payed Lessons here:
14. + 16.05.2024 (13-17h) - Replikationen und Backups Trojanersicher (V2) - on https://cloudisevil.com
You also can searh vor sysops.tv or zfs. rocks on YouTube #miyagi - an english Tutorial will follow!
What it does
Miyagi said, best defense, no be there
Usecase
Proxmox Backupserver is running unnecessarly 24/7
ZFS Replication is usually done by a zfs send, so its a push
What if our Backup/Replicaserver is turned off most the time, nobody can attack it
Consider not using a Gateway, use Routes!
Prerequisites
Proxmox with ZFS on Host to Backup/Replicate - we recommend check_mk Agent for automaticly added Tests
Proxmox with ZFS on Target Machine - it´s WOL MAC Address
Proxmox Backup Server as a VM oder better PCT on Target machine
Proxmox Backup Server Datastore has to be Setup on Source
Your contet of your Public Key of the Target Host .ssh/id_rsa.pub added to
Host to Backup .ssh/authorized_keys
Proxmox Backup Server on Target Host .ssh/authorized_keys
ssh one from your Target Host to Source Host and PBS to confirm Host Key with a yes
At all Proxmox 'apt install zfs-auto-snapshot -y'
Target Hosts needs the following tools to be installed
https://github.com/bashclub/zsync
https://github.com/bashclub/check-zfs-replication
Any Host waking up the Target Host or a daily Cronjob
What we do...
Turning on the Computer with a @reboot Cron
@reboot /root/pbs-zfs-daily.sh -c 200-ssd.conf && /root/pbs-zfs-daily.sh -c 200-hdd.conf
So Miyagi at this point can pull two ZFS-Datastores to one Target Datastore, using the full Path of ZFS for Naming.
If you have multiple Target Datasets, please run multiple Configs and disable Proxmox Backup Server!
Miyagi will tag your Source for Replication with Zsync!
Replicating by a Pull with https://github.com/bashclub-zsync
Pushing a Report to Proxmox Source Host, monitored by Check_MK unsing https://github.com/bashclub/checkzfs
The Report will be found by Check_MK´s built in Autdiscovery for new Services.
Data will be valid vor about one Day (90000s) bevore it expires.
So there nothing can go wrong!
Regarding to the Weekday doing Maintenance on Proxmox Backup Server for getting more free Space.
Triggering a Push Backup with PBS (only Way to do) Backup to PBS and checking Exitcode for Report
Reporting PBS Result with 100% certainty in compact OK/WARN State to backuped PVE Host using Check_MK
Verifying older Backups
Protecting the PBS Backups with a ZFS Snapshot
Updating the Host and PBS
TURNING OFF THE MACHINE!!!
Test Targets Tank if less than 75% free, otherwise report to Check_MK
Test Disks after PBS Maintenance, before shutdown with SmartCTL Short Test and report to Check_MK
Support multiple Sources

1
config.example
View File

@@ -4,6 +4,7 @@ SSHPORT='22' #SSH Port, usually default 22 internally
BACKUPSERVER=no #use yes for triggering Proxmox Backup to Store
MAINTDAY=7 #1 Monday to 7 Sunday, dont start your System too late
SHUTDOWN=no #No be there anymore
UPDATES=yes #Do PVE and PBS Updates after run
SOURCEHOST='192.168.50.200' # IP from Proxmox VE System to be backuped and replicated daily
SOURCEHOSTNAME='pve3' #Hostname of Proxmox VE System to be backuped and replicated daily

18
dynroute.sh Normal file
View File

@@ -0,0 +1,18 @@
# DDNS Name und Gateway
# usage dynroute.sh ddnsname yourgatewayrouter
# assuming a shutdown after usage, old routes will "not" be deleted
DDNS_HOSTNAME=$1
GATEWAY=$2
#DNS via One
ip route add 1.1.1.1 via 192.168.66.1
echo "nameserver 1.1.1.1" > /etc/resolv.conf
# ddns auflösen
CURRENT_IP=$(dig +short $DDNS_HOSTNAME)
if [[ -z "$CURRENT_IP" ]]; then
echo "Failed to resolve IP for $DDNS_HOSTNAME"
exit 1
fi
# route setzen
ip route add $CURRENT_IP via $GATEWAY

15
pbs-zfs-daily.sh
View File

@@ -28,8 +28,10 @@ echo "zfs_auto_snapshot_label=$ZPUSHLABEL" >> /etc/bashclub/$SOURCEHOST.conf
/usr/bin/bashclub-zsync -d -c /etc/bashclub/$SOURCEHOST.conf
CHECKZFS=$(which checkzfs)
# So one Day has 1440 Minutes, so we go condition Yellow on 1500
/usr/local/bin/checkzfs --source $SOURCEHOST --replicafilter "$ZFSTRGT/" --filter "#$ZFSROOT/|#$ZFSSECOND/" --threshold 1500,2000 --output checkmk --prefix pull-$(hostname):$ZPUSHTAG> /tmp/cmk_tmp.out && ( echo "<<<local>>>" ; cat /tmp/cmk_tmp.out ) > /tmp/90000_checkzfs
$CHECKZFS --source $SOURCEHOST --replicafilter "$ZFSTRGT/" --filter "#$ZFSROOT/|#$ZFSSECOND/" --threshold 1500,2000 --output checkmk --prefix pull-$(hostname):$ZPUSHTAG> /tmp/cmk_tmp.out && ( echo "<<<local>>>" ; cat /tmp/cmk_tmp.out ) > /tmp/90000_checkzfs
scp /tmp/90000_checkzfs $SOURCEHOST:/var/lib/check_mk_agent/spool/90000_checkzfs_$(hostname)_$ZPOOLSRC
@@ -84,8 +86,15 @@ scp /tmp/90000_checkpbs root@$SOURCEHOST:/var/lib/check_mk_agent/spool
#doing updates without regeret
apt dist-upgrade -y
ssh $PBSHOST apt dist-upgrade -y
if [[ "$UPDATES" == "yes" ]]
then
apt dist-upgrade -y
ssh $PBSHOST apt dist-upgrade -y
else
echo no Updates configured - Consider updating more often!
fi
if [[ "$SHUTDOWN" == "yes" ]]
then