1. Backup VMware ESXi virtual machines
Introduction
This document presents various methods and strategies for backing up VMware ESXi using vSphere and Bacula Enterprise Edition versions 8.0, 8.2, and 8.4. The Bacula Enterprise Edition plug-in for backing up VMware virtual machines with vSphere gives you the ability to restore the original state of a virtual machine, while file backup at the guest VM level simplifies the protection of mission-critical application data. VMware backup uses a technology called Changed Block Tracking (CBT) to ensure that only those blocks that have changed since the original full and/or last block are sent to the current incremental or differential backup stream in order to create more efficient backups and reduce network load. incremental and/or differential backup.
Key Features of VMware Backup
- Online backup via VADP
- Create VSS snapshots inside guest OS to pause applications
- Full, differential and incremental VM backup at the image level
- Restoring a full VM image
- Restoring vmdk files to an alternative directory
- Access to VMware storage, both over TCP/IP and SAN (FC/ISCSI)
Overview of VMware Backup
The current version of the plugin for VMware vSphere supports vSphere versions 6.0, 5.5, 5.1, 5.0, 4.1 (minimum version 7 virtual hardware). This document provides software solutions Bacula Enterprise Edition 8.0 and later versions, which are not applicable to earlier versions of the software.
VMware Backup Glossary
This document uses the following terms related to how to back up VMware:
- CBT– technology for tracking changed blocks.
- Datastore is the name used by VMware to refer to data stores.
- vSphere- is a VMware technology for OS virtualization and cloud computing.
- VDDK is a set of C/C++ libraries that allows you to create and access VMware virtual disks. The VDDK is used in parallel with the vSphere API to write backup and restore software or similar applications.
- When using a VMware ESXi server, virtual machine files are placed on large external memory.
- NBD– network block device. vSphere allows you to access files hosted in the Datastore using direct file access technology, access over NBD, NBD over SSL, or SAN. When accessing files via NBD, the network protocol is TCP/IP.
- SAN. vSphere allows you to access files in the data store using direct access technology. SAN can use Fiber Chanel (Lan free backup technology) or ISCSI over TCP/IP technology.
- VMware ESX and VMware ESXi is a hypervisor architecture installed on a server without an operating system. The smaller ESXi codebase means a smaller attack surface and a smaller patch code size, which improves system reliability and security.
- VCB- VM Consolidated Backup Method An older VMware API that is generally no longer used. The VMware plugin does not use VCB technology.
- VADP is the next generation of VMware data protection infrastructure, implemented in vSphere 4.0, allowing backup software to create centralized, efficient VMware backups off host machines and without loading the local network.
- .vmdk- file format used for virtual appliances developed for VMware products.
- .bvmdk- internal file format used by the Bacula Enterprise plugin for handling VMware sparse blocks and differential/incremental binary backups. After conversion with the vddk tool, the file becomes a raw image of the original disk, which can be converted to vmdk format using the qemu-img utility.
- ESX 3.x uses version 4 of the virtual hardware, vSphere 4.x uses version 7, and vSphere 5 uses version 8.
- The fingerprint can be generated from the ESXi host
openssl x509 -sha1 -in /etc/vmware/ssl/rui.crt \-noout -fingerprint | cut -d ‘=’ -f 2 - guestfish- shell and command line tool for viewing and modifying the VM file system.
- VM (or VM) – an abbreviation for the term "virtual machine".
- vSphere is a server virtualization platform with the ability to consistently manage virtual data centers.
- SELinux- Security-Enhanced Linux (SELinux, Security-Enhanced Linux) is a Linux kernel security module that provides a mechanism to support access control security policies, including Authoritative Access Control (MAC).
1.1 How to backup VMware in guest OS
1.1.1 Installing the Bacula Client in each guest OS
The first strategy does not involve the use of a plugin Bacula Enterprise Edition for vSphere. Instead, the Bacula Enterprise File Daemon is installed on each VM as if the VMs were normal physical servers. Tasks are used to optimize I/O flows on VMware ESX/ESXi servers. Schedule, priority and Maximum Concurrent Jobs to distribute backup tasks in the backup window. Since all servers use the same set of disks, performing all backup tasks at the same time, it is possible to bottleneck the disk/network subsystem.
Figure 1: Installing bacula-fd on each guest VM
Installing the Bacula Enterprise File Daemon on each VM allows you to manage virtual servers as if they were physical servers, as well as use all the features of the Bacula Enterprise software, such as:
- Quick recovery of individual files
- Checksum calculation for individual files to detect viruses and spyware
- Checking the task
- File/directory exclusion (such as swap files and temporary files)
- File-level compression, etc.
1.1.2 Backing up VMware with the Bacula Enterprise Edition plug-in for vSphere
In the case of a strategy for creating a backup of a VMware virtual machine image, the plugin Bacula Enterprise Edition for vSphere saves Client disks as raw images in a VMware/vSphere context. In order to implement this strategy, it is not necessary to install the Bacula File daemon on every guest machine.
The Bacula plugin for vSphere will communicate with the VMware ESXi host to read and store the contents of the VM disks via NBD or SAN. With direct access to the image vmdk, saved in data store, Bacula software does not have to run through the Client's file system to open/read/close files. Accordingly, the software will consume less ESXi infrastructure resources than if VMware backups were created on each guest machine. At the same time, Bacula will also read and store useless data such as swap files and temporary internet files.
Figure 2: Creating a TCP backup using NBD
If the vSphere backup plugin uses the NBD data transport method, the data is streamed to the backup storage server through the ESXi system's VMkernel port.
The Bacula Enterprise plugin for vSphere can also use the SAN infrastructure to reduce the load on ESXi servers. However, despite consuming fewer resources on the ESXi server, data will still need to be read from your disks, which can lead to conflicts when trying to send/receive data at the same time.
When using block differential methods, such as those used by the vSphere plugin, all incremental backups must be available for recovery. If at least one backup task is missing at the time of restoration, the Bacula plugin will not be able to recreate the correct image. The use of differential backups allows you to reduce the number of tasks required for recovery, thereby reducing the risk of possible data loss. To prevent the loss of important tasks for creating incremental backups, retention periods Volume retention must be large enough to recover all the data.
1.1.3 Comparison of VMware backup strategies
Table 1. Comparison of backup strategies
The procedure for restoring individual files from a backup of VMware machines created using the plugin for vSphere is described in section 2 on page 27.
1.2 Installation
Documentation detailing the installation process is available upon request.
1.2.1 Configuration
The File Daemon's Plugin Directory parameter, stored in /opt/bacula/etc/bacula-fd.conf, must point to where the plugin is installed vsphere-fd.so. As a rule, by default the Bacula plugin is installed in the directory: /opt/bacula/plugins
The File daemon must have direct access to the vSphere network or access through the SAN. You can check the connection using the telnet program. vSphere network access to ESX or the vCenter server must be configured in /opt/bacula/etc/vsphere_global.conf.
Figure 3. Creating a backup over a SAN
| Parameter | Required | Default value | Description |
| General settings section global | |||
| keep_generation | Not | 100 | Max. number of backups between two full backups. |
| profile_all_vm | Not | vsphere_all_vm.profile | The name of the internal file used to store VM profile information. |
| root_directory | Not | /opt/bacula/working/vsphere | The root directory of the vSphere plugin. |
| vddk_path | Not | /opt/bacula/bin/vddk | |
| Settings section vsphere | |||
| username | Yes | The username that is allowed to connect to vSphere. | |
| password | Yes | The password for the username that is allowed to connect to vSphere. | |
| hpassword | Not | Hidden password for the username that is allowed to connect to vSphere. | |
| timeout | Not | 60 | Timeout to connect to the vSphere server in seconds. |
| thumbprint | Yes | SSL thumbprint of the vSphere server certificate. | |
| server | Yes | The vSphere ESXi server used to create the backup. | |
| url | Yes | The address of the vSphere ESXi or vCenter server used to make the call using SOAP. | |
| Default_datastore | Not | datastore1 | The default recovery storage. |
| default_restore_host | Not | ESX server used by default for recovery if multiple servers are available in vCenter. | |
| default_ovf | Not | The default OVF description used in case the current OVF description cannot be loaded into VMWare . | |
| root_directory | Not | /opt/bacula/working/vsphere | The directory used to store the plugin's internal data. |
| datastore_minimum_space | Not | The minimum size to store data in the data store. For example, 5GB. | |
| datastore_allow_overprovisioning | Not | Yes | Allows you to restore a VM using the Over Provisioning feature. If the parameter is set to " Not”, when restoring, you must ensure that the size of all disks matches the size of the Datastore. |
| datastore_refresh_interval | Not | 600 | The interval used to update data storage statistics in the Datastore. |
Table 2. Configuring a vSphere connection using the vsphere_global.conf file
The fingerprint can be obtained using the console screen by pressing F2 and then logging in. The Thumbprint will appear in the window View Support Information under SSL Thumbprint (SHA1). Or you can connect via ssh:
Using multiple vSphere servers
You can specify multiple vsphere servers in the vsphere_global.conf file. When using this function, you need to set the server=xxx parameter in the plugin's command line. It is also mandatory to specify an alternate directory in case your VM has the same MoRef value.
Take into account the fact that the default section is required in the vsphere_global.conf file.
| Parameter | Required | Default value | Description | Example | |
| host | Not | Guest VM name | host=srv1 | ||
| host_include | Not | Guest VM image to include | host_include=srv3 | ||
| host_exclude | Not | Guest VM image to exclude | host_exclude=srv | ||
| disk_exclude | Not | List of drives to be excluded | disk_exclude=0,2,4 | ||
| keep_cbt | Not | Don't try to activate CBT | keep_cbt | ||
| quiesce_host | Yes | Stop the guest VM before taking a snapshot (try, yes, no) | quiesce_host=no | ||
| server | Not | Specify vsphere server | server=vsrv2 | ||
| debug | Not | Allow Debugging | debug | ||
| abort_on_error | Not | Stop a task after an error is detected | |||
| update_timeout | Not | Change initial update timeout | |||
Table 3. vSphere plugin command parameters
Take into account the fact that commands host_include and host_exclude are a Java regular expression.
Hide vSphere password
Starting from plugin version 8.0.3 you can hide the vSphere password in the file vsphere_global.conf. The hidden password field is called hpassword. To generate a hidden password, you can use the command @encode. Take into account the fact that if the string you want to encrypt contains the expression "=", when writing the command you must use the format string= keyword.
Testing the vSphere Configuration
To test the plugin for vSphere, you can use the following command (as root user):
When using the update command vsphere-ctl a list of all VMs that are defined on the ESXi server should appear. If this does not happen, please check that your credentials are correctly configured in the file vsphere_global.conf.
Team list allows you to display information found on ESX hosts and data stores.
Job function example
When starting tasks to create an incremental/differential backup, it is necessary to set the parameter Accurate.
Examples of using the FileSet function
This section presents various options for using the function. FileSet. Please note that the vsphere plugin is not compatible with the FileSet feature for sparse files.
Figure 4. Backup of the VMware guest1 virtual machine on the ESXi server
Testing the FileSet Function
You can use the command estimate to test the FileSet function.
Implementing VMware Block-Level Incremental Backups
Take into account the fact that the CBT utility is not supported by 6th and earlier versions of virtual hardware, or when the virtual disk is connected to a shared virtual SCSI bus.
In order for CBT to be able to determine the changed sectors of the disk since the last ID change, the following conditions must be met:
- Host version ESX/ESXi 4.0 or higher.
- 7 version (and higher) of the VM hardware that owns the disks whose changes are to be monitored.
- I/O operations must be performed through the ESX/ESXi storage element block. NFS is supported as RDM drives in virtual compatibility mode, but not RDM drives in physical compatibility mode. It also uses the VMFS file system with support for SAN, iSCSI, or local disk.
- For the VM, you need to activate the CBT utility (see description below).
- The VM storage should not (permanently or non-permanently) be represented by an independent disk, that is, one that will not be affected by snapshots.
In order for the CBT utility to be able to determine disk sectors using a full backup, you will need to meet the following conditions:
- The virtual disk must be located on a VMFS volume supported by SAN, iSCSI, or a local disk.
- The VM must have zero snapshots (0) when CBT is activated to implement the so-called. clean launch.
When using "Thick Provisioned Eager Zeroed" disks, VMWare CBT will show all blocks as used during the full backup. For VMs that do not support CBT, the plugin for vSphere will always perform a full backup of virtual disks. To check if the virtual disk CBT utility has been activated, open the vSphere client, select the command powered-offvirtual machine without snapshots(turn off the VM without creating snapshots).
- Right-click on the VM and select edit settings Edit Settings.
- Go to the tab Options.
- Click on tab General under the tab Advanced, and then by item Configuration Parameters. The parameter configuration dialog will open.
- Click on an item Add Row.
- Add a parameter ctkEnabled and give it a value true.
- Click on Add Row, add parameter scsi0:0.ctkEnabled and give it a value true.
Attention: line scsi0:0 in parameter scsi0:0.ctkEnabled indicates the SCSI device assigned to the hard drive added to the VM. Each hard drive added to a VM gets its own SCSI device, referred to as scsi0:0, scsi0:1, or scsi1:1. During the creation of the first full VMware backup, the vSphere plugin will try to automatically activate the CBT utility when the VM is turned off. To disable this feature, enter the command keep_cbt on the plugin command line.
Problems when using CBT
If you are reverting to a snapshot earlier than the last incremental backup, you must create a full backup of the VM before reusing incremental backups. This issue has been resolved in vSphere 4.1 and vSphere 4.0 Update 3. Instead of possibly providing incomplete data, a change identification number obtained before reverting to a previous snapshot is now correctly treated as invalid (http://kb.vmware.com/kb/1021607).
Compress backup size by resetting CBT
Once a block is marked as "used" by VMWare CBT, the system will continuously back up that particular block when performing a full backup, even if that block is marked as "free" by the guest OS. After some time, a situation may arise in which a large full VMware backup will be created with a small amount of disk space used.
By re-creating the disc with VMotion, you can reset the CBT table to mark only the actually used blocks. To perform this operation, you must first clear the disk of the guest VM by writing "zero" blocks to cover all the free space. Take into account the fact that the operation will use resources, so it must be performed outside business hours.
On Windows, the procedure can be performed using the utility Microsoft delete, available at http://technet.microsoft.com/en-us/sysinternals/bb897443.aspx
On Linux, you can use the built-in tool dd. Take into account the fact that you can limit dd to not completely fill up the entire disk.
Upon completion of the operation, you must stop the guest VM. This can be done via the ESXi shell interface as follows:
Information about the location of the disk and the configuration file can be found as follows:
After that, zero blocks of VMDK files must be cleared through the ESXi shell interface as follows:
Upon completion of the operation, you must deactivate CBT for the guest drives that you want to compress. You can also edit them through the vSphere management console, or IN AND.
Then you need to enable/disable the guest VM in order to apply the changes to the CBT utility. You can wait until the host is fully up and running.
Now you should not see files like “*-ctk.vmdk” and you can re-enable CBT in the host config file and start your guest VM.
Files like “*ctk.vmdk” will be recreated. Team estimate bacula plugin should display files bvmdk smaller size.
Since this procedure is quite complicated, we recommend that you try it out through the sandbox first. If the ESXi SSH interface is enabled, then you can script anything.
Determination of unavailability of CBT
If the CBT (Changed Block Tracking) utility is not available for disk, the file vsphere-ctl*log may contain the following error:
When this error occurs, the vSphere plugin will automatically create a full disk image backup. To enable CBT for a specific disc, see section 1.2.1 on page 14.
Activation of access through SAN
You may have difficulty configuring SAN access on the host. VixDiskLib VMWare library compiled for Redhat 5 64bit version. On later OSes like Ubuntu or Redhat 6, you need to compile and install the 1.95.7 library. Please note that the Bacula Enterprise plugin for vSphere contains this library in the package bacula-enterprise-vixdisk.
In order to use the SAN data movement technology, the backup server on which the vsphere plugin is installed must have access to all LUNs exported to the ESX server. Packages like multipathd, will not have problems with devices with different connections. If your drives are visible as /dev/sda, /dev/sdb, … the vSphere plugin will open each drive to get a UUID and compare it to the one provided by the ESX server. For example, when using iSCSI, the lsscsi command will list the drives as follows:
You can verify that the method of transferring data across the SAN is being used by using the debug function debug on the plugin command line and make sure that the file vddk trace contained in the following location:
If the SAN data transfer mode is not available, the plugin for vSphere will automatically switch to the nbd data transfer mode.
Removing old snapshots
If the VMware system contains snapshots that were not automatically removed by the vSphere plugin, you can clean up the system using the vSphere Plugin version 6.6.3 and higher using the following commands.
- Removing old snapshots and previous failed snapshots
vsphere-ctl clean-snapshot --snapshot myhost
- Deleting old snapshots with a name starting with a string
vsphere-ctl clean-snapshot --snapshot-base pluginTest myhost
- Deleting all snapshots with all derivatives; possibly faster)
vsphere-ctl clean-snapshot --snapshot --snapshot-delete-child myhost
When starting a new backup task, the vSphere plugin will automatically check for problems with the previous task and delete any old snapshots if necessary.
Debug trace
The plugin for vSphere uses various technologies and third-party libraries. As a result, the system makes extensive use of the trace function. The user will be able to use the help of the following files:
Table 4. Tracing methods used by the plugin for vSphere
To extract a bvmdk file without converting it with vddk during restore, you need to set the FileDaemon debug level to 1000. During restore, Bacula may generate incorrect file size reports.
Working files
The plugin for vSphere creates special files in working directory. These files are required for the VMWare CBT utility to work. To clear the vSphere plugin working directory, you can use the command vsphere-ctl:
This will remove the 30-day files and directories. This period should correspond at least to the period for creating a full backup, plus a few days for security. During the backup, if the plugin can't find working files during the last backup, the vSphere plugin will create a full backup of all disks.
Disk exception
To exclude a specific disk from the procedure, you can activate independent mode through the vSphere console, or use the function disk_exclude(see table 1.2.1 on page 11). To find diskid in order to use it in a function disk_exclude, you can use the command estimate listing. 0.bvmdk is the diskid 0 image.
1.3 VMware vSphere backup and restore procedures
1.3.1 Backup
Figure 5. Excluding a disk from a backup
1.3.2 Recovery
Bacula Enterprise software allows you to restore any file (bvmdk, ovf, ...) on local drives. After that you can mount the image locally using the VMWare tool vmware mount tool or qemu-nbd and perform a file-level restore. When using the parameter where=/path/to/dir in the restore function, the plugin will automatically restore the selected files to the specified location.
It is also possible to copy the raw image to any device, or mount it and restore the files directly.
Recovery on a new guest VM
If you start the recovery procedure of your VM with the where=/ option, and select all files in the directory vm, the plugin for vSphere will try to restore your disks to a new VM created during restore with the existing attributes (disks, controller, CPU type, ...).
Enhanced SAN transfer mode is currently not supported for recovery. The plugin for vSphere uses NBD data transfer.
The ESX host and storage that will be used to restore the guest VM will be automatically determined. However, you can change the default location by changing the plugin restore options via the bconsole menu:
Or you can use the BWeb interface (see Figure 6)
Figure 6: Selecting a datastore, ESXi server, or hostname at the time of recovery
Take into account the fact that you need to configure at least one VM on your ESX server in order to automatically recover a VM with Bacula. We plan to remove this limitation in the future.
Starting with Bacula Enterprise 6.2.4, the plugin for vSphere supports automatic network topology creation. Thus, if your ESX host does not provide the correct vSwitch configuration for the VM, the Bacula plugin will have to regenerate all network settings during recovery.
Starting with Bacula Enterprise 8.2.1, the plug-in for vSphere can check for available memory in the Datastore during restore. The user can prohibit the increase in the spare area and reserve the minimum amount of memory in storage. These two options can be configured in the file vsphere_global.conf and can be overwritten from the recovery menu.
server=192.168.0.68
url = https://192.168.0.68/sdk
datastore_minimum_space = 64MB
datastore_refresh_interval = 10
datastore_allow_overprovisioning = false
The "unallocated" amount of memory returned by the vSphere server is not always accurate. The refresh rate can be changed using the method described in the manual at http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2008367
Sometimes Bacula software fails to load the OVF file describing the guest VM to the vSphere or vCenter server. In particular, this is due to certain VMware restrictions, such as “you cannot use an OVF containing references to a mounted CDROM”… The vSphere plugin uses workarounds to solve these problems, but it does not solve all problems. If you are having similar difficulties, you can use the parameter default_ovf in file vsphere_global.conf. As a rule, you need to configure the parameter default_ovf so that it references an existing simple OVF template. During the recovery process, this template will be used automatically and you will need to configure the VM later with values such as CPU number, RAM size, etc.
On Windows, in some cases, after the restore process actually completes, you may need to perform additional tasks. For example, if the restored system will not boot, you may need to use the Windows Recovery tools to debug the system. For servers with Active Directory installed, you may need to review Microsoft's guides to get your AD databases consistent and in sync with other AD servers. If the installation involves dynamic disks, you must import them into a freshly restored system after a reboot. You can import using the Disk Manager or using the "diskpart" function by selecting one of the dynamic disks and entering the "import" command.
Recovery without plugin for vSphere
If you are trying to recover disks in a File Daemon that does not have the Bacula Enterprise plugin for vSphere installed, you will need to convert the bvmdk files to raw files using the vddk command from the command line:
Format bvmdk used by the vSphere Plugin to ensure data integrity and efficient handling of sparse information by the CBT utility.
1.4 Suspending a guest VM
To correctly suspend a guest VM, you need to install and update the VMware Tools on the Linux/Windows Virtual Machine VM.
Plugin team quiesce_host=Try/yes/no allows you to control the procedure for stopping guest VMs using vSphere before capturing a snapshot. The default value is try. In this mode, the plugin will try to stop the guest VM when taking a snapshot, and if the snapshot fails, the plugin will try to re-create the snapshot without stopping the guest VM. The first attempt will be logged in the task log as an error.
For more information about the specific error message, see the vSphere console log.
Warning message from ESXi: the guest OS has reported an error during quescing. Error code was: 2 the error message was: custom quiesce script failed. (Error message from ESXi: The guest OS reported an error while shutting down. Error code 2: Stop script error)
An error occurred while saving the snapshot: Failed to quit the virtual Machine (An error occurred while saving the snapshot: Unable to stop the VM)
1.4.1 Linux
By creating a special script in /usr/sbin/pre-freeze-script, you will be able to stop your system automatically when you create a snapshot with vSphere. vSphere will try to execute the script /usr/sbin/post-thaw-script in case it will be present in the guest OS.
1.4.2 Windows VSS
The plugin enhances Windows security by creating VSS-based snapshots before backing up to stop VSS-activated applications.
Pre-freeze and post-thaw scripts for VSS. Starting with ESX/ESXi 3.5 U2 and later, VMware Tools first searches alphabetically for scripts in C:/Program Files/VMware/VMware Tools/backupScripts.d, calling them with an argument freeze, and then in reverse alphabetical order calls with an argument thaw(or freezeFail in the event of an unsuccessful stop).
1.5 Supported platforms
The plugin for VSphere supports the following products on the VMware platform:
- ESX/ESXi versions: 6.0, 5.5, 5.1, 5.0, 4.1
We are currently testing the correct operation of the plugin for VSphere with the following products on the VMware platform:
- vCenter Server versions 6.0, 5.5, 5.1, 5.0, 4.1 managing ESX/ESXi 4.1 and later
- VirtualCenter version 2.5 managing ESX/ESXi 4.1
To manipulate files and snapshots, the plugin for VSphere uses the vStorage API. This extension requires a valid non-free VMWare license.
- The VSphere plugin has been tested (and supported) on the following Linux based platforms: RHEL 6, 7 (Red Hat Enterprise Linux) 64bitSLES 11 (SUSE Linux Enterprise Server) 64bit
1.6 Restrictions
Plugins may not be compatible with the default VirtualFull tasks. Please contact Bacula Systems Support to make sure you are using the optimal settings.
2 Overview of the VMware Single File Recovery Procedure
This section provides information on how to use the single file recovery function. VMware via Bacula Enterprise Edition and plugin for vSphere.
Brief Description of Functions
Single file recovery tool Bacula Enterprise Edition allows you to use the following functions:
- Console interface
- Bweb Management Suite Interface
- Support for full/differential/incremental backups
- Support Windows 2003 to 2012
- Linux support (ext3, ext4, btrfs, lvm, xfs)
- ESX 5.x and 6 support
2.1 Installation
Documentation detailing the installation procedure is available upon request.
2.2 Recovery scripts
This feature allows you to quickly find and restore specific files from a directory in a VMware environment.
2.2.1 Through the text console interface
Single file recovery plugin ( VMware single file restore) allows you to use a simple software console that provides access to files inside the VM. The process of restoring a single file begins with mounting VM backups:
Choose the right client first
Then, select the task you want to restore.
Then select the desired VM.
Now select the location of the guest file system (locally or via SMB)
At this stage, the VM file system is mounted locally (in the example above, the files are available at /opt/bacula/working/vmware/5. As with the standard file system, it is possible to find directories and copy files (via cp, scp, ftp) from another terminal session using Unix “root” and “bacula” accounts. If you need to use a different Unix account to work with files, use the function -o allow_other” when running the script mount-vmware.
To clear the session, simply press "Enter" in the terminal session in which the script was run. mount-vmware.
Starting with Bacula Enterprise 8.4.8, you can limit the Job list with the following command lines:
- -s=
limit the list of tasks to the last XXX days - -l=
limit task list to last entered numbers - -f=
specify advanced filter based on task name and/or FileSet name
2.2.2 Restoring VMware from the interface Web Management Suite
Single file recovery function VMware single file restore can be implemented using Web Management Suite. This utility is a recovery wizard that allows you to easily and easily restore files from a guest VM. First you need to select the client on which the task of creating a backup using vSphere was performed (see Figure 7).
Figure 7. Client selection
After the Client is selected, the administrator must select the Job (Restore Point) to restore. (see figure 8 on another page). If the selected Job is a valid vSphere task, i.e. can be executed, the third step will display the list of virtual machines included in the FileSet (see Figure 9 on the next page).
At this point, Bacula should create a virtual image of the selected VM. Need to recover a couple of small files from each Job that make up the selected restore point Restore Point. After completing the Bacula software procedure, you need to mount the disk of the selected VM in the system. These steps are usually completed quickly, however, the time taken depends largely on the configuration being used. Indexes are created and maintained during this phase to speed up subsequent restore requests.
After the disk is mounted, the files of the selected VM will be displayed in the file manager. In it you will be able to select files or directories for recovery. (see figure 10 on page 31). The administrator can then create a ZIP or TAR archive. The archive will be created automatically and saved to / opt/bacula/working. A link will be created to securely download the archive via HTTP. The administrator will be able to provide this link to the end user.
Each time the administrator selects files, he will be able to choose the method of recovering the compressed file in tar or zip format. (see figure 11 on page 32). After recovery, it is important to end the session in order to free up the resources used for recovery.
Figure 8. Selecting a restore point
Figure 9. Selecting a VM
Figure 10. File selection
Figure 11. File access
2.3 Notes
2.3.1 Cache directory
To speed up subsequent single file restore procedures, some files created during a restore session are stored in the cache directory.
After a while, you can delete the cache files. If necessary, they will be recreated.
2.4 Restrictions
- The VMware Single File Recovery feature uses the Bacula BVFS interface to display a list of files and directories. In the case of MySQL; despite MySQL's limitations with indexes on TEXT columns, the procedure does not have a significant performance impact on MySQL. However, for best results, we recommend using PostgreSQL.
Two methods can be used to create VMware backups in Handy Backup: internal and external.
Internal method
A copy of Handy Backup is installed on a VMware virtual machine running Windows or Linux. Operating Handy Backup on a virtual machine is no different in principle from using a similar solution on "physical" computers.
External method
Handy Backup runs on a VMware virtual machine server to copy images of specific VMware copies as normal files. Handy Backup uses a special plug-in to back up VMware machines and arrays, which works in "hot" mode (without stopping the VMware machine).
How to save a VMware virtual machine image
Copying a VMware backup image is done using a specialized tool. With the help of the VMware plugin settings, it can also be achieved to stop the copied VMware machine and then restart it for a "cold" copy.
- Open Handy Backup and create a new task by pressing Ctrl+N or by selecting a menu item. Select a backup task.
- On Step 2 select plugin " VMware Workstation".
- Double-click on the line “New configuration” to select a configuration for accessing VMware.
- In the dialog that opens, make a choice between the modes " Hot" (backup without stopping the machine) and " Enable suspend" (with stopping the virtual machine to get its exact image).
- Next, select in the dialog the specific machine image to which this configuration will be applied.
- Click "OK" and continue creating the task as usual.
The sequence of actions described above will stop and then restart VMware virtual machines without any additional intervention.
At the moment, there are several manufacturers of programs for backup storage, both paid and free. We decided that free programs are either inconvenient to use (difficult installation, constant risk of failure, lack of their own interfaces), or they lack the most important backup options. In this case, it is worth purchasing a paid program, which, unlike a free one, will be fully functional with all the basic backup functions. Below is a list of the best backup solutions according to experts:
Veeam BackUp & Replication 5
These programs are the main backup programs used by most users:
Data Recovery with VCenter Server support
As already written in the past, this is the surest way to create a backup of the machine if you bought VCenter Server and no longer have the desire or the means to deal with this issue. This technology is quite easy to set up, a complete guide can be found at the following link:
This solution works both with and without VCenterServer, but it will not be possible to configure the backup by time. We'll cover all the main features just below when we compare all the products.
Emergency System Recovery
Server Consolidation
Licensing type
Specifications
Volume and frequency of backup
This product is now quite popular, since the type of licensing of this product (licensed by sockets) for server rooms with low-power servers will be extremely beneficial. Below we will look at several configurations of servers and consider price characteristics. Also, this product supports the option of instant data recovery after a failure thanks to its vPower technology.
Also recently releases tools for backup in virtual environments. In addition, Symantec is the only one of all backup solutions that uses V2P technology (converting a virtual environment to physical servers). True, Vcenter has such technology, but no longer within the framework of backup technology
But the capabilities of Acronis are not limited to this. Acronis has included another feature in the Acronis Backup & Recovery 10 AdvancedServer VirtualEdition package, this is server consolidation for moving systems from physical to virtual platforms, and with a built-in task scheduler. As a result, we have that this program performs 2 main functions:
Main advantages compared to other technologies:
The ability to work both with physical servers and in a virtual environment, which allows, at the initial stages of the company's development, to combine consolidation with high reliability ratings
Wide range of supported backup storage devices (up to optical devices and magnetic tapes)
Creating an Acronis Secure Zone partition on the same VM server, which allows you to restore the machine in a short time, and this partition will be protected by deduplication mode on another server
If backup is one of our main goals, regardless of price, we should definitely choose between solutions from Veeam, Acronis or Symantec. Both of these products are leaders in data backup and storage, and have a number of individual benefits.
In choosing between these technologies, it is necessary to draw a line on several distinctions:
With different types of licensing, it is worth deciding which one suits us best. If we have powerful servers with a large number of sockets for processors, we should lean in favor of Acronis and Symantec. If we have a lot of weak servers with few sockets, Veeam is the best option.
It is also necessary to understand what we need, that is, what requirements we will impose on backup, the availability of those functions that are especially important to us.
A very important factor is how often and at what time we will backup. If we need to create backup machines for the entire infrastructure, and the frequency is not important, this is one thing (we set the backup scheduler at night). It is completely different if we have one priority machine, for which we need to create a backup with a period of several minutes (we will already need a backup throughout the day).
After determining for a particular case all these parameters, it will not be difficult for us to decide for ourselves which product suits us best.
There is an excellent free script for backing up virtual machines on a VMWare ESXi server, and it works on free versions of ESXi 4 and 5 without installing any additional VMA gimmicks, etc. The only problem is that the instructions there are not entirely accurate, so I fiddled with this script for a long time so that it would still work in automatic mode ...
I will not describe in detail how to connect to ESXi via SSH, I will only describe the setup steps with which everything worked for me.
First, download the script from the link above and upload it to the server, you need to upload it directly in the archive! The easiest way to do this is through the vSphere Client. I have two disks on the server - machines work on one, and all sorts of iso-images and backups themselves lie on the other. The disks are called datastore1 and datastore2 respectively. All backups, script and configs are in the backup folder. Also note that the names of files and folders are case sensitive, so if the folder is called backup, and you write in a script Backup, then it won't work!
- Upload the archive with the script here /vmfs/volumes/datastore2
- Next to SSH cd /vmfs/volumes/datastore2- go to the directory with the script
- Unpacking the script from the archive tar -zxvf archive_filename.tar.gz
- Through vSphere, rename the unpacked folder to something simpler, for example, just backup
- Now let's go to this folder - cd backup
- Create a folder inside it to store individual configs mkdir BackupConfig
- Now in BackupConfig drop the necessary individual configs for machines, if they are not needed and all machines need to be backed up with the same settings, you can leave it empty
- Correct the variables in the configuration file through the vi editor, the main thing is the backup paths, i.e. Change the first line to this: VM_BACKUP_VOLUME=/vmfs/volumes/datastore2/backup, well, then see for yourself what else you need - vi ghettoVCB.conf
- Create script StartBackup.sh(2 lines) - vi StartBackup.sh
2nd line, where the call of the script itself, you can remake for yourself
cd /vmfs/volumes/datastore2/backup
./ghettoVCB.sh -a -g ./ghettoVCB.conf -c BackupConfig -l ghettoVCB.log - Run chmod +x ghettoVCB.sh
- Run chmod +x StartBackup.sh
Stage 1 completed! Now if you run StartBackup.sh, the backup will start. For the duration of debugging, you can change the 2nd line to something like this ./ghettoVCB.sh -a -g ./ghettoVCB.conf -c BackupConfig -l ghettoVCB.log -d dryrun- this will allow you to run the script and track the progress without copying the disks. To backup more efficiently and quickly, I recommend setting the disk type in the settings thin.
Configuring Cron (to automatically run a script)
- Give permission to write to a file chmod +w
- Add a line through vi to /var/spool/cron/crontabs/root
15 0 */3 * * /vmfs/volumes/datastore2/backup/StartBackup.sh
Launches at 00:15 at night every three days. My time zone is +4 Moscow, i.е. actually the script is run at 4:15 am, this will be visible by the date the log was modified through vSphere. Of course, you can choose another time and frequency. - Now you need to run two commands to restart cron
kill $(cat /var/run/crond.pid)
crond - Add with vi 3 lines to the very end of the file /etc/rc.local
This is necessary because after rebooting the server, the contents of the file from the 2nd point with the launch of our script will be restored to the previous state, so in rc.local we indicate that after rebooting, the following commands must be executed - stopping cron, adding a line to automatically run the script and starting cron .
/bin/kill $(cat /var/run/crond.pid)
/bin/echo "15 0 */3 * * /vmfs/volumes/datastore2/backup/StartBackup.sh" >> /var/spool/cron/crontabs/root
crond - Now let's run the command /sbin/auto-backup.sh to make sure all our changes are saved.
A little explanation - why you need to create a script StartBackup.sh, and not just take and put its contents into /var/spool/cron/crontabs/root? There is some limit on the size of this file and some of the lines in it simply will not work, although you can try to do it this way, at first it worked for me, but then, apparently, some patches came out and stopped. Moreover, it's just more convenient - if you need to change the backup schedule, then you just edit the file StartBackup.sh and there is no need to dance with a tambourine around cron with its restart and making the same changes to /etc/rc.local.
PS: Time passes, everything changes, the script itself changes, ESXi5 has already been released, so somewhere, something may no longer work 🙂
Appendix: Cron Syntax
The cron command looks like this:
1 2 3 4 5 /vmfs/volumes/datastore2/backup/StartBackup.sh
Where,
1: Minutes (0-59)
2: Clock (0-23)
3: Days (0-31)
4: Months (0-12 )
5: Day of the week (0-7)
A few examples:
- Run at 5 minutes past midnight, every day
5 0 * * * /vmfs/volumes/datastore2/backup/StartBackup.sh - Launch at 2:15 every first day of the month
15 14 1 * * /vmfs/volumes/datastore2/backup/StartBackup.sh - Start at 22:00 every working day
0 22 * * 1-5 /vmfs/volumes/datastore2/backup/StartBackup.sh - Runs at 23 minutes after midnight and every two hours thereafter (2:23, 4:23… etc.), every third day
23 0-23/2 * * */3 /vmfs/volumes/datastore2/backup/StartBackup.sh
If you have a desire to understand in more detail the processes of setting up and ensuring the integrated security of a local and network infrastructure built on the Linux OS, I recommend getting acquainted with OTUS. The course is not for beginners, for admission you need to pass.
If you have not yet configured the free hyper-v hypervisor, I recommend that you familiarize yourself with my material on this topic -. If you already have it configured and the question arose of how to quickly, conveniently and free of charge back it up, then I recommend using the free HV Backup utility.
HVBackup utility description for hyper-v
The main goal of the HVBackup project was to create a free, simple and effective backup solution for hyper-v virtual machines running both in single mode and in a cluster.
The utility works on all versions of the operating system, from Windows Server 2008 onwards, both in graphical mode and Server Core installations. The utility also supports the free edition of the Hyper-V Server hypervisor.
HVBackup supports app consistent and crash consistent hyper-v backups using VSS built into the system. There are many commercial solutions based on the same method of backing up virtual machines, but among the free ones, I know only this product.
Company codeplex, the manufacturer of the program, claims that it regularly uses HVBackup in its infrastructure, which guarantees high quality work and timely corrections or changes.
The program can be launched from command line, called from Powershell scripts, or be built into any .Net application through the appropriate library.
The principle of operation of this utility is quite simple. The program creates in the specified directory a separate zip archive for each virtual machine, which includes all the information related to it.
System requirements:
.Net Framework 3.5
Installing .Net Framework 3.5 on the command line
To install .Net Framework 3.5 you need to run the following commands in Powershell.
Windows Server or Hyper-V Server 2012:
Install-WindowsFeature NET-Framework-Core
Windows Server 2008 or 2008 R2 (non-core):
Ocsetup NetFx3
Windows Server (core) or Hyper-V Server 2008 / 2008 R2:
Ocsetup NetFx3-ServerCore
HVBackup usage example
Perform a full VSS backup of all host virtual machines:
HVBackup -a -o d:\vm-backup
Perform a full VSS backup of the list of virtual machines (quotes must be used if machine names contain spaces). Backup is performed on a remote server:
HVBackup -l "VM1,VM2" -o \\backup-server\wm-backup
Make a full backup of virtual machines, the list of which is contained in a text file (each VM on a separate line):
HVBackup -f list.txt -o d:\wm-backup
Script for automatic backup
If you want to schedule automatic backups, then it is most convenient to make a simple bat file for this purpose and schedule its execution. Let's create backup-all.cmd the following content:
Set BCKPATH="\\backup-server\wm-backup" rem net use %BCKPATH% /user:
This script will delete zip files older than 7 days in the specified directory before performing a backup of all virtual machines. You can edit the paths and parameters to suit your needs.
Now let's create a scheduled task on the server to periodically perform backups using the previously created script:
Schtasks.exe /create /tn HVBackup /tr c:\backup-all.cmd /sc DAILY /ru
If you want to delete a job:
Schtasks.exe /delete /tn HVBackup /f
Immediate task execution:
Schtasks.exe /run /tn HVBackup
HVBackup command line options
Usage: HVBackup.exe-b, --backup
Run a backup (default).
-r, --restore
Perform recovery.
-f, --file
A text file containing a list of virtual machines, one per line.
-l, --list
List of virtual machines for backup.
-a, -all
Backup of all virtual machines on the server.
-n, --name
If specified, only the listed virtual machines are backed up.
-g, guid
If specified, the listed virtual machines are backed up by GUID.
-o, --output
Required parameter. Destination folder for archives. Can be in the form of a local or network path.
--output format
The format of the virtual machine archive name. (0) machine name, (1) machine GUID, and (2) current date and time. Default: "(0)_(2:yyyyMMddHHmmss).zip"
-s, --singlevss
Make one snapshot for all virtual machines.
--help
Display command help:
Conclusion
That's all. The only remark is that I did not test the performance drop during the backup. By eye, I did not notice any brakes in the operation of virtual machines. But they were all unloaded for me. The time it takes to back up each machine is significant. But of course it depends on many factors.
There is a good opportunity to combine HVBackup and . For example, make archives locally on a server with hyper-v, mount a folder to the linux server and use rsync to grab zip files with virtual machines. Or just do a backup on a Linux samba, for example. The link above has an example on this topic.
Online Linux Security Course
If you have a desire to understand in more detail the processes of setting up and ensuring the security of a local and network infrastructure built on the Linux OS, I recommend that you get acquainted with online course "Linux Security" in OTUS. The training lasts 3 months, after which successful graduates of the course will be able to be interviewed by partners. The course program is designed as a series of practical workshops and is aimed at prepared students who already have knowledge and experience in Linux administration. What will you learn:- Apply the best world practices and information security standards (Debian, RedHat, MitRE);
- Use tools to detect and exploit vulnerabilities in Linux OS;
- Install and configure network systems for detecting and preventing attacks;
- Work with monitoring and containerization systems based on Docker;