Cloud-Init позволяет клонировать виртуальные машины Windows с необходимыми сетевыми настройками (IP, маска подсети, основной шлюз) и данными авторизации. Данный метод позволяет автоматизировать процесс развертывания среды без непосредственной ее настройки при каждом развертывании. В случае с Proxmox взаимодействие Cloud-Init в связке с API или CLI дает безграничные возможности автоматизации.
Cloud–init — это, по сути, программа, которая запускается на гостевой машине при загрузке и ищет данные конфигурации для применения к гостевой системе во время инициализации.
В конце мануала мы получим шаблон виртуальной машина Windows 2019 для Proxmox, который можно клонировать с нужными настройками сети и паролем администратора без непосредственного взаимодействия с системой Windows.
Начнем процесс настройки. Сначала необходимо создать новую виртуальную машину и установить Windows 2019 по инструкции.
После того, как виртуальная машина с MS Windows 2019 на борту развернута добавим следующее оборудование на одноименной вкладке:
- Последовательный порт
2. Диск CloudInit
После стартуем виртуальную машину и подключимся к ней по RDP, настройки сети для гостевой машины были описаны тут.
Скопируем на гостевую машину и установим CloudBaseinitSetup_0_9_11_x64
Не забудьте тут указать порт COM1, который мы только, что создали во вкладке Оборудование.
C:\Program Files\Cloudbase Solutions\Cloudbase-Init\conf
[DEFAULT] username=Administrator groups=Администраторы netbios_host_name_compatibility=true inject_user_password=true first_logon_behaviour=no config_drive_raw_hhd=true config_drive_cdrom=true config_drive_vfat=true locations=cdroom bsdtar_path=C:\Program Files\Cloudbase Solutions\Cloudbase-Init\bin\bsdtar.exe mtools_path=C:\Program Files\Cloudbase Solutions\Cloudbase-Init\bin\ metadata_services=cloudbaseinit.metadata.services.configdrive.ConfigDriveService verbose=true debug=true ntp_use_dhcp_config=true real_time_clock_utc=true ntp_enable_service=true rdp_set_keepalive=true enable_automatic_updates=true logdir=C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\ logfile=cloudbase-init.log default_log_levels=comtypes=INFO,suds=INFO,iso8601=WARN,requests=WARN logging_serial_port_settings=COM1,115200,N,8 local_scripts_path=C:\Program Files\Cloudbase Solutions\Cloudbase-Init\LocalScripts\ plugins=cloudbaseinit.plugins.common.mtu.MTUPlugin, cloudbaseinit.plugins.windows.ntpclient.NTPClientPlugin, cloudbaseinit.plugins.common.sethostname.SetHostNamePlugin, cloudbaseinit.plugins.windows.createuser.CreateUserPlugin, cloudbaseinit.plugins.common.networkconfig.NetworkConfigPlugin, cloudbaseinit.plugins.common.sshpublickeys.SetUserSSHPublicKeysPlugin, cloudbaseinit.plugins.windows.extendvolumes.ExtendVolumesPlugin, cloudbaseinit.plugins.common.setuserpassword.SetUserPasswordPlugin, cloudbaseinit.plugins.common.localscripts.LocalScriptsPlugin allow_reboot=true stop_service_on_exit=false check_latest_version=true
В каталоге C:\Program Files\Cloudbase Solutions\Cloudbase-Init\LocalScripts создадим файл 01ActivateAdministrator.py
import os,json,sys,subprocess,configparser,platform
def find_drive(file_path):
for number in range(65,91):
drive_letter = chr(number)
if os.path.exists(drive_letter+file_path):
return drive_letter+file_path
print("\n Searched file could not be found on any drive with path:" + file_path)
return False
def load_json_file(file_path,variable):
file = open(file_path)
data = json.load(file)
file.close()
return data.get(variable)
def get_administrator_status():
command = "(Get-LocalUser | Where-Object{$_.SID -like \"S-1-5-*-500\"}).Enabled"
run = subprocess.run(["powershell", "-Command", command], stdout=subprocess.PIPE, universal_newlines=True)
print("Is admin account enabled already: " + run.stdout)
return run.stdout
def get_administrator_name():
command = "(Get-LocalUser | Where-Object{$_.SID -like \"S-1-5-*-500\"}).Name"
run = subprocess.run(["powershell", "-Command", command], stdout=subprocess.PIPE, universal_newlines=True)
print("Administrator username: " + run.stdout)
return run.stdout
def enable_administrator_account():
command = "(Get-LocalUser | Where-Object{$_.SID -like \"S-1-5-*-500\"}).Name | Enable-LocalUser"
run = subprocess.run(["powershell", "-Command", command], stdout=subprocess.PIPE, universal_newlines=True)
print("\n Administrator account is activated by localscript")
return run.stdout
def is_os_64bit():
return platform.machine().endswith('64')
def get_data(variable,path):
configParser = configparser.RawConfigParser()
configParser.read(path)
data = configParser.get('DEFAULT',variable)
return data
# variables
meta_data_path = find_drive(":\OPENSTACK\LATEST\META_DATA.json")
admin_name = get_administrator_name()
# execute
if (meta_data_path) and ("admin_username" in load_json_file(meta_data_path,"meta")):
meta_data = load_json_file(meta_data_path,"meta")
meta_username = meta_data["admin_username"]
print("Meta_Data admin_username is :" + meta_username)
else:
if is_os_64bit():
conf_path = r'C:\Program Files\Cloudbase Solutions\Cloudbase-Init\conf\cloudbase-init.conf'
print("System architecture is 64 bit.")
else:
conf_path = r'C:\Program Files (x86)\Cloudbase Solutions\Cloudbase-Init\conf\cloudbase-init.conf'
print("System architecture is 32 bit.")
meta_username = get_data('username', conf_path)
print("Conf username:"+meta_username)
if meta_username in admin_name and "False" in get_administrator_status():
run = enable_administrator_account()
sys.exit(1001)
else:
print("Cloud-init user is not Administrateur/Administrator or Admin account is already enabled, script aborted.")
sys.exit(0)
Создадим пользователя Administrator и добавим его в группу Администраторы
На этом настройки операционной системы Windows окончены, выключим гостевую машину и перейдем к настройкам Proxmox.
Войдем на ноду Proxmox с помощью WinSCP и перейдем в директорию:
/usr/share/perl5/PVE/API2/Qemu.pm
Найдем код:
if (defined(my $cipassword = $param->{cipassword})) {
# Same logic as in cloud-init (but with the regex fixed...)
$param->{cipassword} = PVE::Tools::encrypt_pw($cipassword)
if $cipassword !~ /^\$(?:[156]|2[ay])(\$.+){2}/;
}
И заменим на:
my $conf = PVE::QemuConfig->load_config($vmid);
my $ostype = $conf->{ostype};
if (defined(my $cipassword = $param->{cipassword})) {
# Same logic as in cloud-init (but with the regex fixed...)
if (!(PVE::QemuServer::windows_version($ostype))) {
$param->{cipassword} = PVE::Tools::encrypt_pw($cipassword)
if $cipassword !~ /^\$(?:[156]|2[ay])(\$.+){2}/;
}
}
Аналогично откроем файл для редактирования:
/usr/share/perl5/PVE/QemuServer/Cloudinit.pm
В этом модуле необходимо заменить функцию:
sub configdrive2_metadata {
my ($uuid) = @_;
return <<"EOF";
{
"uuid": "$uuid",
"network_config": { "content_path": "/content/0000" }
}
EOF
}
на
sub configdrive2_metadata {
my ($conf, $vmid, $user, $network) = @_;
my $uuid = Digest::SHA::sha1_hex($user.$network);
my $password = $conf->{cipassword};
my ($hostname, $fqdn) = get_hostname_fqdn($conf, $vmid);
my $startConfig = <<"EOF";
{
"hostname": "$hostname",
"uuid": "$uuid",
"admin_pass": "$password",
EOF
if (defined(my $keys = $conf->{sshkeys})) {
$startConfig .= " \"network_config\": { \"content_path\": \"/content/0000\" },\n";
$keys = URI::Escape::uri_unescape($keys);
$keys = [map { my $key = $_; chomp $key; $key } split(/\n/, $keys)];
$keys = [grep { /\S/ } @$keys];
$startConfig .= " \"keys\": [\n";
$startConfig .= " {\n";
my $keyCount = @$keys;
for (my $i=0; $i < $keyCount; $i++) {
# $startConfig .= " $keyCount "
if ($i == $keyCount-1){
$startConfig .= " \"key-$i\": \"".$keys->[$i]."\"\n";
} else {
$startConfig .= " \"key-$i\": \"".$keys->[$i]."\",\n";
}
}
$startConfig .= " }\n";
$startConfig .= " ]\n";
} else{
$startConfig .= " \"network_config\": { \"content_path\": \"/content/0000\" }\n";
}
$startConfig.= "}";
return $startConfig;
}
А так же заменим в этом же модуле:
if (!defined($meta_data)) {
$meta_data = configdrive2_gen_metadata($user_data, $network_data);
}
на:
if (!defined($meta_data)) {
$meta_data = configdrive2_metadata($conf, $vmid, $user_data, $network_data);
}
Далее в консоли перезагрузим: pvedaemon
systemctl restart pvedaemon.service
В консоли проверим заданные настройки cloudinit коман
qm cloudinit dump 102 user #cloud-config
После применения настроек во вкладке Cloud-Init, пробуем подключиться по RDP к серверу с заданным IP адресом и паролем администратора.
Консультирую по разным конфигурациям 1С, дорабатываю, разрабатываю расширения, отчеты и обработки, администрирую базы 1С на linux.
Ten years ago the word Cloud was the buzz word in every IT conference. The Cloud promised to bring the modern IT business to a new level of opportunities. Back at that time I was an engineer with little IT experience, attracted by the big transformations ahead and wondering where is this going to end. Almost a decade later, we have three types of cloud solutions: on-premise private cloud, public cloud and the mix of both worlds called hybrid cloud. The competition between them and the increased number of business requirements, set the need for new kind of automation tools and products to bridge the gaps and offer seamless experience. With the recent announcement of VMware Cloud Services, VMware is heading strong in this direction. Behind the scenes, VMware Cloud Services is an extensible SaaS solution designed to help us operate, secure and monitor workloads across different cloud providers.
Now that we have the flexibility to choose, which workload goes where, we have to solve another common problem. We need to find out how to customize the guest instance to fit our software application(s).
Thanks to the Cloud-Init package developed by Canonical, this is an easy task for the most popular Linux and FreeBSD operating systems. However, Cloud-Init doesn’t support windows guest instances and this is a problem for all applications that require windows to run.
The good news is that a company called “Cloudbase Solutions” developed equivalent of Cloud-Init for windows. Cloudbase-init provides out of the box: user creation, password injection, static network configuration, hostname, SSH public keys and user-data scripts, such as PowerShell, Batch, Python, Cloud Config and others.
In this blog post tutorial, I will demonstrate how to prepare Azure windows image with cloudbase-init and then customize guest instance provisioned from that image. The procedure is identical for all supported cloud vendors and versions of windows. The tutorial is split in two parts, the image preparation and the guest instance customization.
There are multiple ways to create a custom Azure windows image. To keep things simple for the purpose of this demonstration, I will customize one of the windows 10 Azure marketplace images and then capture it as custom image.
Prepare Cloudbase-init Azure windows 10 image
- Login to the Microsoft Azure Management portal.
- Click Create a Resource and search for windows 10 image in the marketplace search bar.
- Click Create and provide the required inputs to provision new windows guest instance from this image.
Note: Make sure to request public IP address and allow RDP inbound access, since we need them to connect to the new windows 10 instance and install cloudbase-init.
- When the machine is up and running, login with RDP client and follow the steps below to setup cloudbase-init:
- Download the cloudbase-init installation binaries from https://github.com/openstack/cloudbase-init.
Supported cloudbase-init cloud providers (stable version):
- OpenStack (web API)
- OpenStack (configuration drive)
- Amazon EC2
- CloudStack
- OpenNebula
- Ubuntu MaaS
The guest instance metadata is picked up by the so called cloudbase-init metadata services. Then they exposed the data to the cloudbase-init plugins, responsible for the guest instance customization.
For example, configure custom instance host name, network address, username, password and others.Available cloudbase-init plug-ins (stable version):
- Setting host name (main)
- Creating user (main)
- Setting password (main)
- Static networking (main)
- Saving public keys (main)
- Volume expanding (main)
- WinRM listener (main)
- WinRM certificate (main)
- Scripts execution (main)
- Licensing (main)
- Clock synchronization (pre-networking)
- MTU customization (pre-metadata-discovery)
- User data (main)
- Configuring selected plugins
Available cloudbase-init user-data interpretors (stable version):
- PEM certificate
- Batch
- PowerShell
- Bash
- Python
- EC2 format
- Cloud config
- Multi-part content
- Sysnativeness
The guest instance user-data is called with different names across the different cloud providers (Azure: custom data, AWS/Openstack/Ubuntuu MaaS/CloudStack: userdata). This is custom user content exposed to the guest instance by the currently deployed and running cloud infrastructure. Its purpose is to provide additional data for the instance to customize it as much as you need, if the cloud initialization service does support this feature.
- Run the CloudbaseInitSetup_x64 installation binary and follow the on-screen instructions.
-
Click Install and once the installation completes, configure the cloudbase-init config, before finalizing the installation.
-
Navigate to the chosen installation path and under the conf directory, edit the file “cloudbase-init-unattend.conf”
The various config options above are used by the cloudbase-init services and plugins to customize the guest instance user experience. - Save the file and finish the cloudbase-init installation using the options below.
- Once the sysprep process is complete, the windows 10 machine will shut down automatically. Wait until its state in the Azure Management portal is set to Stopped.
- Click Capture and provide the required inputs to convert the windows 10 instance to new custom image.
- Note down the image name and click Create.
Now that we have the image created, we can easily customize the guest instance based on the predefined options in the cloudbase-init config file or via the optional instance user-data. To achieve this, we just need to request new instance from that image via the Azure Management portal, the Azure API, the Azure CLI or via the VMware Cloud Assembly blueprint.
I will use the last option as a good opportunity to show how this can be done with VMware Cloud Services. Moreover, I will create a cloud agnostic blueprint, because it will let me deploy and customizable instance in other cloud providers, as soon as I have their cloudbase-init windows image available.
- Download the cloudbase-init installation binaries from https://github.com/openstack/cloudbase-init.
Deploy and customize windows guest instance with VMware Cloud Services.
- Login to VMware Cloud Services and select VMware Cloud Assembly.
- Navigate to Infrastructure tab and setup an Azure Cloud account.
- The next step is to create Cloud Zone associated with the Cloud Account.
Note: Add a capability tag, which will be later on used to link our new blueprint with this cloud zone.
- Next we need to create new Project associated with the Cloud Zone.
- To configure the same image size across the different cloud vendors, we need to create a Flavor Mapping.
- Similar to the Flavor Mapping, we need to create new Image Mapping to describe the image name across the different cloud vendors.
If you don’t see the image created in step 1, go to Cloud Accounts and re-sync all images. By default image data-collection is automatically performed every 24 hours.
-
Create Storage Profile to allow Cloud Assembly provisioning of Azure images based on managed disks.
By default Cloud Assembly will create Azure classic Availability Set, which doesn’t support images based on managed disks, so we need to set Cloud Assembly to create Azure managed Availability Set for this to work.
-
Navigate to the Blueprints tab and create new Cloud Assembly blueprint.
YAML code:
Most of the YAML properties are self-explanatory, so I will comment just the few that are more specific to the topic.
The remoteAccess properties will set the guest instance metadata with the values provided for username, password and authentication. Then this will be picked-up by the cloudbase-init metadata services and the guest OS will be configured via the Setting Password cloudbase-init plug-in.
The cloudConfig property will set the userdata and expose it to the guest instance during provisioning. In this case the first line indicates that the script below is Cloud-Init YAML code and cloudbase-init will use its cloud-init plug-in to execute it.
The _cloudConfigContentPreserveRawValue is an important one, since it tells Cloud Assembly to keep the raw user-data provided in the cloudConfig property. If we don’t set this property to true, then Cloud Assembly will convert some of the other properties to Cloud-Init YAML code, for example it will create the Cloud-Init YAML code needed to set the remote access authentication type, username and password.
The storage constraint, tells Cloud Assembly that we want to use the custom Storage profile that requires Azure managed Availability Set. - Click Deploy to provision new Azure windows 10 machine, which will be customized by the configured CloudConfig script.
- Login to the new instance with RDP client and the credentials specified in the blueprint and verify that the CloudConfig script was successfully executed.
- Check the custom file created in C:\test
- Check the customized machine hostname
- Check the custom file created in C:\test
I hope you found the blog post useful!
If you want to learn more, check out the VMware Cloud Services and Cloudbase-Init documentation pages and stay tuned for more on VMware Blogs!
Инструмент cloud-init позволяет настроить конфигурацию облачного сервера в момент его создания.
Все операционные системы на базе Linux, предоставляемые для серверов проекта CLO, поддерживают инициализацию с помощью cloud-init.
Для инициализации серверов под управлением Microsoft Windows Server используется аналогичная по функциям утилита cloudbase-init.
С помощью cloud-init можно автоматизировать выполнение целого ряда операций над сервером, например:
- настройка конфигурации облачных серверов
- создание пользователей и групп, предоставление пользовательских прав
- добавление сертификатов и ключей доступа
- установка и запуск нужного программного обеспечения на сервере
- автоматическая установка обновлений и управление их источниками
- создание директорий, настройка доступа к данным
Подготовка скрипта cloud-init для CLO
Данные для cloud-init помещаются в специально подготовленный файл (в дальнейшем — скрипт) инициализации. Скрипты cloud-init записываются в формате YAML.
Подробная информация о конфигурировании cloud-init доступна в официальной документации по ссылке.
Примечание. Для утилиты cloudbase-init, используемой в Microsoft Windows Server, также существует подробное руководство.
Для передачи скрипта на сервер необходимо перевести его текст в строку, закодированную в формате base64. Для этого можно использовать утилиту командной строки или любой подходящий онлайн-кодировщик.
Вызов cloud-init
Инструмент cloud-init вызывается с помощью программного интерфейса Public API. Скрипт передаётся при создании нового сервера с помощью параметра user_data. Значением этого параметра является строка в формате base64, содержащая закодированный скрипт.
Пример использования cloud-init
Приведённый далее пример скрипта добавляет на сервер ключ для доступа по SSH, который становится доступен при перезагрузке сервера.
Шаг 1. Напишите скрипт инициализации в формате YAML:
#cloud-config
packages:
- wget
- pwgen
ssh_authorized_keys:
- ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnKO3KL0ZCegdL6bRqiiVxcE8ECVCb2yya4OpbZcOYaZXYao/GT9+t+P5pufF5nvkrGgfgh5TJYwGu9lrcZCL+IHh4U9KeacZiHa3IWroC33+2Nu3H+T3mxgofQn49WGDT6Y4Uew9+Gv1sGLVTdut7PjtSwUYyP7e/yoAxkt3qasdfXQahqWrQRNEhCe6lyrTEIuPdHjnkE/FJuifRMFOSmigjdBfff7VsuLRR8F0CIx8dtqYyKEMNVMAe86VzEW0Q1k0Hk+mMQzHbV67CJp6kTlO3hk7QFdPVCcsqbG0EvADBciuVHvbmnWfQOmWhHsiXhdoYlAOqlX/PysYzaDgc+Z some_ssh_key
Шаг 2. Закодируйте текст скрипта в формате base64.
Шаг 3. Сформируйте запрос к API на создание сервера и передайте сформированную строку в теле запроса как значение параметра user_data.
{"name":"cloud_init_test",
"image": "5f94e693-3917-42fc-802d-51901080712b",
"flavor": {"vcpus":1, "ram": 2},
"storages": [{"bootable": true, "size": 10}],
"addresses": [{"version": 4, "external": true}],
"user_data": "user_data: I2Nsb3VkLWNvbmZpZwpwYWNrYWdlczoKICAtIHdnZXQKICAtIHB3Z2VuCnNzaF9hdXRob3JpemVkX2tleXM6CiAgLSBzc2gtcnNhIEFBQUFCM056YUMxeWMyRUFBQUFEQVFBQkFBQUJBUURuS08zS0wwWkNlZ2RMNmJScWlpVnhjRThFQ1ZDYjJ5eWE0T3BiWmNPWWFaWFlhby9HVDkrdCtQNXB1ZkY1bnZrckdnZmdoNVRKWXdHdTlscmNaQ0wrSUhoNFU5S2VhY1ppSGEzSVdyb0MzMysyTnUzSCtUM214Z29mUW40OVdHRFQ2WTRVZXc5K0d2MXNHTFZUZHV0N1BqdFN3VVl5UDdlL3lvQXhrdDNxYXNkZlhRYWhxV3JRUk5FaENlNmx5clRFSXVQZEhqbmtFL0ZKdWlmUk1GT1NtaWdqZEJmZmY3VnN1TFJSOEYwQ0l4OGR0cVl5S0VNTlZNQWU4NlZ6RVcwUTFrMEhrK21NUXpIYlY2N0NKcDZrVGxPM2hrN1FGZFBWQ2NzcWJHMEV2QURCY2l1Vkh2Ym1uV2ZRT21XaEhzaVhoZG9ZbEFPcWxYL1B5c1l6YURnYytaIHNvbWVfc3NoX2tleQo="
}
Технические ограничения
Внимание! Ошибочные настройки параметров сервера с помощью cloud-init могут привести к некорректной работе созданного сервера.
Параметр user_data используется для передачи в создаваемые серверы информации о предустановленном ПО и о ключах для доступа по SSH. Поэтому невозможно использовать параметр user_data одновременно с keypairs и recipe в теле запроса на создание сервера.
WinCloudInit
CloudInit for Windows
WinCloudInit provides a framework for early guest (Windows) initialization
similar to CloudInit.
In VMware and Hyper-V environments it replaces guest customization with
more flexible and extendable mechanism.
Compared to CloudbaseInit it’s
written purely on PowerShell.
List of bundled plugins:
- 01-reboot.ps1
- 02-sysprep.ps1
- 04-network.ps1
- 08-hostname.ps1
- 10-posh.ps1
- 11-cert.ps1
- 12-winrm.ps1
- 16-rdp.ps1
- 20-firewall.ps1
- 24-activation.ps1
- 28-disk.ps1
- 32-user.ps1
Currently the only supported config source is cloud-config.json file stored on floppy/cdrom or C:\cloud-config folder:
{
"HostName": "WINCLOUDINIT",
"HDD": [
{
"Capacity": 50,
"DeviceNode": "scsi0:0",
"Uuid": "6000c2913a13b84ecf0ba5000ddc71d2"
},
{
"Capacity": 100,
"Label": "Data",
"MountPoint": "C:\data",
"ClusterSizeKB": 64,
"DeviceNode": "scsi1:0",
"Uuid": "6000c29d238336928f301e14245e80b4"
}
],
"NIC": [
{
"Ip": [
"10.240.157.10/24",
"10.240.157.11/24"
],
"Gw": "10.240.157.1",
"Name": "PrivateConnection",
"Mac": "00:50:56:96:1d:27"
},
{
"Ip": [
"10.250.157.10/24",
],
"Name": "PrivateConnection2",
"Mac": "00:50:56:96:1d:28"
}
],
"DNS": {
"DomainSearch": [
"domain.local",
"company.net"
],
"Servers": [
"8.8.8.8"
]
},
"Sysprep": {
"Org": "Company",
"Owner": "SysAdmin",
"TimeZone": "PST"
},
"ProductKey": "D2N9P-3P6X9-2R39C-7RTCD-MDVJX",
"WinRM": {
"Https": true,
"Certificate": "812FF641630C82CFC1114597409DB086FA43E480",
"UserMapping": {
"automation": "5CBF402EA2AA8481FB7677705471E84C32A9833B"
}
},
"ExecutionPolicy": "Restricted",
"RDP": false,
"Firewall": {
"Disabled": false
},
"Certificates": [
{
"File": "wincloudinit-winrm.pfx",
"Store": [ "LocalMachine\\My" ],
"Password": [
"tV+j7zZWctl3HvccGzJUetBblhV5qOjhaPdlvr/FFSZEWBmyCoYCB6A0V0iYyqIU",
"4JrU7pljAnzvccXK1eh4PD4yr6S9wj/7bBynxsYGH2YlYI4uDwv5sFUP8p2kwNAd",
"qafVuuItZFWzNDeg/Ta/w+UJbXjsXn7+PfAc5wo1sMX9tgUV7G6FJh7kVwzxl7Ax",
"/Oegu0a94fUMSZJaU8cJJ4JFUHpopmtSkOQyVxykHPKYCX8njdaabnRwhzc0jugU",
"+l5pCx0ljpEJbkVtfbOHij5IZFD0AKsoWJg1Uzmfjs7hwcFQmDaXUajBK3Fq5XNs",
"WiQIKnXzk5ppTxJEL66KtQ=="
]
},
{
"File": "GeoTrust Global CA.cer",
"Store": [ "LocalMachine\\CA" ]
},
{
"Url": "https://www.server.com/automation.cer",
"Store": [ "LocalMachine\\Root", "LocalMachine\\TrustedPeople" ]
}
],
"Groups": [ "PowerUsers" ],
"Users": [
{
"Name": "admin",
"OldName": "Administrator",
"Password": [
"OV+j7zZWctl3HvccGzJUetBblhV5qOjhaPdlvr/FFSZEWBmyCoYCB6A0V0iYyqIU",
"4JrU7pljAnzvccXK1eh4PD4yr6S9wj/7bBynxsYGH2YlYI4uDwv5sFUP8p2kwNAd",
"qafVuuItZFWzNDeg/Ta/w+UJbXjsXn7+PfAc5wo1sMX9tgUV7G6FJh7kVwzxl7Ax",
"/Oegu0a94fUMSZJaU8cJJ4JFUHpopmtSkOQyVxykHPKYCX8njdaabnRwhzc0jugU",
"+l5pCx0ljpEJbkVtfbOHij5IZFD0AKsoWJg1Uzmfjs7hwcFQmDaXUajBK3Fq5XNs",
"WiQIKnXzk5ppTxJEL66KtQ=="
]
},
{
"Name": "automation",
"Password": [
"PV+j7zZWctl3HvccGzJUetBblhV5qOjhaPdlvr/FFSZEWBmyCoYCB6A0V0iYyqIU",
"4JrU7pljAnzvccXK1eh4PD4yr6S9wj/7bBynxsYGH2YlYI4uDwv5sFUP8p2kwNAd",
"qafVuuItZFWzNDeg/Ta/w+UJbXjsXn7+PfAc5wo1sMX9tgUV7G6FJh7kVwzxl7Ax",
"/Oegu0a94fUMSZJaU8cJJ4JFUHpopmtSkOQyVxykHPKYCX8njdaabnRwhzc0jugU",
"+l5pCx0ljpEJbkVtfbOHij5IZFD0AKsoWJg1Uzmfjs7hwcFQmDaXUajBK3Fq5XNs",
"WiQIKnXzk5ppTxJEL66KtQ=="
],
"Groups": [ "PowerUsers", "Administrators" ]
}
]
}
DeviceNode HDD field is required on Hyper-V platform
Uuid HDD field is required on VMware platform
To install the module:
- download from GitHub and place into ‘C:\Program Files\WindowsPowerShell\Modules’
(requires system-wide location) - OR grab it from PowerShell Gallery with
Install-Module WinCloudInit -Scope AllUsers
To enable WinCloudInit upon system reboot run Set-WinCloudInit -Enabled, you will
be prompted for Administrator credentials (after sysprep module will switch to SYSTEM
account).
Put unencrypted private RSA key (private.pem) to module openssl subfolder so that
WinCloudInit can decrypt user/certificate passwords, supplied via cloud-config.json
Log is stored in C:\Windows\Temp\WinCloudInit-#date#.log
The module targets PowerShell v4 installations and has been tested on:
- Windows Server 2008 R2
- Windows Server 2012 R2
- Windows Server 2016
- Windows Server 2019
For developers
To develop a new plugin write a PowerShell script that starts with:
[CmdletBinding()]
param(
$Config
)
Inject the necessary configuration data in cloud-config.json so it is exposed
to your source through $Config variable. Prepend the name
with double-digit number according to the order when the plugin is intended
to be run and put it into the plugins folder.
If your plugin is going to reboot the system, prior to restart send ‘reboot’
string to stdout so the module can suspend execution of the next plugin
and resume after system has been restarted. To handle reboots the module keeps
a state file in the module directory where it stores a current execution step.
To reset the state run Set-WinCloudInit -ResetState
openssl rsautl
generate rsa key pair
openssl genrsa -out keypair.pem -aes128 4096
export public key
openssl rsa -in keypair.pem -outform PEM -pubout -out public.pem
export unencrypted private key (to be stored in a template)
openssl rsa -in keypair.pem -out private.pem -outform PEM
encrypt data
windows
New-Password | cmd '/c openssl rsautl -inkey public.pem -pubin -encrypt | openssl enc -base64'
linux
echo 'password' | openssl rsautl -inkey public.pem -pubin -encrypt | openssl enc -base64 > encrypted
decrypt data
windows
cat encrypted | cmd '/c openssl enc -d -base64 | openssl rsautl -inkey private.pem -decrypt'
linux
cat encrypted | openssl enc -d -base64 | openssl rsautl -inkey private.pem -decrypt
In the last post I covered deploying Linux-based Proxmox instances using Terraform, leveraging Cloud-Init and cloud-config to register these instances with SaltStack.
The instances in the previous post were both Linux distributions (Debian and Fedora). This is because Cloud-Init was initially built for Linux, and then ported to other Unix-like systems (e.g. FreeBSD, OpenBSD and more).
Thanks to a company called Cloudbase Solutions an equivalent of Cloud-Init is also available for Windows, known as Cloudbase-Init.
Cloudbase-Init
Cloudbase-Init is written in Python, and was originally developed to allow customization of Windows images on OpenStack. However it also supports a number of different metadata services (everything from AWS and Azure to VMware).
Cloudbase-Init also supports a subset of the cloud-config format, including write_files (i.e. creating files on the instance), users, groups, runcmd (i.e. running arbitrary commands) and more.
Preparing the template machine
The first step before creating your instances is to create a template Windows image. This image can then be used as a base image for other instances (using cloning).
Cloudbase Solutions provide tools to build Windows images ready to be deployed with Cloudbase-Init installed, but they rely on Hyper-V on Windows. If you do not have access to a Windows machine to generate the images, do not have access to Hyper-V, or simply want to go through the process yourself, then you can install Cloudbase-Init in a Windows instance yourself.
Installing Windows
In this post, I am using Windows Server Core 2019 (i.e. without the Desktop Experience installed). As Proxmox uses KVM as its underlying hypervisor, you will also need to download the VirtIO-Win ISO. This contains all the necessary drivers for Windows for VirtIO-based NICs, storage and serial devices. Without this, you would need use emulated devices (i.e. emulated SATA or network devices), which are less efficient than the VirtIO equivalents.
First, download the VirtIO ISO and place it in the /var/lib/vz/templates/iso directory on your Proxmox host. This is where Proxmox expects to find all ISO images by default.
After this, place the ISO for your Windows version of choice in the same directory. If you already have the ISOs, then use something like scp or rsync to place it on the Proxmox server. Alternatively, you can download trial versions of Windows from the Microsoft Evaluation Center. These trial versions can be activated later with the appropriate license key and service contract with Microsoft.
Now create the instance. The settings in the below images should work on a default Proxmox install, but you may need to change them if your environment differs (e.g. using Ceph/ZFS for storage, OpenVSwitch for networking): —
Create the VM
General Options
Choose the OS
System hardware (Disk controller and graphics
Create the disk
Choose the CPU and core count
Choose the amount of memory
Choose the network
After creating the instance, select it, go to Hardware, and then add a CD-ROM drive: —
Choose the virtio-win.iso — this contains all the necessary Windows drivers: —
When you start the machine, you should now be presented with the following: —
Loading files
Booting
Choose your language and timezone
Click Install Now
Choose which version (Desktop Experience = GUI)
Choose Custom installation (because there is no existing installation to upgrade)
At this point, you will see that there is no hard disk to install to. This is because Windows does not include the VirtIO drivers by default: —
We can resolve this by clicking on Load Driver, which will give the option to browse for drivers: —
Navigate to the driver with the virtio-win-0.x.x ISO: —
Scroll down to viostor: —
Select the driver and click on Next: —
You should now see the disk to install to: —
At this point, you can either click Next and finish the installation, or you can click Load Driver to add more VirtIO drivers. The main ones to install are: —
- Balloon — This allows for dynamic memory management, optimizing memory usage where appropriate
- NetKVM — This installs the network drivers for the VirtIO network card
- qxldod — If you require graphics drivers
- vioserial — This installs serial drivers, which we will need to use later for Cloudbase-Init (mainly for logging)
You can choose to install the drivers later, using either the GUI if you chose the desktop experience, or using pnputil if using Server Core.
Once done, click Next to finish the installation: —
Once Windows is installed and restarted, you will be given the option to create a password for the Administrator user: —
Unlock the instance (using Ctrl, Alt and Delete
Proxmox Ctrl, Alt and Delete in NoVNC
Confirm changing of the password
Change the password
Logged in as the Administrator
At this point, we are now ready to customize the image.
Customization
To prepare the image, the minimum we require for Cloud-Init is the Cloudbase-Init utility. Before installing Cloudbase-Init I also do the following: —
- Install OpenSSH Server for remote management
- Install Salt
- Disable the firewall and enable RDP
The reason for disabling the firewall is that in the environments I work in, host-based firewalls are not used. Instead, they are managed either via network firewalls or via security groups (if used within a cloud provider). If you require the firewall to be enabled, then you will need to customize the firewall rules to match your environment.
Installing OpenSSH Server
To install the OpenSSH Server on Windows 2019, you can follow the steps in this post. We are then able to SSH into the machine: —
$ ssh [email protected]
[email protected]'s password:
Microsoft Windows [Version 10.0.17763.737]
(c) 2018 Microsoft Corporation. All rights reserved.
administrator@WIN2019-01 C:\Users\Administrator>
Install Salt
To install the Salt Minion, again you can follow the steps in this post. However, do not follow the steps for updating the minion configuration post-install. The minion configuration will be customized by Cloudbase-Init based upon the provided User Data.
Disable the firewall and enable RDP
While most tasks can be completed using Powershell in Windows Core, there is no inbuilt text editor. You can either install something like ViM or you can use RDP so that notepad.exe is available for file editing.
We also disable the firewall at this point, as otherwise we will need to add an RDP-specific rule to the firewall to allow us to access the machine.
# Disable the firewall
Set-NetFirewallProfile -Profile Domain,Public,Private -Enabled false
# Enable RDP
Set-ItemProperty -Path 'HKLM:\System\CurrentControlSet\Control\Terminal Server' -name "fDenyTSConnections" -value 0
You can also use the sconfig tool to enable RDP as well: —
PS C:\Users\Administrator> sconfig
Microsoft (R) Windows Script Host Version 5.812
Copyright (C) Microsoft Corporation. All rights reserved.
Inspecting system...
===============================================================================
Server Configuration
===============================================================================
1) Domain/Workgroup: Workgroup: WORKGROUP
2) Computer Name: TEST
3) Add Local Administrator
4) Configure Remote Management Enabled
5) Windows Update Settings: DownloadOnly
6) Download and Install Updates
7) Remote Desktop: Disabled
8) Network Settings
9) Date and Time
10) Telemetry settings Unknown
11) Windows Activation
12) Log Off User
13) Restart Server
14) Shut Down Server
15) Exit to Command Line
Enter number to select an option: 7
(E)nable or (D)isable Remote Desktop? (Blank=Cancel) E
1) Allow only clients running Remote Desktop with Network Level Authentication (more secure)
2) Allow clients running any version of Remote Desktop (less secure)
Enter selection: 2
Enabling Remote Desktop...
Installing Cloudbase-Init
To install Cloudbase-Init, go to the Cloudbase-Init product page, scroll down to Downloads, and choose the version appropriate for your environment. I am running Windows 2019 on a 64-bit x86 processor, so I chose the Stable Cloudbase-init x64 version.
The following steps will download and then install Cloudbase-Init: —
# Download the file
PS C:\Users\Administrator> Invoke-WebRequest -uri https://cloudbase.it/downloads/CloudbaseInitSetup_Stable_x64.msi -outfile cloudbase-init.msi
# Install the MSI file non-interactively
PS C:\Users\Administrator> msiexec /i cloudbase-init.msi /qn /l*v log.txt
You can install Cloudbase-Init interactively and follow the on-screen instructions to customize the configuration. However we will need to change some options within the Cloudbase-Init configuration anyway, so there is little to benefit in using the GUI installer to configure the Cloudbase-Init service.
We can check that it is installed using: —
PS C:\Users\Administrator> Get-Service cloudbase-init
Status Name DisplayName
------ ---- -----------
Stopped cloudbase-init cloudbase-init
Configuring Cloudbase-Init
The default configuration files for Cloudbase-Init enable a number of different metadata services, most of which are not applicable to Proxmox.
We also need to enable the User-Data plugin otherwise Cloudbase-Init will not be able access any configuration we provide via Cloud-Init. The User Data can be in the form of Batch scripts (i.e. cmd.exe), Powershell, BASH (if it is installed natively under Windows), Python and cloud-config.
The following configuration files tell Cloudbase-Init to use the ConfigDrive2 format for user-data (the default format for a Windows guest on Proxmox), and enables the plugins required to make changes to the network, files and source changes from user-data: —
C:\Program Files\Cloudbase Solutions\cloudbase-init\conf\cloudbase-init.conf
[DEFAULT]
username=Admin
groups=Administrators
inject_user_password=true
config_drive_raw_hhd=true
config_drive_cdrom=true
config_drive_vfat=true
bsdtar_path=C:\Program Files\Cloudbase Solutions\Cloudbase-Init\bin\bsdtar.exe
mtools_path=C:\Program Files\Cloudbase Solutions\Cloudbase-Init\bin\
verbose=true
debug=true
logdir=C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\
logfile=cloudbase-init.log
default_log_levels=comtypes=INFO,suds=INFO,iso8601=WARN,requests=WARN
logging_serial_port_settings=COM1,115200,N,8
mtu_use_dhcp_config=true
ntp_use_dhcp_config=true
local_scripts_path=C:\Program Files\Cloudbase Solutions\Cloudbase-Init\LocalScripts\
check_latest_version=true
metadata_services=cloudbaseinit.metadata.services.configdrive.ConfigDriveService
plugins=cloudbaseinit.plugins.common.networkconfig.NetworkConfigPlugin,
cloudbaseinit.plugins.common.setuserpassword.SetUserPasswordPlugin,
cloudbaseinit.plugins.common.userdata.UserDataPlugin,
cloudbaseinit.plugins.common.sethostname.SetHostNamePlugin,
cloudbaseinit.plugins.common.mtu.MTUPlugin,
cloudbaseinit.plugins.windows.extendvolumes.ExtendVolumesPlugin
C:\Program Files\Cloudbase Solutions\cloudbase-init\conf\cloudbase-init-unattend.conf
[DEFAULT]
username=Admin
groups=Administrators
inject_user_password=true
config_drive_raw_hhd=true
config_drive_cdrom=true
config_drive_vfat=true
bsdtar_path=C:\Program Files\Cloudbase Solutions\Cloudbase-Init\bin\bsdtar.exe
mtools_path=C:\Program Files\Cloudbase Solutions\Cloudbase-Init\bin\
verbose=true
debug=true
logdir=C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\
logfile=cloudbase-init-unattend.log
default_log_levels=comtypes=INFO,suds=INFO,iso8601=WARN,requests=WARN
logging_serial_port_settings=COM1,115200,N,8
mtu_use_dhcp_config=true
ntp_use_dhcp_config=true
local_scripts_path=C:\Program Files\Cloudbase Solutions\Cloudbase-Init\LocalScripts\
check_latest_version=false
metadata_services=cloudbaseinit.metadata.services.configdrive.ConfigDriveService
plugins=cloudbaseinit.plugins.common.mtu.MTUPlugin,
cloudbaseinit.plugins.common.sethostname.SetHostNamePlugin,
cloudbaseinit.plugins.windows.extendvolumes.ExtendVolumesPlugin,
cloudbaseinit.plugins.common.networkconfig.NetworkConfigPlugin,
cloudbaseinit.plugins.common.userdata.UserDataPlugin
allow_reboot=false
stop_service_on_exit=false
You can either edit these files in notepad, or you can use SCP (as we enabled SSH already) to transfer the files from another machine.
We also set all Cloudbase-Init logs to be exposed on the Serial port, allowing us to see the changes being made by Cloudbase-Init (as well as any errors/warnings) separately from the boot and login process itself on screen/VNC.
After this is done, you can then run sysprep, which “generalizes” a Windows image. This removes install-specific information (e.g. passwords, unique IDs etc) in preparation for the image to be reused/cloned. The Cloudbase-Init installation also includes an Unattend answer file. This is similar to a Kickstart or PXE script in the Linux world, providing answers to questions usually presented to users during initial setup.
The included Unattend file looks like the below: —
<?xml version="1.0" encoding="utf-8"?>
<unattend xmlns="urn:schemas-microsoft-com:unattend">
<settings pass="generalize">
<component name="Microsoft-Windows-PnpSysprep" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xm
lns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<PersistAllDeviceInstalls>true</PersistAllDeviceInstalls>
</component>
</settings>
<settings pass="oobeSystem">
<component name="Microsoft-Windows-Shell-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" x
mlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State">
<OOBE>
<HideEULAPage>true</HideEULAPage>
<NetworkLocation>Work</NetworkLocation>
<ProtectYourPC>1</ProtectYourPC>
<SkipMachineOOBE>true</SkipMachineOOBE>
<SkipUserOOBE>true</SkipUserOOBE>
</OOBE>
</component>
</settings>
<settings pass="specialize">
<component name="Microsoft-Windows-Deployment" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xm
lns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<RunSynchronous>
<RunSynchronousCommand wcm:action="add">
<Order>1</Order>
<Path>cmd.exe /c ""C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\Scripts\cloudbase-init.exe" --config-file "C:\Program Files\Cloudbase
Solutions\Cloudbase-Init\conf\cloudbase-init-unattend.conf" && exit 1 || exit 2"</Path>
<Description>Run Cloudbase-Init to set the hostname</Description>
<WillReboot>OnRequest</WillReboot>
</RunSynchronousCommand>
</RunSynchronous>
</component>
</settings>
</unattend>
To start the sysprep process using this Unattend file, run: —
# Go to the Cloudbase-Init directory
PS C:\> cd 'C:\Program Files\Cloudbase Solutions\Cloudbase-Init\conf'
# Run sysprep using the unattend.xml
PS C:\Program Files\Cloudbase Solutions\Cloudbase-Init\conf> c:\Windows\System32\Sysprep\sysprep.exe /generalize /oobe /unattend:Unattend.xml
This will prepare the instance and tell the system to run cloudbase-init.exe on boot.
Creating a template
There are two ways to convert the machine into a template in Proxmox. The first is to right click on the created machine, and select Convert to template: —
The other is to find the ID of the machine in the Proxmox CLI, and then do the following: —
# Retrieve the ID
root@pve-01:~# qm list
VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID
9003 win2k19-cloudinit stopped 2048 32.00 0
# Convert to a template
root@pve-01:~# qm template 9003
We can now use this template when defining an instance within Terraform
Terraform
Create the Cloud-Init template
The first step to building an instance using Terraform is to prepare the Cloud-Init template.
There are some caveats to be aware of before we start: —
- Currently not all
cloud-configmodules are supported in Cloudbase-Init - The format that Proxmox provides some of the Cloud-Init data is not 100% compatible with Cloudbase-Init
In regards to the first, Cloudbase-Init supports the following cloud-config modules (at the time of writing): —
write_files— Creating files (e.g. configuration files)set_timezoneset_hostnamegroups— Creating local groups on the machine, as well as assigning existing users to themusers— Creating local usersntp— Setting NTP serversruncmd— Running a set of arbitrary commands, executed withcmd.exe(not Powershell)
In regards to the latter, there is an issue with how Proxmox provides the DNS Nameservers. Proxmox provides dns_nameservers as the option, whereas Cloudbase-Init expects dns-nameservers (associated GitHub Issue). This issue, combined with Cloudbase-Init not supporting cloud-config network module (to customize our network configuration) means that we must rely on DHCP.
In most cloud environments, IP addressing normally ephemeral (i.e. not persistent). If you treat the instance in this manner, then DHCP provides the same sort of functionality. However if you are expecting to be able to use static addressed instances, then you may need to look at other methods than cloud-config.
The basic template we are going to use looks like the below: —
Content-Type: multipart/mixed; boundary="===============1598784645116016685=="
MIME-Version: 1.0
--===============1598784645116016685==
Content-Type: text/cloud-config; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="cloud-config"
write_files:
- path: c:\salt\conf\minion
content: |
master: salt-master.${domain}
id: ${hostname}.${domain}
nodename: ${hostname}
startup_states: highstate
--===============1598784645116016685==
Content-Type: text/x-cfninitdata; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="cfn-userdata"
#ps1
Restart-Service salt-minion
We use a feature called multi-part content to leverage both cloud-config and Powershell to configure the instance. You could extend this to use more complex Powershell scripts, setting the timezone, creating users and more. In our file, we update the Salt Minion configuration (using the write_files module in cloud-config), and then restart the salt-minion service (using Powershell).
Define the instance
The instance is defined similarly to the instances in the previous post: —
# Source the Cloud Init Config file
data "template_file" "cloud_init_win2k19_vm-01" {
template = "${file("${path.module}/files/cloud_init_win2k19.cloud_config")}"
vars = {
hostname = "win2k19-vm-01"
domain = "yetiops.lab"
}
}
# Create a local copy of the file, to transfer to Proxmox
resource "local_file" "cloud_init_win2k19_vm-01" {
content = data.template_file.cloud_init_win2k19_vm-01.rendered
filename = "${path.module}/files/user_data_cloud_init_win2k19_vm-01.cfg"
}
# Transfer the file to the Proxmox Host
resource "null_resource" "cloud_init_win2k19_vm-01" {
connection {
type = "ssh"
user = "root"
private_key = file("~/.ssh/id_rsa")
host = "pve-01.yetiops.lab"
}
provisioner "file" {
source = local_file.cloud_init_win2k19_vm-01.filename
destination = "/var/lib/vz/snippets/cloud_init_win2k19_vm-01.yml"
}
}
# Create the VM
resource "proxmox_vm_qemu" "win2k19-vm" {
## Wait for the cloud-config file to exist
depends_on = [
null_resource.cloud_init_win2k19_vm-01
]
name = "win2k19-vm-01"
target_node = "pve-01"
# Clone from debian-cloudinit template
clone = "win2k19-cloudinit"
#os_type = "cloud-init"
# Cloud init options
cicustom = "user=local:snippets/cloud_init_win2k19_vm-01.yml"
ipconfig0 = "ip=dhcp"
memory = 2048
agent = 1
# Set the boot disk paramters
bootdisk = "scsi0"
scsihw = "virtio-scsi-pci"
disk {
id = 0
size = 20
type = "scsi"
storage = "local-lvm"
storage_type = "lvm"
iothread = true
}
# Set the network
network {
id = 0
model = "virtio"
bridge = "vmbr0"
}
# Ignore changes to the network
## MAC address is generated on every apply, causing
## TF to think this needs to be rebuilt on every apply
lifecycle {
ignore_changes = [
network
]
}
}
One notable difference is that we set the ipconfig0 = "ip=dhcp" option. Without this, the Terraform provider will crash. We should be able to provide a static IP here, but as noted previously, the format Cloudbase-Init provides DNS nameservers in is not compatible with Cloudbase-Init, meaning that we would not be able to access any hostname-based service (e.g. our Salt server, or downloading dependencies).
Other than this, we should now be able to build a Windows instance.
Build the instance
Now that we have defined the instance within Terraform, we can go ahead and build it: —
$ terraform apply
data.template_file.cloud_init_win2k19_vm-01: Refreshing state...
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# local_file.cloud_init_win2k19_vm-01 will be created
+ resource "local_file" "cloud_init_win2k19_vm-01" {
+ content = <<~EOT
Content-Type: multipart/mixed; boundary="===============1598784645116016685=="
MIME-Version: 1.0
--===============1598784645116016685==
Content-Type: text/cloud-config; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="cloud-config"
write_files:
- path: c:\salt\conf\minion
content: |
master: salt-master.yetiops.lab
id: win2k19-vm-01.yetiops.lab
nodename: win2k19-vm-01
startup_states: highstate
--===============1598784645116016685==
Content-Type: text/x-cfninitdata; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="cfn-userdata"
#ps1
Restart-Service salt-minion
EOT
+ directory_permission = "0777"
+ file_permission = "0777"
+ filename = "./files/user_data_cloud_init_win2k19_vm-01.cfg"
+ id = (known after apply)
}
# null_resource.cloud_init_win2k19_vm-01 will be created
+ resource "null_resource" "cloud_init_win2k19_vm-01" {
+ id = (known after apply)
}
# proxmox_vm_qemu.win2k19-vm will be created
+ resource "proxmox_vm_qemu" "win2k19-vm" {
+ agent = 1
+ balloon = 0
+ bios = "seabios"
+ boot = "cdn"
+ bootdisk = "scsi0"
+ cicustom = "user=local:snippets/cloud_init_win2k19_vm-01.yml"
+ clone = "win2k19-cloudinit"
+ clone_wait = 15
+ cores = 1
+ cpu = "host"
+ force_create = false
+ full_clone = true
+ hotplug = "network,disk,usb"
+ id = (known after apply)
+ ipconfig0 = "ip=dhcp"
+ kvm = true
+ memory = 2048
+ name = "win2k19-vm-01"
+ numa = false
+ onboot = true
+ preprovision = true
+ scsihw = "virtio-scsi-pci"
+ sockets = 1
+ ssh_host = (known after apply)
+ ssh_port = (known after apply)
+ target_node = "pve-01"
+ vcpus = 0
+ vlan = -1
+ vmid = 0
+ disk {
+ backup = false
+ cache = "none"
+ format = "raw"
+ id = 0
+ iothread = true
+ mbps = 0
+ mbps_rd = 0
+ mbps_rd_max = 0
+ mbps_wr = 0
+ mbps_wr_max = 0
+ replicate = false
+ size = "20"
+ storage = "local-lvm"
+ storage_type = "lvm"
+ type = "scsi"
}
+ network {
+ bridge = "vmbr0"
+ firewall = false
+ id = 0
+ link_down = false
+ macaddr = (known after apply)
+ model = "virtio"
+ queues = -1
+ rate = -1
+ tag = -1
}
}
Plan: 3 to add, 0 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
local_file.cloud_init_win2k19_vm-01: Creating...
local_file.cloud_init_win2k19_vm-01: Creation complete after 0s [id=66f7140731ba60729758ed365fed0ba7a10f3766]
null_resource.cloud_init_win2k19_vm-01: Creating...
null_resource.cloud_init_win2k19_vm-01: Provisioning with 'file'...
null_resource.cloud_init_win2k19_vm-01: Creation complete after 0s [id=219014806438120636]
proxmox_vm_qemu.win2k19-vm: Creating...
proxmox_vm_qemu.win2k19-vm: Still creating... [10s elapsed]
proxmox_vm_qemu.win2k19-vm: Still creating... [20s elapsed]
proxmox_vm_qemu.win2k19-vm: Still creating... [30s elapsed]
proxmox_vm_qemu.win2k19-vm: Still creating... [40s elapsed]
proxmox_vm_qemu.win2k19-vm: Still creating... [50s elapsed]
proxmox_vm_qemu.win2k19-vm: Still creating... [1m0s elapsed]
proxmox_vm_qemu.win2k19-vm: Still creating... [1m10s elapsed]
proxmox_vm_qemu.win2k19-vm: Still creating... [1m20s elapsed]
proxmox_vm_qemu.win2k19-vm: Still creating... [1m30s elapsed]
proxmox_vm_qemu.win2k19-vm: Still creating... [1m40s elapsed]
proxmox_vm_qemu.win2k19-vm: Still creating... [1m50s elapsed]
proxmox_vm_qemu.win2k19-vm: Still creating... [2m0s elapsed]
proxmox_vm_qemu.win2k19-vm: Still creating... [2m10s elapsed]
proxmox_vm_qemu.win2k19-vm: Still creating... [2m20s elapsed]
proxmox_vm_qemu.win2k19-vm: Still creating... [2m30s elapsed]
proxmox_vm_qemu.win2k19-vm: Still creating... [2m40s elapsed]
proxmox_vm_qemu.win2k19-vm: Still creating... [2m50s elapsed]
proxmox_vm_qemu.win2k19-vm: Still creating... [3m0s elapsed]
proxmox_vm_qemu.win2k19-vm: Still creating... [3m10s elapsed]
proxmox_vm_qemu.win2k19-vm: Still creating... [3m20s elapsed]
proxmox_vm_qemu.win2k19-vm: Still creating... [3m30s elapsed]
proxmox_vm_qemu.win2k19-vm: Still creating... [3m40s elapsed]
proxmox_vm_qemu.win2k19-vm: Still creating... [3m50s elapsed]
proxmox_vm_qemu.win2k19-vm: Still creating... [4m0s elapsed]
Error: error unmarshalling result Could not parse fe80::dc7e:d119:c72d:dbac%2 as IP
on win-vm.tf line 34, in resource "proxmox_vm_qemu" "win2k19-vm":
34: resource "proxmox_vm_qemu" "win2k19-vm" {
The last part of this is due to running the Qemu Guest Agent. Proxmox sees that the IP address return is an IPv6 address, and returns it to the Terraform provider. Unfortunately the provider is expecting either an IPv4 address, or an IPv6 address without %2 at the end of it. However if we look in the Terraform state, we are still managing this machine: —
$ terraform state list
data.template_file.cloud_init_win2k19_vm-01
local_file.cloud_init_win2k19_vm-01
null_resource.cloud_init_win2k19_vm-01
proxmox_vm_qemu.win2k19-vm
We can also verify it exists in the Proxmox console: —
It appears that while the terraform apply operation produces an error, it is after all configuration has taken place. Terraform does however mark the instance as tainted (i.e. required to be destroyed and rebuilt). To avoid rebuilding the instance on your next terraform apply, run terraform untaint proxmox_vm_qemu.win2k19-vm to untaint it. You can then verify with a terraform plan that instance matches what we defined: —
# Untaint the instance
$ terraform untaint proxmox_vm_qemu.win2k19-vm
Resource instance proxmox_vm_qemu.win2k19-vm has been successfully untainted.
# Run a plan to ensure no changes are required
$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
data.template_file.cloud_init_win2k19_vm-01: Refreshing state...
local_file.cloud_init_win2k19_vm-01: Refreshing state... [id=66f7140731ba60729758ed365fed0ba7a10f3766]
null_resource.cloud_init_win2k19_vm-01: Refreshing state... [id=219014806438120636]
proxmox_vm_qemu.win2k19-vm: Refreshing state... [id=pve-01/qemu/100]
Alternatively, if you do not install the Qemu Guest Agent, then the Terraform provider will complete without issues. It is worthwhile running the Guest Agent (safer shutdowns, exposes information to the hypervisor regarding the instance), but it is not necessary to the day to day running of the machine.
Salt
The Salt setup and states are the same as we used in this post. We use Salt to: —
- Deploy Consul as an agent
- Register with the Consul server (the Salt server)
- Deploy the Windows Exporter so that Prometheus can monitor the instance
As per the previous post, we also run a highstate (i.e. all applicable Salt states) when the Minion is accepted by the Salt server.
Accept the key
To accept the minion on the master, we need to check that it has registered first: —
$ sudo salt-key -L
Accepted Keys:
pve-01.yetiops.lab
salt-master.yetiops.lab
Denied Keys:
Unaccepted Keys:
win2k19-vm-01.yetiops.lab
Rejected Keys:
Now we can accept the key: —
$ sudo salt-key -a 'win2k19*'
The following keys are going to be accepted:
Unaccepted Keys:
win2k19-vm-01.yetiops.lab
Proceed? [n/Y] y
Key for minion win2k19-vm-01.yetiops.lab accepted.
We can check what states will be applied to this minion as well: —
$ sudo salt 'win2k19-vm-01*' state.show_states
win2k19-vm-01.yetiops.lab:
- consul.windows
- exporters.windows_exporter.win_exporter
- exporters.windows_exporter.windows_exporter
Verify the highstate
As mentioned, the Minion configuration specifies that it will run a highstate when the key is accepted. We can verify this seeing if the instance has registered with Consul and that the Windows Exporter is running: —
$ consul members
Node Address Status Type Build Protocol DC Segment
salt-master 10.15.31.249:8301 alive server 1.8.1 2 yetiops <all>
WIN-75R8JQ6IARE 10.15.31.166:8301 alive client 1.7.2 2 yetiops <default>
pve-01 10.15.31.7:8301 alive client 1.8.1 2 yetiops <default>
$ curl 10.15.31.166:9182/metrics | grep -i 2019
windows_os_info{product="Microsoft Windows Server 2019 Standard Evaluation",version="10.0.17763"} 1
Looks like they are working! It is worth nothing that we do not set a hostname in cloud-config for our machine, so it uses a randomly generated hostname instead.
As a last step of verification, we can run another highstate to ensure that no configuration files change, or any other packages are due to be installed: —
$ sudo salt 'win2k19*' state.highstate
win2k19-vm-01.yetiops.lab:
----------
ID: consul_user
Function: user.present
Name: Consul
Result: True
Comment: User Consul is present and up to date
Started: 20:04:56.229276
Duration: 29925.002 ms
Changes:
----------
ID: consul_base_dir
Function: file.directory
Name: C:\consul
Result: True
Comment: Directory C:\consul updated
Started: 20:05:26.157279
Duration: 24.024 ms
Changes:
----------
grant_perms:
----------
Users:
----------
perms:
full_control
perms:
----------
Users:
----------
grant:
full_control
----------
ID: consul_conf_dir
Function: file.directory
Name: C:\consul\conf
Result: True
Comment: Directory C:\consul\conf updated
Started: 20:05:26.181303
Duration: 25.994 ms
Changes:
----------
grant_perms:
----------
Users:
----------
perms:
full_control
perms:
----------
Users:
----------
grant:
full_control
----------
ID: consul_data_dir
Function: file.directory
Name: C:\consul\data
Result: True
Comment: Directory C:\consul\data updated
Started: 20:05:26.208305
Duration: 13.041 ms
Changes:
----------
grant_perms:
----------
Users:
----------
perms:
full_control
perms:
----------
Users:
----------
grant:
full_control
----------
ID: c:\consul\conf\consul.hcl
Function: file.managed
Result: True
Comment: File c:\consul\conf\consul.hcl is in the correct state
Started: 20:05:26.227289
Duration: 795.02 ms
Changes:
----------
ID: modify_consul_service
Function: module.run
Name: service.modify
Result: True
Comment: Module function service.modify executed
Started: 20:05:27.022309
Duration: 23.919 ms
Changes:
----------
ret:
----------
BinaryPath:
"c:\consul\consul.exe" agent -config-dir=c:\consul\conf
DisplayName:
HashiCorp Consul Client Agent
StartType:
Auto
----------
ID: running_consul_service
Function: service.running
Name: consul
Result: True
Comment: The service consul is already running
Started: 20:05:27.046228
Duration: 5.0 ms
Changes:
----------
ID: c:\consul\conf\windows_exporter.hcl
Function: file.managed
Result: True
Comment: File c:\consul\conf\windows_exporter.hcl is in the correct state
Started: 20:05:27.051228
Duration: 96.359 ms
Changes:
----------
ID: consul_reload_windows_exporter
Function: cmd.run
Name: c:\consul\consul.exe reload
Result: True
Comment: Command "c:\consul\consul.exe reload" run
Started: 20:05:27.149588
Duration: 6710.107 ms
Changes:
----------
pid:
1208
retcode:
0
stderr:
stdout:
Configuration reload triggered
----------
ID: exporter_user
Function: user.present
Name: exporter
Result: True
Comment: User exporter is present and up to date
Started: 20:05:33.859695
Duration: 93.698 ms
Changes:
----------
ID: exporter_base_dir
Function: file.directory
Name: C:\exporter
Result: True
Comment: Directory C:\exporter updated
Started: 20:05:33.953393
Duration: 15.64 ms
Changes:
----------
grant_perms:
----------
Users:
----------
perms:
full_control
perms:
----------
Users:
----------
grant:
full_control
----------
ID: exporter_textfile_dir
Function: file.directory
Name: C:\exporter\textfile
Result: True
Comment: Directory C:\exporter\textfile updated
Started: 20:05:33.969033
Duration: 15.618 ms
Changes:
----------
grant_perms:
----------
Users:
----------
perms:
full_control
perms:
----------
Users:
----------
grant:
full_control
----------
ID: windows_exporter_dir
Function: file.directory
Name: C:\exporter\windows_exporter
Result: True
Comment: Directory C:\exporter\windows_exporter updated
Started: 20:05:33.984651
Duration: 0.0 ms
Changes:
----------
grant_perms:
----------
Users:
----------
perms:
full_control
perms:
----------
Users:
----------
grant:
full_control
Summary for win2k19-vm-01.yetiops.lab
-------------
Succeeded: 13 (changed=8)
Failed: 0
-------------
Total states run: 13
Total run time: 37.743 s
Everything looks good!
Prometheus
As noted in the previous post, we are using the same Prometheus setup as we do here.
Also, as we are using the same Salt states, we should expect to see the instance like we do in this post: —
There it is!
Grafana
Again, everything in this post regarding Grafana is applicable too. We can use the Windows Node Dashboard to verify the metrics: —
Alternatively you can use this dashboard, which has now been updated to use the Windows Exporter metric naming (i.e windows_ instead of wmi_, as the exporter changed from wmi_exporter to windows_exporter): —
Summary
Using Cloudbase-Init, it is possible to customize a Windows image when it starts for the first time, allowing you to bootstrap a machine ready to be used in your environment. This takes away the work of installing machines manually, while also being able to manage Windows in a similar way to Linux and Unix variants (e.g. FreeBSD, OpenBSD) that support Cloud-Init.
As we have also seen, it is not without caveats currently. Some of this is down to Proxmox in how it presents Cloud-Init data, some of it is down to the Cloudbase-Init utility that is expecting configuration in a certain format but could potentially be in multiple formats (i.e. both dns-nameservers and dns_nameservers are possible within Cloud-Init, but Cloudbase-Init currently understands the former), and some are down to the Terraform provider that does not seem to understand some of values presented by the Proxmox API (specifically the IPv6 addressing).
Over time these will improve, as issues have been raised for all of these cases. This should not discourage you from using any of these tools, as all of them together bring Windows deployment on Proxmox on par with Linux (in terms of bootstrapping and Terraform usage).
Cloudbase-Init was initially created for use with OpenStack, and the fact it works with other providers and infrastructure is a nice bonus.