Blog Marcina Bojko

Linux,Windows,serwer, i tak dalej ;)

Archive for the ‘work’ Category

Hyper-V Packer’s Virtual Machine Generation 2 Templates – Windows 2016 and CentOS 7.4

leave a comment »

Selection_999(897)

As you probably know – HashiCorp’s Packer (https://www.packer.io/) is a great tool for generating customized OS images for future use.

You can use many builders, starting from Virtualbox, Azure, Amazon EC2. You can also use mechanizm of providers to customize your image even more.
That becomes very handy when you have to manage your infrastructure fighting for every minute your machine will be available, and later on – ready to work. It’s advisable to put some effort to prepare latest images possible, with your organization custom needs.

However great tool, it still lacked a lots of decent documentation how to prepare images using less common builder – Microsoft’s Hyper-V. Hyper-V is a part of Microsoft Windows for a long time, it can be used as standalone product (Microsoft Hyper-V Server) or a additional feature (Hyper-V Feature in Windows 2016/10). When using packer with so-called „generation 1” images packer templates are no different from common examples prepared for popular Virtualbox builder.

With „Generation 2” images it required some more work to achieve this goal.

What’s the use for Gen-2 images? Few features than can be useful:

  • secure boot (Windows/Linux)
  • UEFI boot
  • no IDE/COM burden from previous generations
  • boot from SCSI devices

If you want to know more about „Genration 2” – read this post: https://www.altaro.com/hyper-v/comparing-hyper-v-generation-1-2-virtual-machines/

It took me a while to prepare them Hyper-V ready, also with SystemCenter Virtual Machine Support. They were (and are) tested with both (2012 and 2016) editions and just became quite important part of my deploys.

Long story short – here is my github repository with Hyper-V Windows 2016 and CentOS 7.4 Generation 2 templates.

https://github.com/marcinbojko/hv-packer

Leave me a sign if these were useful to you 🙂

 

 

Written by marcinbojko

Listopad 11, 2017 at 19:30

Napisane w work

Tagged with , , ,

Centos 6/7 virtual machines and Production Checkpoints in Hyper-V 2016.

As we may know – Microsoft introduced new way of doing snapshots/checkpoints in Hyper-V 2016. However term „production” is misleading, implying Standard checkpoints are not production ready – which is simply not true.
The biggest difference is that Production checkpoints are mostly used with VSS-aware applications (like MS SQL/Exchange, MS Windows itself) allowing them to flush/sync/commit changes to filesystem.

As a major difference – production checkpoints don’t save memory or cpu state, starting always with machine powered off after restore.

You can choose which way you want to do your snapshots here:

Selection_999(411)

Windows-based virtual machines have supported this since previous versions of integration services (2012 R2, 8/8.1) and from the start in case of Windows 2016/10. What about Linux-based, Centos 6/7 machines?

When installed out of the box, without any additional packages, trying to do a production snapshot of Centos 7 (with all updates) we got something like this:

Selection_999(410)

Quick how-to.

  1. If youre using external LIS (Linux Integration Services) from Microsoft, as an external package – remove it. It’s a piece of crap,  breaking kernels from time to time, packed with ‚latest’ errors and workaround rejected by linux kernel maintainers. It’s really not worth risk to have it installed: 
    yum remove microsoft-hyper-v kmod-microsoft-hyper-v

    or

    yum remove $(yum list installed|grep microsoft)
  2. Check if your Hyper-V offers all Integration Services for this VM.Selection_999(412)
  3. Check  and install hyperv-daemons
     yum info hyperv-daemons

    Available Packages
    Name : hyperv-daemons
    Arch : x86_64
    Version : 0
    Release : 0.29.20160216git.el7
    Size : 4.5 k
    Repo : base/7/x86_64
    Summary : HyperV daemons suite
    URL : http://www.kernel.org
    Licence : GPLv2
    Description : Suite of daemons that are needed when Linux guest : is running on Windows Host with HyperV

    yum install hyperv-daemons -y
  4. Enable and start services
    systemctl enable hypervfcopyd
    systemctl enable hypervkvpd
    systemctl enable hypervvssd
    
    systemctl start hypervkvpd 
    systemctl start hypervvssd 
    systemctl start hypervfcopyd
  5. Check status
    [root@centos7 ~]# systemctl status hypervkvpd
    ● hypervkvpd.service - Hyper-V KVP daemon
     Loaded: loaded (/usr/lib/systemd/system/hypervkvpd.service; static; vendor preset: enabled)
     Active: active (running) since Wed 2017-07-26 02:37:30 CDT; 14s ago
     Main PID: 3478 (hypervkvpd)
     CGroup: /system.slice/hypervkvpd.service
     └─3478 /usr/sbin/hypervkvpd -n
    
    Jul 26 02:37:30 centos7 systemd[1]: Started Hyper-V KVP daemon.
    Jul 26 02:37:30 centos7 systemd[1]: Starting Hyper-V KVP daemon...
    Jul 26 02:37:30 centos7 KVP[3478]: KVP starting; pid is:3478
    Jul 26 02:37:30 centos7 KVP[3478]: KVP LIC Version: 3.1
    [root@centos7 ~]# systemctl status hypervvssd
    ● hypervvssd.service - Hyper-V VSS daemon
     Loaded: loaded (/usr/lib/systemd/system/hypervvssd.service; static; vendor preset: enabled)
     Active: active (running) since Wed 2017-07-26 02:37:30 CDT; 27s ago
     Main PID: 3485 (hypervvssd)
     CGroup: /system.slice/hypervvssd.service
     └─3485 /usr/sbin/hypervvssd -n
    
    Jul 26 02:37:30 centos7 systemd[1]: Started Hyper-V VSS daemon.
    Jul 26 02:37:30 centos7 systemd[1]: Starting Hyper-V VSS daemon...
    Jul 26 02:37:30 centos7 hypervvssd[3485]: Hyper-V VSS: VSS starting; pid is:3485
    Jul 26 02:37:30 centos7 hypervvssd[3485]: Hyper-V VSS: VSS: kernel module version: 129
    [root@centos7 ~]# systemctl status hypervfcopyd
    ● hypervfcopyd.service - Hyper-V FCOPY daemon
     Loaded: loaded (/usr/lib/systemd/system/hypervfcopyd.service; static; vendor preset: disabled)
     Active: active (running) since Wed 2017-07-26 02:37:30 CDT; 44s ago
     Main PID: 3492 (hypervfcopyd)
     CGroup: /system.slice/hypervfcopyd.service
     └─3492 /usr/sbin/hypervfcopyd -n
    
    Jul 26 02:37:30 centos7 systemd[1]: Started Hyper-V FCOPY daemon.
    Jul 26 02:37:30 centos7 systemd[1]: Starting Hyper-V FCOPY daemon...
    Jul 26 02:37:30 centos7 HV_FCOPY[3492]: starting; pid is:3492
    Jul 26 02:37:30 centos7 HV_FCOPY[3492]: kernel module version: 1

    As a result:
    Selection_999(413)
    and in /var/log/messages

    Jul 26 02:43:27 centos7 journal: Hyper-V VSS: VSS: op=FREEZE: succeeded
    
    Jul 26 02:39:25 centos7 systemd: Time has been changed
    
    Jul 26 02:39:25 centos7 journal: Hyper-V VSS: VSS: op=THAW: succeeded

 

Written by marcinbojko

Lipiec 26, 2017 at 18:05

Napisane w work

Tagged with , , , , ,

Petya(notPetya) ransomware attack and how to (quickly) vaccinate lot’s of machines

There was a lot of nice summary articles about latest „ransomware” attack caused by Petya. Soon, researchers started to claim almost permanent vaccine for this type of worm.

https://www.bleepingcomputer.com/news/security/vaccine-not-killswitch-found-for-petya-notpetya-ransomware-outbreak/

Even patched OS won’t save you from infection as one infected machine quickly spreads the infection using other protocols like WinRM.

So, how should one on its vast server farm vaccinate hundrets of machines?

For example, like this 🙂

win_manage:
  dsc_file:    
    petya_vaccine1:
      dsc_destinationpath: C:\Windows\perfc
      dsc_type: file
      dsc_attributes: readonly
      dsc_contents: ""
    petya_vaccine2:
      dsc_destinationpath: C:\Windows\perfc.dat
      dsc_type: file
      dsc_attributes: readonly
      dsc_contents: ""
    petya_vaccine3:
      dsc_destinationpath: C:\Windows\perfc.dll
      dsc_type: file
      dsc_attributes: readonly
      dsc_contents: ""

 

Written by marcinbojko

Lipiec 1, 2017 at 11:14

Po LDI 2017

2017-04-22 20.14.36

Written by marcinbojko

Kwiecień 22, 2017 at 20:16

Napisane w work

LDI 2017 – przydatne linki

Prezentacja: https://goo.gl/1HJZjI

Github repo: https://github.com/marcinbojko/ldi2017

Written by marcinbojko

Kwiecień 22, 2017 at 12:40

Napisane w work

5 serious issues/deal breakers with System Center Virtual Machine Manager.

System Center Virtual Machine Manager was Microsoft’s answer to VMWare’s vSphere. It’s Microsoft, so what could have gone wrong? It’s Microsoft – so everything.
Below is a list of most annoying things, some of them are so serious it makes you wonder – maybe Powershell is the answer? Seriously? In 2017, Microsoft, you FORCE everybody to use text console again?
In a moment of doubt we used to call it overgrown cancer over Powershell commands.

Let’s start, sorted by weight of crime:

Deal breakers:

1) Terrible things you cannot do in SCVMM but you can in Hyper-V Manager, Failover Cluster or Powershell like:

  • rename you machine when its powered on (sic)
  • change its MAC from Dynamic to Static other way like manually copy it character by character.
  • change booting order (sic) of machines and templates
  • select all Integration Services offered
  • change location of smart paging file
  • change affinity with cluster (high/medium/low/do not autostart)

and so on.

2) Console. Console is so terrible, that its sorry state is just good meme source.

First – console from Hyper-V Manager/FailoverCluster

Selection_670.png

Then from SCVMM

Selection_671.png

  • you cannot attach console and then power on machine. You HAVE to – power on machine, wait few sec for console button to be available then race through time to start it BEFORE OS starts. You have better chances of winning some of Grand Prix then finish the trick above on first run.
  • only actions you have is Reconnect and Send CTRL+ALT+Delete. Never working ‚Clipboard’ added in SCVMM 2016 requires you to paste text HERE, then it’s pasted in VM console
  • when it start before machine starts – you have to kill an application. It’s no good to use it ever again, it won’t ‚click’ with machine you’ve started. Exit? Something terrible may happen.

Selection_672.png

3) Requirements

  • MS SQL Server Standard or Enterprise. https://technet.microsoft.com/en-us/system-center-docs/system-requirements/sql-server-version-compatibility
  • 4 GB Ram required, 16 GB recommended (don’t even bother going below)
  • A lot of not-really-so-working tricks to use it to manage hosts from other domains, especially without 2 way trusts settled.
  • Price. With whole gang of System Center tools, prepare to be robbed in a daylight. Doesn’t matter you have no intention to use other components – you have to pay for it. You cannot just pick and buy needed component – you have to buy-and-pay with bulk.

4) GUI

  • General slowness of GUI, regardless of hosts number, running tasks, library sizes.
  • Jobs window – generally unusable with more than one admin or more than one job running- lots of informational comments. Important actions (like: who deleted or altered machine) quickly goes off the screen, covered by messages like: refresh was completed.
  • Oh, did I mention ‚Refresh’ habit? Learn it. Learn it, and let your fingers memorize this config, as you will be using it a lot.

Refresh is required almost on everything. In options like: you DID change something via Powershell and HV-Manager – I can understand, refresh may be required. But you will have to hit REFRESH before, in time, and after ANY action you would like to perform. If not – expect the worst. Virtual machine seems to be non responding on your commands? Maybe its locked for backup, maybe it hanged, maybe it migrated to another host – you have to refresh, refresh and refresh to persuade SCVMM that you have most recent data.

Sometimes even refresh doesn’t work. Like in recovery or cluster node failure, you shouldn’t count on SCVMM to update its status before timer reaches day or two. Take your time! Sometime you will have to reboot SCVMM to persuade it to have the latest data. So, when your action fails – search no more, VM is probably locked, on other host or powered off. SCVMM takes its auto-refresh very slowly.

  • General over complexity in Logical Network and Switches. It’s like you have to create every VLAN again, even if you’ve done it on dozen of network devices, fill variables like subnets, gateways. You have to group it all together and again, attach to every Hyper-V switch on hosts you have.
  • Adding you own custom fields and filling them is, again, over complicated and requires you to do a lot of scripting and scheduling them in a Windows manner.
  • You cannot add, change or sort fields like Operating systems. What Microsoft got you are values like this:
    • Microsoft Windows Server 2012 R2
    • 64-bit edition of Microsoft Server

Selection_673.png

  • Hyper-V integration Services are always few releases behind. It started to change with Windows 2016 and idea to install them via Windows Update.
  • Inability to rename vm folder when machine changes its name. This way you will have to do a Live Migration to rename folder.
  • Complexity of generated script.One will think generating a new machine is easy: New-SCVirtualMachine with a lot of parameters. No. Script is long, heavy, complex and tries to do things in complete different matter.
  • Templates – only way to refresh a template is to create it again, or replace vhdx in library, and do some internal tricks.
  • Inability to do anything with machine when the job is running – all fields are grayed out and you have to wait for jobs to end or fail.

5) Agent

  • if you’re lucky, agents are deployed ALMOST instantly, but adding host to SCVMM requires it to restart
  • if you’re not lucky, then in case of SCVMM upgrade, you will have to manually redeploy and reinstall all agents. Quite common I’d say.
  • [IMPORTANT] The mess agent leaves on filesystem is just legendary. Lets say we would like to migrate our machine from folder d:\vm to e:\vm.After migration (when we choose right option) we will got:

– empty files in d:\vm\machinename

– machine in e:\vm\machinename

Let’s say we would like to migrate it back for some reason

We will get:

  • empty d:\vm\machinename
  • empty e:\vm\machinename
  • machine in e:\vm\machinename (1)

And migration is just done twice. Do you see the pattern? After few migrations we have complete chaos on filesystems with a lots of empty, semi-empty, almost empty and ‚soon-to-be-empty folders’. You’ll end up with removing them manually – again, if you’re lucky.

  • locked folder after failed job. Yes, when you migration failed, you will end with d:\vm\machinename which you’re not able to delete. Sometimes it can be deleted after some time, sometimes after SCVMM/host reboot, sometimes never.

Above list, not fully completed can be seen in SCVMM 2012 R2 and SCVMM 2016 versions. It’s clear that SCVMM is not very high on Microsoft ‚to do’ list as same errors and mistakes are transferred to newer version and hunts us until this day.

 

UPDATE (1)
Changed from Requirements (Enterprise) to (Standard , Enterprise)

Written by marcinbojko

Luty 4, 2017 at 20:08

Napisane w work

Tagged with , , , ,

Does Foreman speak SQL? It does ;)

So, the question is – how to deploy and maintain farm of Microsoft SQL Servers? With different domains, install sources, roles, features. Should we create unattended installers for every single instance?

No, we shouldn’t.

We should use Powershell DSC – https://github.com/PowerShell/xSQLServer
With Foreman/Puppet and win_manage, we have something like:

The simplest way:

instance1:
  dsc_instancename: MSSQLSERVER
  dsc_sourcepath: "\\our.server.com\ourshare"
  dsc_sourcecredential:
    user: anonymous
    password: anonymous
  dsc_setupcredential:
    user: DOMAIN\someuser
    password: somepassword

Or, more sophisticated:

instance1:
  dsc_instancename: MSSQLSERVER
  dsc_sourcepath: "\\our.server.com\ourshare"
  dsc_sourcecredential:
    user: anonymous
    password: anonymous
  dsc_setupcredential:
    user: DOMAIN\someuser
    password: somepassword
  dsc_features: SQLENGINE
  dsc_forcereboot: true
  dsc_agtsvcaccount:
    user: DOMAIN\scvaccount
    password: somepassword
  dsc_sqlsvcaccount:
    user: DOMAIN\sqlaccount
    password: somepassword
  dsc_sqlcollation: SQL_Latin1_General_CP1_CI_AS
  dsc_sqlsysadminaccounts:
    - DOMAIN\someadmin
    - DOMAIN\someotheradmin
  dsc_securitymode: SQL
  dsc_sapwd:
  user: sa
  password: paaaswordisverysecure
  dsc_sqluserdbdir: D:\MSSQL\Data
  dsc_sqluserdblogdir: E:\MSSQL\Data
  dsc_browsersvcstartuptype: Disabled

Whole install from local source takes aprox. 400 seconds (with other OS related settings) to finish.

Written by marcinbojko

Styczeń 22, 2017 at 21:11

%d blogerów lubi to: