YAML it Rhymes with Camel

I’ve blogged before about my passion for automation and the use of ARM templating in the Azure world to eradicate the burden of dull and mundane tasks from the daily routine of system administrators for whom I do consulting for.

I loath repetitive tasks, its in this space where subtle differences and inconsistency love to live. Recently I was asked to help out with a simple task, provisioning a couple of EC2 Windows servers in AWS. So in the spirit of infrastructure as code, I thought, there is no better time to try out AWS CloudFormation to describe my EC2 instances . I’ve actually used CloudFormation before in the past, but always describing my stack in JSON. CloudFormation also supports YAML, so challenge accepted and away I went. . .

So what is YAML anyway. . .Yet Another Mark-up Language. Interestingly its described at the official YAML website (https://yaml.org) as a “YAML Ain’t Markup Language” rather,  “human friendly data serialisation standard for all programming languages”.

What attracted me to YAML is its simplicity, there are no curly braces {} just indenting. Its also super easy to read. So if JSON looks a bit to cody for your liking, YAML may be a more palatable alternative.

So how would you get started? As you’d expect AWS have extensive CloudFormation documentation. The AWS::EC2::Instance resource is described here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-instance.html#cfn-ec2-instance-volumes. You’ll notice that there is a Syntax description for JSON and YAML. The YAML looks like this:

[code language=”javascript”] Type: AWS::EC2::Instance Properties: Affinity: String AvailabilityZone: String BlockDeviceMappings: – EC2 Block Device Mapping CreditSpecification: CreditSpecification DisableApiTermination: Boolean EbsOptimized: Boolean ElasticGpuSpecifications: [ ElasticGpuSpecification, … ] ElasticInferenceAccelerators: – ElasticInferenceAccelerator HostId: String IamInstanceProfile: String ImageId: String InstanceInitiatedShutdownBehavior: String InstanceType: String Ipv6AddressCount: Integer Ipv6Addresses: – IPv6 Address Type KernelId: String KeyName: String LaunchTemplate: LaunchTemplateSpecification LicenseSpecifications: – LicenseSpecification Monitoring: Boolean NetworkInterfaces: – EC2 Network Interface PlacementGroupName: String PrivateIpAddress: String RamdiskId: String SecurityGroupIds: – String SecurityGroups: – String SourceDestCheck: Boolean SsmAssociations: – SSMAssociation SubnetId: String Tags: – Resource Tag Tenancy: String UserData: String Volumes: – EC2 MountPoint AdditionalInfo: String [/code]

With this as a starting point I was quickly able to build a EC2 instance and customise my YAML so as to do some extra things.

If you’ve got this far and YAML is starting to look like it might be the ticket for you, its worth familiarising yourself with the CloudFormation built-in functions. You can use these to do things like assign values to properties that are not available until runtime.

Fn::Base64
Fn::Cidr
Condition Functions
Fn::FindInMap
Fn::GetAtt
Fn::GetAZs
Fn::Join
Fn::Select
Fn::Split
Fn::Sub
Fn::Transform
Ref

The link to the complete Intrinsic Function Reference can be found here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference.html

With a learning curve of a couple of hours including a bit of googling and messing around I was able to achieve my goal. I built an EC2 instance, applied tagging, installed some Windows features post build via a PowerShell script (downloaded from S3 and launched with AWS::CloudFormation::Init cfn-init.exe), all without having to logon to the server or touch the console. Here is a copy of my YAML. . .

[code language=”javascript”] AWSTemplateFormatVersion: “2010-09-09” Description: CloudFormation Template to deploy an EC2 instance Parameters: Hostname: Type: String Description: Hostname – maximum 15 characters MaxLength: ’15’ LatestAmiId : Type: ‘AWS::SSM::Parameter::Value’ Default: /aws/service/ami-windows-latest/Windows_Server-2019-English-Full-Base InstanceSize: Type: String Description: Instance Size Default: t2.micro AllowedValues: – “t2.micro” – “t2.small” – “t2.medium” AvailabilityZone: Type: String Description: Default AZ AllowedValues: – ap-southeast-2a – ap-southeast-2b – ap-southeast-2c Default: ap-southeast-2a KeyPair: Type: String Description: KeyPair Name Default: jtwo S3BucketName: Default: NotARealBucket Description: S3 bucket containing boot artefacts Type: String # tag values awPurpose: Type: String Description: A plain English description of what the object is for. Default: WindowsServer2019 Domain Controller awChargeTo: Type: String Description: Billing Code for charge back of resource. Default: IT-123 awRegion: Type: String Description: Accolade Wines Region not AWS.  Default: Australia awExpiry: Type: String Description: The date when the resource(s) can be considered for decommissioning. Default: 01-01-2022 awBusinessSegment: Type: String Description: Agency code. Default: ICT awEnvironment: Type: String Description: Specific environment for resource. AllowedValues: – prod – prodServices – nonprod – uat – dev – test awApplication: Type: String Description: A single or multiple word with the name of the application that the infrastructure supports. “JDE”, “AD”, “Apache”, “Utility”, “INFOR”, “PKI”. Default: AD Mappings: SubnetMap: ap-southeast-2a: prodServices: “subnet-idGoesHere” ap-southeast-2b: prodServices: “subnet-idGoesHere” ap-southeast-2c: prodServices: “subnet-idGoesHere” # Resources Resources: # IAM Instance Profile Profile: Type: ‘AWS::IAM::InstanceProfile’ Properties: Roles: – !Ref HostRole Path: / InstanceProfileName: !Join – ” – – ‘instance-profile-‘ – !Ref S3BucketName HostRole: Type: ‘AWS::IAM::Role’ Properties: RoleName: !Join – ” – – ‘role-s3-read-‘ – !Ref S3BucketName Policies: – PolicyDocument: Version: 2012-10-17 Statement: – Action: – ‘s3:GetObject’ Resource: !Join – ” – – ‘arn:aws:s3:::’ – !Ref S3BucketName – ‘/*’ Effect: Allow PolicyName: s3-policy-read Path: / AssumeRolePolicyDocument: Statement: – Action: – ‘sts:AssumeRole’ Principal: Service: – ec2.amazonaws.com Effect: Allow Version: 2012-10-17 # ENI NIC1: Type: AWS::EC2::NetworkInterface Properties: Description: !Sub ‘ENI for EC2 instance: ${Hostname}-${awEnvironment}’ GroupSet: – sg-050cadbf0e159b0ac SubnetId: !FindInMap [SubnetMap, !Ref AvailabilityZone, !Ref awEnvironment] Tags: – Key: Name Value: !Sub ‘${Hostname}-eni’ # EC2 Instance Instance: Type: ‘AWS::EC2::Instance’ Metadata: ‘AWS::CloudFormation::Authentication’: S3AccessCreds: type: S3 buckets: – !Ref S3BucketName roleName: !Ref HostRole ‘AWS::CloudFormation::Init’: configSets: config: – get-files – configure-instance get-files: files: ‘c:\s3-downloads\scripts\Add-WindowsFeature.ps1’: source: https://NotARealBucket.s3.amazonaws.com/scripts/Add-WindowsFeature.ps1 authentication: S3AccessCreds configure-instance: commands: 1-set-powershell-execution-policy: command: >- powershell.exe -Command “Set-ExecutionPolicy UnRestricted -Force” waitAfterCompletion: ‘0’ 2-rename-computer: command: !Join – ” – – >- – powershell.exe -Command “Rename-Computer -Restart -NewName ” – !Ref Hostname waitAfterCompletion: forever 3-install-windows-components: command: >- powershell.exe -Command “c:\s3-downloads\scripts\Add-WindowsFeature.ps1” waitAfterCompletion: ‘0’ Properties: DisableApiTermination: ‘false’ AvailabilityZone: !Sub “${AvailabilityZone}” InstanceInitiatedShutdownBehavior: stop IamInstanceProfile: !Ref Profile ImageId: !Ref LatestAmiId InstanceType: !Sub “${InstanceSize}” KeyName: !Sub “${KeyPair}” UserData: !Base64 ‘Fn::Join’: – ” – – “\n” – “cfn-init.exe ” – ” –stack ” – “Ref”: “AWS::StackId” – ” –resource Instance” – ” –region ” – “Ref”: “AWS::Region” – ” –configsets config” – ” -v \n” – “cfn-signal.exe ” – ” —exit-code 0″ – ” –region ” – “Ref”: “AWS::Region” – ” –resource Instance” – ” –stack ” – “Ref”: “AWS::StackName” – “\n” – “\n” Tags: – Key: Name Value: !Sub “${Hostname}” – Key: awPurpose Value: !Sub “${awPurpose}” – Key: awChargeTo Value: !Sub “${awChargeTo}” – Key: awRegion Value: !Sub “${awRegion}” – Key: awExpiry Value: !Sub “${awExpiry}” – Key: awBusinessSegment Value: !Sub “${awBusinessSegment}” – Key: awEnvironment Value: !Sub “${awEnvironment}” – Key: awApplication Value: !Sub “${awApplication}” NetworkInterfaces: – NetworkInterfaceId: !Ref NIC1 DeviceIndex: 0 Outputs: InstanceId: Description: ‘InstanceId’ Value: !Ref Instance Export: Name: !Sub ‘${Hostname}-${awEnvironment}-InstanceId’ InstancePrivateIP: Description: ‘InstancePrivateIP’ Value: !GetAtt Instance.PrivateIp Export: Name: !Sub ‘${Hostname}-${awEnvironment}-InstancePrivateIP’ [/code]

So my question now is, why doesn’t Azure also support YAML?

Add VC Accounts to Microsoft Teams Channels with Azure Automation

At cloudstep.io® HQ Microsoft Teams is a big part of how we organise digital asset structure in the business. We are a consulting firm by trade, as new prospects become paying customers we add them as a team. The default General channel is used for administration and accounts, additional channels are created per project name or scope of works. We find ourselves no longer needing to going into dark corners of SharePoint administration (commonly referred to in the office as ‘SwearPoint!’). We have adopted Microsoft Teams as our preferred web, audio and video conferencing platform for internal and external meetings. Our board room video conferencing unit runs a full version of Windows 10 and Microsoft Teams that we setup as a ‘do it yourself‘ versus the off the shelf room systems. The VC unit requirements we had were:

  • cloudstep.io®, our web application uses a full desktop browser experience.
  • Mouse and keyboard are preferred for web navigation inside the app.
  • VC to have full OS is preferred to eliminate employees having to BYOD and connect either physically or wirelessly for screen presentation.
  • We can connect to third party conferencing platforms by installing the addons for guest access (zoom, webex, gotomeeting, chime etc) with our partner lead meetings direct onto the machine.
  • Wirelessly present our Macs, iPads, iPhones, Androids and Windows laptops.
  • We are all ‘power users‘ and can handle the meeting join experience in Microsoft Teams client without the need for a single ‘click-to-join’ button on the table which the Microsoft Teams Room (MTR) system provides via a touch device.

We have a boardroom account that has a 365 license to be able to leverage the desktop tools. Windows 10 automatically logs in each morning and the Microsoft Teams client is started automatically. The bill of materials is notably:

  • Intel NUC
  • Windows 10
    • Teams Client
    • Office 365 Pro Plus (Word, Excel, PowerPoint, OneNote)
    • Windows 10 Calendar (Connect to Office 365 Mailbox)
    • AirServer client (ChromeCast, MiraCast, AirPlay)
    • Chrome Browser
  • Office 365 user license
  • Logitech Meetup camera
  • Biggest screen we could fit in the room
  • Microsoft Bluetooth keyboard and mouse

The VC mailbox type is set to ‘room‘ with the following to enhance the experience for scheduled meetings in the board room:

[code language="powershell"]
#Add tips when searching in Outlook
Set-Mailbox -Identity $VC -MailTip "This room is equipped to support native Teams & Skype for Business Meetings remember to add meeting invite details via the Teams outlook button in the ribbon." 

#Auto Accept
Set-CalendarProcessing -Identity $VC -AutomateProcessing AutoAccept -AddOrganizerToSubject $false -RemovePrivateProperty $false -DeleteComments $false -DeleteSubject $false –AddAdditionalResponse $true –AdditionalResponse "Your meeting is now scheduled and if it was enabled as a Teams Meeting will be available from the conference room client."
[/code]

This has worked well in the last 12 months, the only user experience problem we have had is when running a meeting from the VC unit, the account isn’t a member of the team where the data attempting to be presented is stored and therefor cannot see/open the content. A simple solution for this is automation. We looked to investigated two automation solutions available in the Microsoft services offering we have available.

  1. Flow (Office 365 Suite)
  2. Azure Automation (Azure Subscription)

Unfortunately option 1 didn’t have any native integration for triggers based on Office 365 groups or teams creation. So we resorted to a quick Azure Powershell Runbook that executes on a simple schedule. The steps needed to run were:

  1. Get a list of all the teams.
  2. Query them against the UnifiedGroup properties to get…
    1. AccessType equals ‘Public
    2. CreationDate within 2 days
  3. Check the newly created teams group membership for the VC unit username.
  4. If it doesn’t exist add the VC unit as the role ‘member‘.
[code language="powershell"]
Write-verbose "Getting Credentials ..." -Verbose
$Credentials = Get-AutomationPSCredential -Name 'Admin-365'
Write-verbose  "Credential Imported : $($Credentials.UserName)" -Verbose

$o365Cred = New-Object System.Management.Automation.PSCredential ($Credentials.UserName, $Credentials.Password)
Write-verbose  "Credential Loaded : $($o365Cred.UserName)" -Verbose
Write-verbose 'Connecting to 365 ...' -Verbose
$Session = New-PSSession –ConfigurationName Microsoft.Exchange -ConnectionUri https://outlook.office365.com/powershell-liveid/ -Credential $Credentials -Authentication Basic -AllowRedirection
Write-verbose 'Importing UnifiedGroups PowerShell Commands ...' -Verbose
Import-PSSession -Session $Session -DisableNameChecking:$true -AllowClobber:$true | Out-Null
Write-verbose 'Connecting to Teams ...' -Verbose
Connect-MicrosoftTeams -Credential $Credentials

$creationdate = ((Get-Date).AddDays(-2))
$teams = get-team
#$groups = Get-UnifiedGroup |Where-Object {$_.WelcomeMessageEnabled -like "False" -and $_.AccessType -like "Public" -and $_.WhenCreated -ge $creationdate}
$TeamsOutput = @()
foreach ($Team in $Teams){
$UnifiedGroup = Get-UnifiedGroup -Identity $Team.GroupId
    if($UnifiedGroup.AccessType -like "Public" -and $UnifiedGroup.WhenCreated -ge $creationdate){
    Write-verbose "Processing team named: $($UnifiedGroup.DisplayName)" -Verbose
        $VC = Get-TeamUser -GroupId $Team.GroupId | Where-Object {$_.User -like "user@domain.com"} 
        If($VC.count -eq 0){
            Write-verbose "VC not member, adding..." -Verbose
            Add-TeamUser -GroupId $Team.GroupId -User "user@domain.com" -Role Member
        }else{Write-verbose "VC is member already" -Verbose}
    }

$TeamsOutput+=$UnifiedGroup
}
Write-verbose "Total teams processed for selection: $($TeamsOutput.Count)" -Verbose 
[/code]

The result is simple

Additional member added via PowerShell

Next day the board room account is logged in, the Microsoft Teams client will have access to all the teams channels, files, OneNote and apps. This is great for native Teams meetings, but also when we have customers in the board room without the need for an online meeting. The VC account has access to see the required teams and channel data to present to the physical display.

This solution doesn’t have to be for a video conferencing units, you may have some standardised members you want on all groups, or it could be certain owner enforcement or member list.

Hello Microsoft Teams! Bye bye SwearPoint, may you remain in the background forever.

Invest one hour in learning about AWS

Getting educated about cloud services early will make it easier to transition and will burnish your CV nicely too!

Amazon Web Services (AWS) offers a free virtual machine instance on AWS Elastic Compute Cloud (EC2). All you have to do is register with an email address and a creditcard. Don’t worry, they really don’t charge your card unless the machine is still running a year later or if you exceed the data allowances. 
A year ago I wanted a USA VPN so I created a free EC2 instance and installed a prepackaged machine image with OpenVPN. I logged in, run a couple of commands and it all worked. I still use it today and the first bill showed up only at the end of the year. Internet data in the USA is very very cheap. 

My starting point was a conference session very like this video: Your first hour on AWS which walks you through how to sign up and set up your first instance on AWS.

Go on! It’s only going to take an hour of your life. At the end you’re one of the cool kids who has an AWS instance in the cloud and no one can ever try to convince you the cloud is “hard” ever again. 

You can experiment with 5 gigabytes of Simple Storage Service (S3) storage too and learn how to make any piece of data a web URI accessible object.

You can learn about AWS Lamda which is computing on demand as I always imagined it would be. Your code is run in response to a trigger like uploading a file to a directory, a URI being accessed or an address receiving an email. You pay nothing until the trigger event. You don’t need to maintain a virtual machine instance either, the code just runs in its own private environment and does whatever you need on demand. You can even build high availability high performance interactive web sites with little more than S3 and Lamda. 

When you’re done with the tutorial you can reflect on how easy all of this is and ask yourself the question “how much faster could my organisation respond to change if we used these tools?” 

AWS obtain PROTECTED level certification for Australian Region

Earlier this week Amazon Web Services made a statement, indicating that the battle of tier-one public cloud providers is still heating up. Yesterday Matthew Graham (AWS Head of Security Assurance for Australia and New Zealand) announced that The Australian Cyber Security Centre (ACSC) had awarded PROTECTED certification to AWS for 42 of their cloud services. 

In what appears to be a tactical move that has been executed hot off the trail of Microsoft announcing their PROTECTED accredited Azure Central Regions in the back half of last year. This clearly demonstrates that AWS aren’t prepared to reduce the boil to a gentle simmer any time soon.

Graham announced “You will find AWS on the ACSC’s Certified Cloud Services List (CCSL) at PROTECTED for AWS services, including Amazon Elastic Compute Cloud (Amazon EC2), Amazon Simple Storage Service (Amazon S3), AWS Lambda, AWS Key Management Service (AWS KMS), and Amazon GuardDuty.”

He continued to state “We worked with the ACSC to develop a solution that meets Australian government security requirements while also offering a breadth of services so you can run highly sensitive workloads on AWS at scale. These certified AWS services are available within our existing AWS Asia-Pacific (Sydney) Region and cover service categories such as compute, storage, network, database, security, analytics, application integration, management and governance. “

Finally, delivering a seemingly well orchestrated jab “Importantly, all certified services are available at current public prices, which ensures that you are able to use them without paying a premium for security.”

It is no secret that the blue team currently charges a premium for entry into their PROTECTED level facility (upon completion of a lengthy eligibility assessment process) due to a finite amount of capacity available.

Both vendors state that consumers must configure services in line with the guidance in the respective ACSC certification report and consumer guidelines. This highlights that additional security controls must be implemented to ensure workloads are secured head to toe whilst storing protected level data. Ergo, certification is not implicit by nature of consuming accredited services.

AWS have released the IRAP assessment reports under NDA within their Artefact repository. For more information, review the official press release here.

Using the AWS CLI for Process Automation

Amazon Web Services is a well established cloud provider. In this blog, I am going to explore how we can interface with the orange cloud titan programmatically. First of all, lets explore why we may want to do this. You might be thinking “But hey, the folks at AWS have built a slick web interface which offers all the capability I could ever need.”Whilst this is true, repetitive tasks quickly become onerous. Additionally, manual repetition introduces the opportunity to introduce human error. That sounds like something we should avoid, right? After all, many of the core tenets of the DevOps movement is built on these principles (“To increase the speed, efficiency and quality of software delivery”– amongst others.)

From a technology perspective, we achieve this by establishing automated services. This presents a significant speed advantage as automated processes are much faster than their manual counterparts. The quality of the entire release process improves because steps in the pipeline become standardised, thus creating predictable outcomes.

Here at cloudstep, this is one of our core beliefs when operating a cloud infrastructure platform. Simply put, the portal is a great place to look around and check reporting metrics. However, any services should be provisioned as code. Once again, to realise efficiency and improve overall quality.

How do we go about this and what are some example use cases?”

AWS provide an open source CLI bundle which enables you to interface directly with their public API’s. Typically speaking, this is done using a terminal of your choice (Linux shells, Windows Command Line, PowerShell, Puty, Remotely.. You name it, its there.) Additionally, they also offer SDK’s which provide a great starting point for developing applications on-top of their services in many different languages (PowerShell, Java, .NET, JavaScript, Ruby, Python, PHP and GO.)   

So lets get into it… The first thing you’ll want to do is walk through the process of aligning your operating environment with any mandatory prerequisites, then you can get install the AWS CLI tools in a flavour of your choice. The process is well documented, so I wont cover it off here.

Link – https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html

Once you have the tools installed, you will need to provide the CLI tools with a base level of configuration which is stored in a profile of your choice. Running “AWS Configure” from a terminal of your choice is the fastest way to do this. Here you will provide IAM credentials to interface with your tenant, a default region and an output format. For the purpose of this example I’ve set my region to “ap-southeast-2” and my output format to “JSON.”

aws configure example

From here I could run “aws ec2 describe-instances” to validate that my profile had been defined correctly within the AWS CLI tools. The expected return would be a list of EC2 instances hosted within my AWS subscription as shown below.

aws ec2 describe-instances example

This shouldn’t take more than 5 minutes to get you up and running. However, don’t stop here. The AWS CLI supports almost all of the capability which can be found within the management portal. Therefore, if you’re in an operations role and your company is investing in AWS in 2019. You should be spending some time to learn about how to interface with services such as DynamoDB, EC2, S3/Glacier, IAM, SNS and SWF using the AWS CLI.

Lets have a look at a more practical example whereby automating a simple task can potentially save you hours of time each year. As a Mac user (you’ve probably already picked up on that) I often need to fire up a windows PC for Visual Studio or Visio. AWS is a great use case for this. I simply fire up my machine when I need it and shut it down when I’m done. I pay them a couple of bucks a month for some storage costs and some compute hours and I’m a happy camper. Simple right?

Lets unpack it further. I am not only a happy camper. I’m also a lazy camper. Firing up my VM to do my day job means:

  • Opening my browser and navigating to the AWS management console
  • Authenticating to the console
  • Navigating to the EC2 service
  • Scrolling through a long list of instances looking for my jumpbox
  • Starting my VM
  • Waiting for the network interface to refresh so I can get the public IP for RDP purposes.

This is all getting too hard right? All of this has to happen before I can even do my job and sometimes I have to do this a few times each day. Maybe its time to practice what I preach? I could automate all of this using the AWS tools for PowerShell, which would allow me to automate this process by running a script which saves me hours each year (employers love that.) Whilst this example wont necessarily increase the overall quality of my work, it does provide me with a predictable outcome every single time.

For a measly 20 lines of PowerShell I was able to define an executable script which authenticates to the AWS EC2 service, checks the power state of my VM in question. If the VM is already running it will return the connectivity details for my RDP client. If the VMis not running, it will fire up my instance, wait for the NIC to refresh and then return the connectivity details for my RDP client. I then have a script based on the same logic to shutdown my VM to save money when I’m not using the service. All of this takes less than 5 seconds to execute.

PowerShell Automation Example

The AWS CLI tools provide an interface to interact with the cloud provider programmatically. In this simple example we looked at automating a manual process which has the potential to save hours of time each year whilst also ensuring a predictable outcome for each execution. Each of the serious public cloud players offer similar capability. If you are looking to increase your overall efficiency, improve the quality of your work whilst automating monotonous tasks, consider investing some effort into learning a how to interface with your cloud provider of choice programmatically. You will be surprised how many repetitive tasks you can bowl over when you maximise the usage of the tools you have available to you.