Month: September 2016

Azure – High level discussion of Azure Storage Architecture

Windows Azure Storage is a cloud solution that provides customers to store seamlessly limitless amount of data for a any duration of time. It has been in production since November 2008. It is used in storing application data, Live Streaming, social networking search, gaming and music content, etc.,

Once you have your data stored in Azure storage, You can access your data any time and from anywhere. And you only pay for what you use and store. Currently we have thousands of customers who are already using Azure Storage Services.

Visit the Azure Portal to create your free subscription and try out Azure Storage. Also, check out the Microsoft article for a jump start and pre-requisites required to use the Azure Storage.

Why use Azure Storage?

Disaster Recovery: Azure Storage stores your data miles apart (minimum 400 miles) in different data centres. This provides a strong guard against natural calamities like earthquakes, tornadoes etc., Replication options like – LRS, ZRS, GRS, RA-GRS are provided, which can be chosen as per the business needs.

Multi Tenancy: As with other services, Azure storage uses the concept of shared tenancy. What this means is, to reduce the storage cost, depending on the varying work loads of the customer, data from multiple customers are served from the same storage infrastructure. This reduces the amount of storage space to be provisioned at a time than having each services run on their own dedicated hardware.

Global Name-space: For ease of use, Azure Storage implements a Global Namespace that allows the data to be stored and accessed in a consistent manner from any location in the world.

Global Partitioned Name-space:

Exabytes of data and beyond are stored in Azure Storage. Azure had to come up with a solution that allowed its clients to store and retrieve data without much of a hassle. To provide this capability, Azure leveraged DNS part of the storage name-space and break it down to three parts: Account Name, a Partition Name and an Object Name.

Syntax: http(s)://AccountName.<service>.core.windows.net/PartitionName/ObjectName

Account Name: This is the customer selected Storage Account Name (entered while creating the storage account– Azure portal or Azure Powershell). This Account Name is used to locate the primary storage cluster and the data centre where the requested data is stored. This primary location is where the preceding requests go to reach the data of that account.

Partition Name: This name locates the data once the request reaches the primary cluster. It is also used to scale out the access to data across the nodes depending on the traffic.

Object Name: This identifies the individual objects within that partition.

For Blobs, the full blob name is the PartitionName.

For Tables, each entity (row) in the table has a primary key that consists of two properties: the PartitionName and the ObjectName. This distinction allows applications using Tables to group rows into the same partition to perform atomic transactions across them.

For Queues, the queue name is the PartitionName and each message has an ObjectName to uniquely identify it within the queue.

Architectural Components:

Azure_Storage

Storage Stamp: This is a cluster of N racks of storage nodes. Each rack is built out as a separate fault domain with redundant networking and power. The goal is to keep the stamp around 70% utilized in terms of capacity, transitions and bandwidth. This is because ~20% is kept as a reserve for (a) disk short stroking to gain better seek timse and higher throughput by utilizing the outer tracks of the disks and (b) to continue providing storage capacity and availability in the presence of a rack failure within a stamp. When the storage stamp reaches 70% utilization, the location service migrates accounts to different stamps using Inter-Stamp replication.

Location Service: Manages all the storage stamps. Also responsible for managing the account name-spaces across all stamps. The LS itself is distributed across two geographical locations for its own disaster recovery.

Azure Storage provides storage from multiple locations. Each location is a data centre, which holds multiple storage stamp. To provision additional capacity, the LS has the ability to add new regions, new locations to regions and new stamps to locations. The LS can then allocate new storage accounts to those new stamps for customers as well as load balance (migrate) existing storage accounts from older stamps to new stamps.

As shown in the figure, When an application requests new Storage Account for storing data, it specifies the location affinity for the storage (Example: US North). The LS then chooses a storage stamp within that location as the primary stamp for the account. The LS then stores the account meta-data information in the chosen storage stamp, which tells the stamp to start taking traffic for the assigned account. The LS then updates the DNS to allow requests to now route from the name https://AccountName.service.core.windows.net/ to that storage stamp’s virtual IP (VIP, an IP address the storage stamp exposes for external traffic).

Three Layers within Storage Stamp:

Stream Layer: This layer stores the bits on disk and is in charge of distributing and replicating the data across many servers to keep data durable within a storage stamp. The stream layer can be thought of as a distributed file system layer within a stamp. It
understands files, called “streams”, how to store them, how to replicate them, etc., but it does not understand higher level object constructs or their semantics.

Partition Layer: The partition layer is built for (a) managing and understanding higher level data abstractions (Blob, Table, Queue), (b) providing a scalable object namespace, (c) providing transaction ordering and strong consistency for objects, (d) storing object data on top of the stream layer, and (e) caching object data to reduce disk I/O.
Front End Layer: The Front-End (FE) layer consists of a set of stateless servers that take incoming requests. Upon receiving a request, an FE looks up the AccountName, authenticates and authorizes the request, then routes the request to a partition server in the partition layer (based on the PartitionName). The system maintains a Partition Map that keeps track of the PartitionName ranges and which partition server is serving which PartitionNames. The FE servers cache the Partition Map and use it to determine which partition server to forward each request to. The FE servers also stream large objects directly from the stream layer and cache frequently accessed data for efficiency.

Two Replication Engines:

Intra-Stamp Replication (Stream Layer): This system provides synchronous replication and is focused on making sure all the data written into a stamp is kept durable within that stamp. It keeps enough replicas of the data across different nodes in different fault domains to keep data durable within the stamp in the face of disk, node, and rack failures. Intra-stamp replication is done completely by the stream layer and is on the critical path of the customer’s write requests. Once a transaction has been replicated successfully with intra-stamp replication, success can be returned back to the customer.

Inter-Stamp Replication (Partition Layer): This system provides asynchronous replication and is focused on replicating data across stamps. Inter-stamp replication is done in the background and is off the critical path of the customer’s request. This replication is at the object level, where either the whole object is replicated or recent delta changes are replicated for a given account. Inter-stamp replication is used for (a) keeping a copy of an account’s data in two locations for disaster recovery and (b) migrating an account’s data between stamps. Inter-stamp replication is configured for an account by the location service and performed by the partition layer.

Note: The above content has been summarized from a technical paper titled: “Windows Azure Storage: A Highly Available Cloud Storage Service with Strong Consistency” released by Microsoft.
You can download the PDF here.

Advertisements

Azure – Setting up Azure Subscription using PowerShell

The very fact that you are here reading this blog is because you have selected to manage your Azure service using Powreshell. Welcome to the team!!

I assume that you are already have a valid Azure subscription. Powershell 3.0 or higher and have the Windows Azure Powershell modules installed. If you do not have the Azure Powershell modules, you can download the Azure PowerShell module here.

Authenticating with a Certificate

You have to download the .publishsettings file from the Microsoft Azure . You can use the below command:

Get-AzurePublishSettingsFile

This will automatically ask you to select your favourite browser, so you can login to Microsoft Azure website.

get-publishfile

Now login with your credentials, that you always do with the Azure Portal

login

The file that we downloaded is very important and we have to handle it with a lot of care. Any one who can get their hands on this file, will have complete access to resources under that subscription. Microsoft imposes a limit on the total number of management certificates that can be associated with a subscription at a time. The number is 100 at the time of writing this blog. Each time you run the Get-AzurePublishSettingsFile cmdlet, Azure generates a new management certificate.

Importing the .pubishsettings file

The next step is to import the .publishsettings file that we just downloaded. I have saved in “E:\Work\Powershell\scripts”, so I am going to run the Import-AzurePublishSettingsFile cmdlet with the complete file path to the settings file.

Import-AzurePublishSettingsFile "E:\Work\Powershell\scripts\Pay-As-You-Go-9-9-2016-credentials.publishsettings"

import-settingfile

As you can see that the cmdlet outputs the subscription information, telling you that the settings are successfully imported.

To double confirm, you can run the Get-AzureSubscription cmdlet.

get-subscription

This cmdlet also tells you, if this subscription is your “Current” / “Default” subscription.

If you have multiple subscriptions, use the Set-AzureSubscription cmdlet to set any azure subscription as “Current” or “Default”.

Also, use the Select-AzureSubscription if you want to switch between subscriptions while working with Powershell.

Azure – RDP from Linux box to Azure Virtual Machine (Windows Server)

This topic is pretty straight forward. Once you have your Azure Windows Server spun up, you can connect to it using RDP tool if your source is a Windows operating System.

But when you are in Linux the story is different, as it does not have the RDP tool. We have plenty of open source tools that we can use to make a connection from Linux Box to Windows Server.

Since we are discussing about Azure, I am using an Azure VM. Below is the details of that machine:

VM Details

We are going to use the “RDesktop” tool to connect from Linux to Windows. Installation of RDesktop is pretty simple. You may google/bing for assistance, or check this link for the complete installation guide in RedHat/CentOS/Fedora Operating System.

Once you have your RDesktop setup. Enable the RDP port- 3389 in your Azure virtual Machine. You can achieve this by adding “Inbound Security Rules” into your Network Security Group, via the Azure Portal.

For demo purposes, I am allowing all incoming traffic for the port – 3389. However, incase of production environment, you may have to provide appropriate rules for your NSG (Network Security Group) and ACLs (Access Control List) for better security.

NSG_Allow_All

Once RDesktop and RDP ports are configured. You may now run the RDesktop command to connect to your Virtual Machine.

Command: sudo rdesktop <IP Address>

rdesktop_fail

As you can see in the above screenshot, the connection fails. This is an expected behavior, because by default the RDP connection is set to “Allow connections only from computers running Remote Desktop with Network Level Authentication”

NLA_Enabled

In order to connect from Linux machines, we have to disable the NLA.

NLA_Disabled

Now give it another try, the remote connection works like a charm !

rdesktop_success

Powershell – Understanding Select-Object

Understanding how “Select-Object” works, can differentiate an average from a good Powershell user. So lets dedicate some time in understanding how Select-Object works.

Select-Object is mainly used to override the default display of the cmdlets. Example:

select

When you run the Get-Service cmdlet, the output displays the Status, Name and DisplayName. By piping the output of Get-Service to Select-Object, you can override the default display for that object type.

(Piping the Get-Service output to Get-Member, allows you to understand what other properties can be used with the Select-Object)

Select-Object changes the object type

Notice the below two examples and observe the TypeName in both the case.

Get-Service | gm

get-service

Get-Service | Select-Object name | gm

get-service-select

From the above two outputs, you can clearly see that the Select-Object is changing the object from System.ServiceProcess.ServiceController to Selected.System.ServiceProcess.ServiceController. That, it is “Selecting” only few properties from the “original” service object. Now, you will not be able to treat the new object as a service object. Hence you cannot run the operations that you were running on the good-old service object. Hence, the Select-Object HAS to be the last cmdlet in the pipeline.

Selecting subset of objects

Select-Object allows us to choose a subset of objects either from the beginning, from the end, or a chunk of object from a random range. Powershell provides, “First”, “Last” and “Skip” parameters just for these operations.

We are running the below three cmdlets as a demonstration:

Get-Service | Sort-Object name | Select-Object -first 5

Get-Service | Sort-Object name | Select-Object -last 5

Get-Service | Sort-Object name | Select-Object -first 3 -Skip 2

subset

Note: No matter in which order you are specifying the parameters to the Select-Object. If you have specified -Skip, it will ALWAYS execute first, and then -First or -Last (if you have specified along with skip).

Select-Object allows Custom Properties

Custom Properties are the ones which do not come pre-loaded with Powershell. instead they are dynamically created by the user on the go.

Example:

Get-Process | select name, id, @{label='TotalMemory';expression={($_.pm + $_.vm)/ 1MB -as [int]}}

custom properties

What did we just do?

  • We wanted to calculate the TotalMemory of a service. Since we do not have a default parameter that provides us this value. We decided to create a custom property of our own.
  • The structure starting with ‘@‘ sign is called a hash table. A Hash Table contains entries in the form of a name-value pair. Custom Properties comes with two pairs. Each pair is separated by a semi-colon.
  • The first key is called as “Label” or “Name“. The value for this key is what you want to appear in the column header of your custom property.
  • The second key is called the “Expression“. The value for this key,  is the code for the powershell to run/execute to create the resulting row in the custom property.
  • The $_ holder is a place holder for the current object in the pipeline. This can be replaced by $PSItem, both have same meaning.
  • Each @ structure represents a single custom property. You can have as many custom property as you like.

Expanding Properties

Imagine a situation, you want to grab process from the computers [dc=company, dc=pri] in your Active Directory.

You are sure that the below command will fetch the desired computers from the Active Directory.

Get-AdComputer -Filter * -SearchBase "dc=company, dc=pri"

And you also know that the Get-Process has a -ComputerName as a parameter. So you go ahead an happily run the below command:

Get-Process -ComputerName (Get-AdComputer -Filter * -SearchBase "dc=company, dc=pri")

To your surprise the command does not generate the desired result. This is because the Get-AdComputer is generating ADComputer objects. However, the -ComputerName parameter in the Get-Process cmdlet is asking for String[] type input. This is where the -ExpandProperty parameter of the Select-Object shines.

Have a look at this command:

Get-Process -ComputerName (Get-AdComputer -Filter * -SearchBase "dc=company, dc=pri" | Select-Object -ExpandProperty Name)

When Powershell executes this command, it is fetching the desired computers and then expanding the Name property. This has an effect of writing a collection of string to the pipeline, instead of bunch of objects.

-ExpandProperty is a handy technique when you want to save property value to variable.

bits1

As shown above, the Select-Object wrote a ServiceController object into the variable. Hence we need to use a sub-expression to fetch the display name.

bits2

In this case, the Select-Object wrote a String into the variable. This way, it is very easy to access the display name. You can expand single property or a bunch or properties.

From Powershell v3, we have a shortcut way to implicitly expand a property:

(Get-Process).name

implicit

Confirming that the output were String objects:

(Get-Process).name | gm

implicit 2

Some properties are collection of other objects. Example:

expanding1

We can use the -ExpandProperty to “expand” them into their stand-alone objects.

expanding2

We can also use the shortcut trick to get the same results.

expanding3

Powershell – Debugging Techniques

This blog discuses some high level Debugging Techniques while working with Powershell. The contents of this blog are taken from the book – “Learn WINDOWS POWERSHELL, In a month of Lunches” By, Don Jones (All the credit goes to Don)

I am writing this blog, because we do not have enuf content on the web that discusses about the “Debugging” or “Troubleshooting” approaches while working with Powershell. And Don’s book helped me a lot in my personal endeavour. Now it is my turn to give it back to the Powershell Community.

First Computer Bug

There are two broad categories of bugs in the Powershell world. Firstly, the syntax errors and secondly, logic errors.

Syntax Errors

A Syntax Error means you typed something wrong. It might be a command name that you misspelled (Gte-Content instead of Get-Content) or it might be that you got the actual syntax of the command wrong (forgetting the hyphen between the verb and noun of a cmdlet name, or using a colon instead of a space to separate a parameter’s name and value). Whatever the cause, correcting your typing will solve the problem.

Powershell will usually tell you, in explicit detail, where the problem is. Powershell might not know what the problem is, but it will usually get pretty close to the location of the error. Example, here is a syntactically incorrect command and the resulting error message:

get-content

That is pretty clear: We used a parameter named -file, and Powershell could not find one. The error occurred on the line 1 of my  command and at character position 18.

Here is a checklist that will help you in overcoming the Syntax Errors:

  •  Make sure you typed the cmdlet name correctly. If you used an alias, make sure it is typed correctly, and that it points to the cmdlet you think it does.
  • Make sure parameter names are preceded by a dash and are followed by a space. Make sure you are using the correct parameter name (read the help!) and if you are abbreviating the parameter name, make sure you are providing enough characters to uniquely identify that parameter.
  • Most of the Powershell’s punctuation comes in pairs: single quotes, double quotes, square brackets, curly braces, and parenthesis are all good examples. Improper nesting, like ({this)}, means you are ending a pair before ending the pair it encloses.
  • Watch your spaces. In Powershell, spaces are special characters that indicate a separation between command elements. Powershell is not that case-sensitive, but it is very space-sensitive. There is a space in between parameter names and values. There is a space after a cmdlet name and before any parameter or values. There is a space after one parameter and before the next. Do not forget those.

Red Alert

The RED Text usually freaks us out. You can change the colour of the error message text to green.

(Get-Host).PrivateData.ErrorForegroundColor = 'green'

That only lasts for the duration of the shell session, so we can either put that in a profile script or we can do it before start working in the shell.

Logic Errors

Logic errors mean that your script or command is not doing what you want it to do, but it is not necessarily generating an error.

Some logic error will provide straightforward errors. You should know what to do with a  “file not found” error, or an “access denied” message, but sometimes errors are not always so clear, Get-WmiObject, for example, can produce an “RPC server not found” error if it is not able to locate a remote computer, etc.,

But the most vexing logic errors are the ones that do not produce any error at all – they just prevent your script or command from working properly. Check the below code snippet:

$name = Read-Host "Enter computer name"
if (Test-Connection $name) {
    Get-WmiObject Win32_BIOS -ComputerName $nmae
}

Logic errors, like syntax errors, can come from typos and one of such errors is listed in the above code snippet.

Debugging causes a lot of frustration, and we need to know the below things before we debug any script or command:

  • You cannot debug a script or command unless you have a clear expectation of what it is going to do.
  • You must execute your script an examine its reality (what  it actually does), and compare that reality to your expectations. When reality and your expectations differ, you have found the bug.
  • While executing the script and examining it, you need to read very, very carefully, so that you can spot typos. Sometimes using a different font can help.

Identifying your expectations

If you do not know what a command or a script should do, then you can never debug it. We are using the same code snippet as above to see the process involved.

$name = Read-Host "Enter computer name"

I expect that this will display “Enter Computer name:” on the screen, and allow me to type something. What ever I type will be stored in the variable $name.

if (Test-Connection $name) {

My expectation is that this will run the Test-Connection command, passing it the computer name in $name. I expect that this will ping the computer name I typed. I see that this is enclosed in an IF construct, so I expect that Test-Connection must return a True or False value. So, if the computer can be pinged, it will return True, and whatever is inside the IF construct will execute next.

Get-WmiObject Win32_BIOS -ComputerName $name

I expect that this will return Get-WmiObject and retrieve the Win32_BIOS class. We know that -class is the first parameter, so the Win32_BIOS will be fed to the -class parameter. I see that the -computername parameter is also specified, and it is being passed the computer name from the $name variable. Oh, wait – there is a typo. See, just a careful read-through of the script found a problem. I am going to leave the typo in there, though, and pretend that I was not being so careful with it (for now). I want to show you some other ways that you might have found it.

}

That closes the IF construct. With my expectations written down, it is time to start seeing where they differ from reality.

Adding Trace Code

The first trick is to add trace code to the script. It all starts with a helpful command called Write-Debug, which simply takes a message that you want it to display.

Write-Debug "Test Message

If you are following along, you will notice that Write-Debug will not produce any output.

Write-Debug actually sends your message to an alternate pipeline called Debug pipeline. Powershell has several of these alternate pipelines: Error, Warning, Debug and so forth. By default, the Debug pipeline’s switch  is set to SilentlyContinue, which is the same as OFF. The result is that all Write-Debug messages are suppressed.

Lets go ahead and change the setting in the script by adding this to the top:

$DebugPreference = "Continue"

Now we are free to add Write-Host statements to our script. Below is the revised script:

$DebugPreference = "Continue"

$name = Read-Host "Enter computer name"
Write-Debug "`$name contains $name"

if (Test-Connection $name) {
Write-Debug "Test-Connection was True"
    Get-WmiObject Win32_BIOS -ComputerName $nmae
} else {
    Write-Debug "Test-Connection was False"
}
You can see that I have added a Write-Debug to the inside of the IF construct, I even added an Else potion to the construct, containing a third Write-Debug message. That way, no matter which way the IF construct’s logic goes, I’ll see some output and know what is happening inside the script.
If you run the command, you will get the below error:
typo
That is where you will realize that there is a typo in that $name variable. Powershell is clearly telling us that there is a problem with the -computername parameter; if we look carefully at just that portion of the script, the $nmae typo is more obvious.
After fixing that and running  the script again:
notypo
That looks like what we want.
Now, we need to test the opposite situation, what happens when we provide a computer name that is not valid?
testfailed
That is certainly not what we are looking for. That is a logic error: the Test-Connection cmdlet clearly is not doing what I expected, which was to return a simple True or False value.
Stepping out of the script and just work with the Test-Connection from the command line:
tesst-connection
failagainOkay, that is definitely not what I expected. When I used the command with a valid computer name, I get back a table of results, not a True or False value. When I use it with an invalid computer name, I still get an error.
Reading the help, Help Test-Connection -full. I see that the command “returns the echo response replies”. The help also says that, “unlike the traditional ‘ping’ command, Test-Connection returns a Win32_PingStatus Object… bot you can use the -quiet parameter to force it to return only a Boolean value.” Yes, Please !
Looking at the -quiet parameter, I see it “supresses all errors and return $True if any pings succeed and $False if all failed.” That is what I want, so I will modify the script accordingly.
$DebugPreference = "Continue"

$name = Read-Host "Enter computer name"
Write-Debug "`$name contains $name"

if (Test-Connection $name -Quiet) {
Write-Debug "Test-Connection was True"
    Get-WmiObject Win32_BIOS -ComputerName $name
} else {
    Write-Debug "Test-Connection was False"
}
Running the script again, testing it with both a valid and invalid computer name. Below is the output:
works
In short:
  • Whenever I change the contents of a variable, I use Write-Debug to output the variable, just to I can check those contents.
  • Whenever I am going to read the value of a property or a variable, I use Write-Debug to output that property or variable, so that I can see what is going on inside the script.
  • Any time I have a loop or logic construct, I build it in such a way that I get a Write-Debug message no matter how the loop or logic works out.

Working with break points

Having to wade through a lot of debug messages is cumbersome in lengthy /complex scripts. So we fall to use Breakpoints, to make the debugging process a bit easy.

A breakpoint is a defined area where a script will pause its execution, allowing you to examine the environment that the script is running within. Powershell can be configured to break when:

  • Your script reaches a certain line
  • A variable is read and/or changed
  • A specific command is executed

In the first instance, you must specify the script that you are referring to. In the second and third situations, you can choose to specify a script, and the breaking point will only be active for that script. If you don’t, the breakpoint will occur globally through out the shell when that variable is read or written, or that command is executed.

Example, suppose I want to have the script stop immediately after Line 1 finishes executing, meaning that I want to break before the Line 2. Then I will run the command:

Set-PSBreakpoint -Script E:\Work\Powershell\scripts\demo.ps1 -Line 2

breakpoint

I also want to be notified whenever the $name variable is accessed, so I will run the below command:

Set-PSBreakpoint -Script E:\Work\Powershell\scripts\demo.ps1 -Variable name -mode Read

anotherbreakpoint

Notice that the variable name is just “name” and not “$name”. Variable names do not  include the dollar sign; $ is just a cue to the shell telling it that you wish to use the contents of a variable.

With those two breakpoints set, I will run the script. After entering the computer name, the script will break. The shell modifies the command-line prompt, indicating that I am in a suspended mode. Here I can examine the contents of variables, execute commands and so on. When I am done, I can run Exit to resume script execution. Here is how it all works:

breakpointfinal

This gives me the chance to test commands, see what is inside variables or properties, and so forth, without having to add a  lot of write-debug commands.

When I am done debugging, I can remove the breakpoints:

Get-PSBreakpoint | Remove-PSBreakpoint

Breakpoints are also supported in Powershell ISE. To set a line breakpoint, move your cursor to the desired line and press F9.The ISE will visually indicate where line breakpoints occur, using a red highlight. If you run the script within the ISE, breakpoints lines will be highlighted in yellow when the script reaches one of those lines. At that time, you can hover your cursor over any variable to see a tooltip with the contents of that variable.