IBM Softlayer uses many types of storage to store the data. One of which is Amazon S3. I have written a simple PowerShell script to delete IBM COS S3 files older than X days.
The PS Script uses AWS CLI commands so you will need AWS CLI installed on the windows machine from where you will run this script. The script can also be scheduled as a task to run every day.
Delete IBM COS S3 files older than X days
This is a continuation of my first blog on Cloudlyn. Below is the link to my first blog. It guides you to register for Cloudlyn if you have an Azure subscription.
AZURE – COST MANAGEMENT BETTER THAN EVER USING CLOUDYN (REGISTRATION)
In this post, let us take a first-look into Cloudlyn’s Cost management console and the features Cloudlyn offers.
Once you have completed the registration with Cloudlyn as explained in my first blog (link shared above). You can access the Cloudlyn’s Cost management portal by:
Once you navigate to the URL, below is the page you see. This is Cloudlyn’s Management Dashboard.
Below is the Annual Projected Annual cost. I have used only Azure Storage and Azure Network. If you have used more services, their costs will also be projected here.
We can also view Current and Previous Month Projected Cost. This is very useful to track changes in infrastructure costing more than usual.
The “Actual Cost Analysis” will provide cost by Services, Providers, and Accounts. This graph can be further customized. Groups (highlighted below) provides plenty of options that we can check as per our requirement. Finally, you can choose an option from many of the Actions (highlighted below) on what to do with the report. Save / Copy / Export etc.,
The “Actual Cost Over Time” lets you pull up a report to analyze the cost over a range of time.
Cloudlyn also offers an “Alert Management” feature that alerts you when certain thresholds are crossed as per alert’s configuration.
“Cost vs. Threshold” is one of many alerts. As you can see Cloudlyn offers you many options to customize the alert policy. Alerts will be sent to an email ID.
Finally, the last noticeable feature is “Data Transfer.” You can either Analyze data transfer usage or view the trend of data transfer.
These are the noticeable features that I felt to be very valuable.
If you found any other feature to be worth mentioning, let me know in the comments section.
If the content of this blog is valuable to you, do consider sharing with your friends and colleagues.
Microsoft’s acquisition of Cloudyn will help Azure customers manage and optimize their cloud usage. Read more about the acquisition here. A message from Sharon Wagner, CEO of Cloudyn.
Azure Cost Management by Cloudyn empowers organizations to monitor cloud spend, drive organizational accountability, and optimize cloud efficiency so they can accelerate future cloud investments with confidence.
Microsoft’s acquisition of Cloudyn will help Azure customers and partners as they face the challenges of growing their multi-cloud environments. It will enable them to gain visibility, understand and optimize cloud consumption, as well as accurately project future usage.
Microsoft will continue to support multi-cloud environments, including Azure, AWS, and GCP. Azure Cost Management by Cloudyn is available for free to customers and partners managing Azure spend. Additional premium capabilities are available at no cost through June 2018, once they will become paid features.
Step 1: Login to your Azure Subscription via the Azure portal. Select the “Cost Management + Billing” blade. Then select “Cost Management” from the options on the left-hand side of the pane.
Click on “Go to Cost Management”
Step 2: Once you click the”Go to Cost Management” button, you will be redirected to Cloudlyn’s page to set up your Cost Management details.
Enter your organization name and the type of Azure access you have on your Azure account. I have a personal subscription, so I have chosen as “Azure Individual Subscription Owner.”
Step 3: Cloudlyn Account name and Tenant ID will be automatically populated. Now, select the offer-ID from the drop-down list.
If you do not know your Offer-ID, then go back to your Azure portal. Click on “Subscriptions,” that should provide you the type of subscription that you have.
Step 4: Click “Next”
Step 5: Click “Next”
Step 6: We are done with the registration with Cloudlyn. Cloudlyn needs about 2 hours for collecting the data.
This document illustrates how we can create a VPN using the AMI – OpenVPN which is available in AWS MarketPlace.
Below shows the architecture that we would be able to achieve:
We will set up an AWS VPC where we will launch our servers. One of the servers will be launched using the OpenVPN AMI. The other servers will be on public and private subnets within the VPC.
Steps to Achieve are as below:
Step 1: Login into AWS Console and choose VPC.
Step 2: Select Service – VPC
Step 3: Create a VPC. Provide the Name Tag and CIDR details. Also, create the subnet under this VPC.
Step 4: Now, Let us go back to EC2 Service and then come back to VPC Later.
Step 5: In EC2 Service, Launch an EC2 Server and go to AWS Marketplace and choose the OpenVPN AMI. You can assign an elastic IP if necessary
Step 6: Launch this instance with all the other necessary configuration. This AMI comes with default sec groups, keep them default and launch it. Note – This is not necessary to be launched in the VPC Created.
Step 7: Launch two more instance one with public IP and the other with private IP under the VPC created.
Step 8: Now let us go back to VPC. We will need to create Internet Gateway and RouteTable. Click on VPC in the console and go to Internet Gateway. Now click on Create Internet Gateway and provide a suitable Name tag. Attached this Gateway to the VPC we created earlier using the option “Attach to VPC”
Step 9: Now, go to Route table and click on create route table. Update the Routes to have 0.0.0.0/0 (opens to the internet) and specify the IGW (internet gateway) which was created.
Step 10: Update the Subnet Association to the
Step 11: This completes the configuration on the AWS Services. Now we will need to configure the OpenVPN.
Step 12: SSH to the OpenVPN box using the username – openvpnas
Step 13: You will need to agree to the terms by typing “Yes”. Keep all the other setting default by just pushing the “Enter” button. By Default, the username will be “openvpn”. You will need to update the password using the command as per the below screenshot: sudo passwd openvpn.
Step 14: Now go to the browser and type the IP address of the OpenVPN box. Ex: https://ipaddress
Step 15: Login in with the password updated earlier on the terminal.
Step 16: Download the OpenVPN client as per the operating system. Install the client.
Step 17: Click on the icon and select the OpenVPN server and click on connect. Once connected you will be able to connect to the servers within the vpc.
Step 18: Now let us ping the Public IP and see the result.We were successful in pinging the Public Server under the VPC.
Step 19: Now let us disconnect the OpenVPN and try to connect to the private server.
Step 20: Now we see the ping requests are getting timed out. We are unable to reach the server using the private IP under the VPC.
Step 21: Let us connect the VPN using the OpenVPN and try to ping the same Private Server.
Today we are looking into PowerShell in Azure Cloud Shell. This is still in public preview as of this writing.
If you are wondering why Microsoft would introduce a PowerShell console inside the Azure Cloud Shell, then have a look at the below features:
Browser-based shell experience
Cloud Shell enables access to a browser-based command-line experience built with Azure management tasks in mind. Leverage Cloud Shell to work untethered from a local machine in a way only the cloud can provide.
Choice of preferred shell experience
Azure Cloud Shell gives you the flexibility of choosing the shell experience that best suits the way you work. Linux users can opt for a Bash experience, while Windows users can opt for PowerShell.
Pre-configured Azure workstation
Cloud Shell comes pre-installed with popular command-line tools and language support so you can work faster.
View the full tooling list for Bash experience and PowerShell experience.
Automatic authentication
Cloud Shell securely authenticates automatically on each session for instant access to your resources through the Azure CLI 2.0.
Connect your Azure File storage
Cloud Shell machines are temporary and as a result, require an Azure file share to be mounted as clouddrive to persist your $Home directory. On the first launch, Cloud Shell prompts to create a resource group, storage account, and file share on your behalf. This is a one-time step and will be automatically attached for all sessions. A single file share can be mapped and will be used by both Bash and PowerShell in Cloud Shell.
Cloud Shell runs on a temporary machine provided on a per-session, per-user basis
Cloud Shell times out after 20 minutes without interactive activity
Cloud Shell can only be accessed with a file share attached
Cloud Shell uses the same file share for both Bash and PowerShell
Cloud Shell is assigned one machine per user account
Permissions are set as a regular Linux user (Bash)
Now that we have some background knowledge on the PowerShell in Cloud Shell, let us dig more into the usage of it:
To access the Cloud Shell, click on the PowerShell icon in the Azure portal:
Once you click on the icon, a pane is opened at the bottom of the screen as shown below. You can choose from two options – BASH or PowerShell. Since we are interested in learning PowerShell in CloudShell, let us choose PowerShell as our desired option.
When you are starting for the first time, the Shell will configure an Azure File Storage. Cloud Shell machines are temporary and as a result, require an Azure file share to be mounted as clouddrive to persist your $Home directory. Alternatively, if you have multiple subscriptions, you will be allowed to choose your favorite subscription to work with.
Azure Authentication, Resource Group, Storage Account and File Storage are automatically created as shown below:
Testing an Azure command. Works perfectly.
If you are idle for more than 20 minutes, you will be kicked off the session, and you will have to start the session again:
Now let us execute the Get-ChildItem cmdlet and see what we can find.
As we can see, running the Get-ChildItem in the current scope will list out the subscriptions that your account is associated with.
Traversing one step deeper into the directory, we can see the resources related to the subscription.
Let us get into the “StroageAccounts” directory to confirm if we get to see a list of Storage Accounts under the selected subscription:
From the below information, we can see that Microsoft provides us two cmdlets to work with the cloud shell.
Get-CloudDrive provides the details of the “Azure File Share” that was created when the cloud shell started. You may continue to use the cloud share. However, if you want a new one, you can dismount and create a new one using the Dismount-CloudDrive cmdlet.
Note: Once you dismount the Azure file share, your current session will be restarted to set up a new cloud share.
I am assuming that Microsoft is using container service infrastructure to provide a session. You will get the below windows path when you query for the temp drive:
C:\Users\ContainerAdministrator\AppData\Local\Temp
Note the administrator is a “ContainerAdministrator.” The container here could be a Windows Server or a Windows Container. I am assuming it is a Windows Container since the underlying “image” comes pre-packaged with below tools and a temporary one. A typical use case scenario for Container technology.
AzCopy is a command-line utility designed for copying data from and to Microsoft Azure Blob, File and Table storage with optimal performance.
This WIKI will explain how to copy to and from Windows Virtual Machine to Azure blob storage. Download the latest AzCopy tool on Windows on your Virtual Machine
Below is the basic syntax of AzCopy:
AzCopy /Source:<source> /Dest:<destination> [Options]
1. Using the above link, once you have downloaded the AzCopy tool. The setup file will show as “MicrosoftAzureStorageTools”
2. Double click on the “MicrosoftAzureStorageTools” installer and the set-up will continue. AzCopy will be installed in a default location – “C:\Program Files (x86)\Microsoft SDKs\Azure\”. You can change the default location, during the installation process.
3. Once the installation is done, you can find the “AzCopy.exe” under the installed folder. The default location is: “C:\Program Files (x86)\Microsoft SDKs\Azure\AzCopy” OR “%ProgramFiles%\Microsoft SDKs\Azure\AzCopy”. If desired, you can add the AzCopy installation location to your system path.
Upload all blobs in a folder
Below is the syntax:
AzCopy /Source:C:\myfolder /Dest:https://myaccount.blob.core.windows.net/mycontainer /DestKey:key /S
Example:
I have created a folder called “CopyData” that has three empty text files. I will be uploading these three files to an Azure Blob container called “test-azcopy”
Now we will use AzCopy to copy this folder (recursively) to Azure Blob container
Command: .\AzCopy.exe /Source:C:\Users\manju\Desktop\CopyData /Dest:https://manjugtestdisks.blob.core.windows.net/test-azcopy /DestKey:<insert_storage_key> /S
(Note: I have removed my storage access key – for obvious reasons). “/S” parameter will recursively copy the folder and its sub folders.
Now go back to the blob container to verify if the folder/files are copied. You may use any tool you want to do this. (Azure Portal, Azure Storage Explorer etc.,)
We can upload single blob to Azure Blob. Below is the syntax:
Upload a single blob to a virtual directory
Syntax:
AzCopy /Source:C:\myfolder /Dest:https://myaccount.blob.core.windows.net/mycontainer/vd /DestKey:key /Pattern:abc.txt
Upload a single blob
Syntax:
AzCopy /Source:C:\myfolder /Dest:https://myaccount.blob.core.windows.net/mycontainer /DestKey:key /Pattern:”abc.txt”
Download all blobs from a container
Below is the syntax:
AzCopy /Source:https://myaccount.blob.core.windows.net/mycontainer /Dest:C:\myfolder /SourceKey:key /S
Example:
Earlier we had uploaded files from a local folder – “CopyData” to Azure Blob container “test-azcopy”. Now we shall download the same three files to a local folder called “DownloadData”.
Command:
.\AzCopy.exe /Source:https://manjugtestdisks.blob.core.windows.net/test-azcopy /Dest:C:\Users\manju\Desktop\DownloadData /SourceKey:<insert_storage_access_key> /S
Similarly, we can download single blobs, below is the syntax
Download a single blob
Syntax:
AzCopy /Source:https://myaccount.blob.core.windows.net/mycontainer /Dest:C:\myfolder /SourceKey:key /Pattern:”abc.txt”
Download a single blob from the secondary region
Syntax:
AzCopy /Source:https://myaccount-secondary.blob.core.windows.net/mynewcontainer /Dest:C:\myfolder /SourceKey:key /Pattern:abc.txt
Download blobs with a specific prefix
Syntax:
AzCopy /Source:https://myaccount.blob.core.windows.net/mycontainer /Dest:C:\myfolder /SourceKey:key /Pattern:a /S
Copy a single blob from one container to another within the same storage account
Syntax:
AzCopy /Source:https://myaccount.blob.core.windows.net/mycontainer1 /Dest:https://myaccount.blob.core.windows.net/mycontainer2 /SourceKey:key /DestKey:key /Pattern:abc.txt
Copy a single blob from one storage account to another
Syntax:
AzCopy /Source:https://sourceaccount.blob.core.windows.net/mycontainer1 /Dest:https://destaccount.blob.core.windows.net/mycontainer2 /SourceKey:key1 /DestKey:key2 /Pattern:abc.txt
Copy a single blob from the secondary region to the primary region
Syntax:
AzCopy /Source:https://myaccount1-secondary.blob.core.windows.net/mynewcontainer1 /Dest:https://myaccount2.blob.core.windows.net/mynewcontainer2 /SourceKey:key1 /DestKey:key2 /Pattern:abc.txt
Copy a single blob and its snapshots from one storage account to another
Syntax:
AzCopy /Source:https://sourceaccount.blob.core.windows.net/mycontainer1 /Dest:https://destaccount.blob.core.windows.net/mycontainer2 /SourceKey:key1 /DestKey:key2 /Pattern:abc.txt /Snapshot
Copy all blobs in a container to another storage account
Syntax:
AzCopy /Source:https://sourceaccount.blob.core.windows.net/mycontainer1 /Dest:https://destaccount.blob.core.windows.net/mycontainer2 /SourceKey:key1 /DestKey:key2 /S
Download a single file
Syntax:
AzCopy /Source:https://myaccount.file.core.windows.net/myfileshare/myfolder1/ /Dest:C:\myfolder /SourceKey:key /Pattern:abc.txt
Download all files in a directory
Syntax:
AzCopy /Source:https://myaccount.file.core.windows.net/myfileshare/ /Dest:C:\myfolder /SourceKey:key /S
Upload a single file
Syntax:
AzCopy /Source:C:\myfolder /Dest:https://myaccount.file.core.windows.net/myfileshare/ /DestKey:key /Pattern:abc.txt
Upload all files in a folder
Syntax:
AzCopy /Source:C:\myfolder /Dest:https://myaccount.file.core.windows.net/myfileshare/ /DestKey:key /S
Upload files matching a specific pattern
Syntax:
AzCopy /Source:C:\myfolder /Dest:https://myaccount.file.core.windows.net/myfileshare/ /DestKey:key /Pattern:ab* /S
Copy from one file share to another
Syntax:
AzCopy /Source:https://myaccount1.file.core.windows.net/myfileshare1/ /Dest:https://myaccount2.file.core.windows.net/myfileshare2/ /SourceKey:key1 /DestKey:key2 /S
Copy from an Azure File share to Blob storage
Syntax:
AzCopy /Source:https://myaccount1.file.core.windows.net/myfileshare/ /Dest:https://myaccount2.blob.core.windows.net/mycontainer/ /SourceKey:key1 /DestKey:key2 /S
Copy a blob from Blob storage to an Azure File share
Syntax:
AzCopy /Source:https://myaccount1.blob.core.windows.net/mycontainer/ /Dest:https://myaccount2.file.core.windows.net/myfileshare/ /SourceKey:key1 /DestKey:key2 /S
Synchronously copy files
Syntax:
AzCopy /Source:https://myaccount1.file.core.windows.net/myfileshare1/ /Dest:https://myaccount2.file.core.windows.net/myfileshare2/ /SourceKey:key1 /DestKey:key2 /S /SyncCopy
Export a table
Syntax:
AzCopy /Source:https://myaccount.table.core.windows.net/myTable/ /Dest:C:\myfolder\ /SourceKey:key
Split an export from Table storage into multiple files
Syntax:
AzCopy /Source:https://myaccount.table.core.windows.net/mytable/ /Dest:C:\myfolder /SourceKey:key /S /SplitSize:100
Import a table
Syntax:
AzCopy /Source:C:\myfolder\ /Dest:https://myaccount.table.core.windows.net/mytable1/ /DestKey:key /Manifest:”myaccount_mytable_20140103T112020.manifest” /EntityOperation:InsertOrReplace
Manjunath Rao 