Hi Hadoop!

Heard Hadoop for many times, never got deep into it. Now is a chance, so I started the experiment in Microsoft Azure.

First, you can find a number of Hadoop packages in Azure Marketplace. I chose Hortonworks Sandbox with HDP 2.5.  I tried Hadoop by Bitnami as well, but it’s usability is a bit tricky, I couldn’t find a way to make Bitnami work without creating a number of accounts and expose more of my own information. I may try it later (and enable the boot diagnostic to find the password in the log when the image starts the first time) when I have time. For now, I stick to Hortonworks.

image

Then just follow the standard Azure procedure –

  • Basics: filling in the VM name, username, SSH key or password, subscription, resource group, location, etc.
  • Size: choose the size of the VM.
  • Settings: choose the storage, network, etc. I suggest to leave boot diagnostics enabled.
  • Confirm on the Summary and Buy.

Notice that on the price page, there is warning on the charge besides the Azure VM itself, also since the HDP Sandbox just showed 0.0000 CAD/hr, I don’t think you need to worry too much about it. BTW, Bitnami’s Hadoop is also free, explicitely mentioned.

image

Wait for a few minutes until the deployments succeeds. You can then check the status of your new Hadoop VM.  Hortonworks suggests that you make the public IP static. You can find more detail information on its tutorial page.

Next is to configure your SSH client. I am using PuTTY on Windows, so there are more mouse-clicks than the config example given in the tutorial.  Basically these settings let you connect to your VM in the Azure cloud using various ports from localhost to the remote VM via the SSH tunnel you set up here.

image

Here is how to configure PuTTY:

  • Fill in the public IP of your Hadoop VM
  • Expand Connection – SSH
  • Click Tunnels
  • Fill the source and destination, then click Add button

image

According to the document, you need to add 8 forward ports

image

So in PuTTY, you can add one by one, it should eventually look like this (scroll up and down to see total 8 lines/ports).

image

You can then go back to the Session page, give a name in “Saved Sessions” and save the configuration. Next time, you only need to load it from there.

One trick is that the VM need some time to start and become stable. My first few login attempts failed, only after 20 or 30 minutes can I eventually login. so be patient. After login, you should be able to see the following directories.

image

Then according to the tutorial, keep the SSH session active, you can use brower to visit this page on your VM.

image

Click on the left icon, you will see the dashboard.

image

Click on the right, you can read more advanced topics including the default username and password, and how to change them.

That is the first step into Hadoop.

Advertisements

PowerShell for Azure Active Directory (AAD) and Office 365

Well, this is just some notes on how to prepare PowerShell to manage Azure Active Directory and Office 365. But similar to connecting to AD in Azure, you also need to go through these steps to connect PowerShell to an Azure subscription.

This is quite interesting actually when I put them together. As AAD still has two active environment versions, same as Azure console – Classic and RM – they belong to different logins, a little confusing to admins. Also PowerShell modules need to be installed and updated to enable different cmdlets set in order to manage different products – cloud, non-cloud, 3rd-parties like AWS, etc. So when something is not working, maybe you are in a wrong dimension or Microsoft wants you to update the binary you are using.

Azure Subscription AAD Module v1 (MSOnline)
– General Availability version
– Public Preview version
AzureAD v2
– General Availability version
– Public Preview version
Pre-requisites:
– Install lastest Azure Tools using Web Platform Installer
> Add-AzureAccount
> Get-AzurePublishSettingsFile
image
> Import-AzurePublishSettingsFile
> Select-AzureSubscription
Pre-requisites:
MS Online Services Sign-in Assistant
Windows AAD Module for Windows PowerShell
Pre-requisites:
PowerShellGet
image
Step 1: Install Azure PowerShell
Install-Module AzureRM
Download and install Azure AD Connect (.msi)
Update AAD Module > 1.0.8070.2
Download from PS Gallery
Install AAD Module
Update AAD Module
Step 2: Connect to an Azure account
Login-AzureRmAccount
Connect to Azure AD

$Msolcred = Get-credential
Connect-MsolService -Credential $MsolCred

Connect to AAD

$AzureAdCred = Get-Credential
Connect-AzureAD -Credential $AzureAdCred

Step 3: Run Azure PowerShell cmdlets

v1 cmdlets are here v2 are here
More information: PowerShell Gallery Office 365 PowerShell

One more thing: remember to check the update time of anything posted online, those older than 3 months might be useless.

Web Role v.s. Worker Role

Azure Cloud Service Role is classic, old style, one of those you won’t see updated counterpart in new Azure Resource Manager (ARM).  In Azure, a Cloud Service Role is a collection of managed, load-balanced, Platform-as-a-Service (PaaS) virtual machines that work together to perform common tasks. Cloud Service Roles are managed by Azure fabric controller and provide the ultimate combination of scalability, control, and customization.

Web Role and Worker Role are under Cloud services (classic).

cs_diagram

Web Role is a Cloud Service role in Azure that is configured and customized to run web applications developed on programming languages / technologies that are supported by Internet Information Services (IIS), such as ASP.NET, PHP, Windows Communication Foundation and Fast CGI.

Worker Role is any role in Azure that runs applications and services level tasks, which generally do not require IIS. In Worker Roles, IIS is not installed by default. They are mainly used to perform supporting background processes along with Web Roles and do tasks such as automatically compressing uploaded images, run scripts when something changes in database, get new messages from queue and process and more.

Differences between Web and Worker Roles – The main difference between the two is that a Web Role support and runs Internet Information Services (IIS), while an instance of a Worker Role does not.

References:
https://docs.microsoft.com/en-us/azure/cloud-services/cloud-services-choose-me
http://cloudmonix.com/blog/what-is-web-and-worker-role-in-microsoft-azure/

Windows Azure Overview

  1. Cloud Computing
    Infrastructure/Platform/Software as a Service
    App-Data-Runtime-Middleware-O/S-Virtualization-Servers-Storage-Networking
    … a boring video about Microsoft moduler datacentre contrainer …
  2. Windows Azure Overview
  3. Windows Azure Components
    SQL Databases, Tables, Blobs.
  4. Operational aspects.
    1. Database High Availability and Disaster Recovery
    2. Skillsets required for the operations team
    3. Monitor and diagnostics
  5. Premier Support for Windows Azure
  6. Premier Services

Again, Microsoft Azure charged about $5 a month for an empty database.

Now I realized, I shouldn’t have touched this MS Azure cloud again when I received a newsletter from Windows Azure Team.  All I did one day last month after I saw that email was created an empty web site, a cloud service, an empty database and a storage.  Not a single byte has been put in so far because I still cannot figure out what would the next step be, even to say ‘Hello World”.

image

Last time I wrote about Azure I have the conclusion that a 200MB database in MS cloud could cost user more than $25,000 each month, so this time I haven’t put a single bit in db01 database I just created.

http://woodaway.wordpress.com/2011/02/17/how-much-does-it-cost-to-store-16kb-data-in-azure/

Now it’s happening again, when my credit card expired by the end of June, they sent me two email today (July 1st), reminding me I should update credit card info and also I owe them $0.20 for the first day of July, for nothing.

The Azure billing detail is more sophisticated than the hydro and gas bill:

  1. I don’t know why they charged me 0.016129 ConsumeUnits for an empty database, there is nothing in it, but at least it’s consistent everyday.
  2. The ConsumeUnits of the website are different every day.  Why?  I have nothing, not even one html file in the web site, what has been consumed differently everyday?

image

This is what Azure charged in June for the empty services.

image

image

And this is what MS cloud charged for July 1st, plus tax I assume.

image

This could be the reason – I enabled four services in June 8th although I didn’t have time to touch them again, but the SQL service charged me for an empty database, which might have consumed some kilobytes on their virtual hard disk somewhere in the cloud.

The unreasonable part is the SQL service was not even working when I logged in yesterday.

image

image
I don’t bother call Microsoft support for the error this time.  I called for some other Azure questions last time, it seemed they don’t care.  I am not sure if it’s because I told them I wanted to try (preview) Azure from SMB (small & medium businesses) standpoint or else.  Bottom line, it’s Microsoft product, if Microsoft people don’t care, why should I?

I just deleted everything  under the account, gonna pay a few loonie or toonie to the MS foundation since they claimed I owe them (maybe in fine print somewhere).

Bye Azure!

Were you on free flight OLSB? Now MSFT is kicking you out to the Cloud… Let us know when you touch the ground.

FACTS:

http://ask.officelive.com/smallbusiness/blogs/team/archive/2011/12/17/update-on-the-future-of-office-live-small-business.aspx#comments

Can my website be transitioned automatically to Office 365?

You must manually recreate your website in Office 365. To do this, you must copy and paste your text, pictures, and any other website elements you choose to move.

(http://www.microsoft.com/office/olsb/FAQ-EN-US.html)

Microsoft has been struggling with its online service since long time ago, maybe since it bought HoTMaiL 14 years ago.  Then it started MSN, including MSN Messenger and MSN Spaces, later rebranded to Windows Live, it also named its small business service to Office Live.  But in recent years, MS re-shuffled all its online service again.  Stopped Live Spaces last year, made Office Live and Live Office more confusing.  Pushing the MS cloud – Azure, including a lite consumer version called Skydrive.  Changing Office Live to Office 365,  et cetera.

But all these are going to fail.  MS had failed on Web 2.0, it has also failed on Social Networking (remember Wallop?).  Now they bet on the Cloud.  If there is no philosophical change in its online business, MS is going to fail again.  Let me tell you why?

(to be continued)