Los Angeles goes to cloud computing with Google

It is somewhat appropriate that the city of Angels makes this move to get into cloud computing.

What is even more ironic is that they are doing it with Microsoft’s money.

http://www.cio.com.au/article/324089/google_apps_scores_la_assist_from_microsoft

“Google has pushed Google Apps as an option for government agencies, promising to ship a product called Government Cloud, which will be certified under the Federal Information Security Management Act (FISMA), sometime next year”

"According to a Sept. 15 memo from the Los Angeles Information Technology Agency, Google will "provide a new separate data environment called 'GovCloud.' The GovCloud will store both applications and data in a completely segregated environment that will only be used by public agencies.""

This is a big win for cloud computing on a few fronts as it continues to be seen as a way to save money while keeping (and at times enhancing) the confidentiality, integrity and availability of information systems.

/*
Joe Stein
http://www.linkedin.com/in/charmalloc
*/

Seeing through Windows into the Cloud at the Eclipse

Microsoft(r) has announced http://blogs.msdn.com/interoperability/archive/2009/10/28/tasktop-soyatec-microsoft-to-foster-eclipse-and-microsoft-platform-interoperability.aspx collaboration for interoperability between Eclipse (my favorite Java IDE) and Microsoft Windows(r).

There are a couple of great highlights here and some fluff.

First the fluff (nothing wrong with looking nice while out on the town). Eclipse is going to be made useful for "next generation" user experience development for Windows 7 features.

Now on to the more exciting juicy pieces.

Microsoft has collaborated with Soyatec, a France-based IT solutions provider, to develop three solutions:

• Windows Azure Tools for Eclipse for PHP developers: www.windowsazure4e.org
• Windows Azure Software Developer Kit (SDK) for Java: www.windowsazure4j.org
• Eclipse Tools for Silverlight: www.eclipse4SL.org

These will open up the Azure cloud solution to not be 100% Microsoft based as well as give Microsoft a new following for it's Silverlight client framework in a community often with Sun in their eyes. More than anything this will open up the storage arena for MS to play a part.

Along with the SDK there is a Storage Explorer of Windows Azure Tools for Eclipse—it allows developers to browse data contained in the Windows Azure storage component, including blobs, tables, and queues. Storage Explorer was developed in Java (like any Eclipse extension), and they realized during the Windows Azure Tools for Eclipse development with Soyatec that abstracting the RESTful communication aspect between the Storage Explorer user interface and the Azure storage component made a lot of sense. So this led them to package the Windows Azure SDK for Java developers as open source, which is available at www.windowsazure4j.org.

Their interoperability strategy and open source direction is becoming competitive.

/*
Joe Stein
http://www.linkedin.com/in/charmalloc
*/

Mobile Internet Outpaces Desktop Internet Adoption

Mobile internet is taking off faster than the desktop.

iPhone + iTouch users = 8X AOL Users 8 Quarters after launch.



Mary Meeker's Awesome Internet Presentation From Web 2.0 http://www.businessinsider.com/henry-blodget-mary-meekers-internet-presentation-from-web-20-2009-10#mobile-adoption-curve-far-steeper-than-desktop-5 (Morgan Stanley). Click Here for the entire presentation.

/*
Joe Stein
http://www.linkedin.com/in/charmalloc
*/

Cloud computing is not about providing a software architecture for scale... that is what Open Source does.

Recently I heard the comment "WOW, elastic cloud computing is great. I can take on a lot more stress with any load and in a few minutes stand up an instance to accommodate usage on demand and keep the app running without long term cost or even contractual commitments". While this person is right they did not know that by starting another instance you are likely just turning on another problem if the software application was never designed to be distributed.

Cloud computing (and the "elasticity" it can provide with Infrastructure as a Service IaaS) is not about providing a software architecture for scale. Let me repeat this again, cloud computing is not about providing a software architecture for scale. So what is it then you ask?

Cloud computing provides an on-demand infrastructure so that your well designed distributed enterprise software application can quickly scale based on the spikes and valleys of usage and interactions of your system (pay as you go for only what you need). IaaS is about hardware resourcing given to your software to reduce expenses but if your software is not designed to take advantage then the opposite will happen with CPU & Memory running away with a false sense of security.

The issue is often that the internal workings of a software system are designed (to coin the phrase) "cloud monolithic". This means that software is usually designed to execute on a single server with a database (often a cluster) and to scale it you just add more servers and put the clusters together. Over the last 3-5 years many *VERY* large cloud based services have emerged and they have open sourced the solutions for how they scaled.

It is important to understand the inner workings of:

1) a-synchronous processing
2) global caching
3) distributed and parallel processing

Without all three of these patterns working together you will actually compound your stress with load bottleneck for each blocking call inside of your software. Your safety net of cloud computing turns into the proverbial wet blanket faster than it ever did before.

Lets break each of these patterns out and how they apply and what solutions exists. In another post I will explain how these apply when dealing with a ridicules amount of information processing on a large scale with the time of process exponentially reduced (because of using the algorithms for map/reduce). I bring this up now because the way the map/reduce algorithms achieve this ability to handle and process so much information exponentially faster is realized from the software written to implement them which also use the technologies that are explained here.

1) A-Synchronous Processing - Ok well this is not really a new one and often there are too many solutions to choose from all with their own pros & cons ( you have to make this call yourself ). Queuing systems have been around for a long time and have numerous implementations in the marketplace. It is so numerous that often each language has it's own set of queue servers to choose from. This being said they are often NOT used correctly because #2 & #3 are not implemented also. I have seen many systems make use of an a-synchronous process which allows the bottleneck of a blocking synchronous call to more expediently return creating a perceived performance gain. The problem here is that you are just passing the problem for another process to either eat up unnecessary cycles or not utilize unused cycles on other parts of your infrastructure (ultimately requiring you to get more servers or now turn on more instances).

Creating a performant software application is about taking both synchronous and a-synchronous operations and making sure that they are utilizing information that has already "crunched" by other parts of the infrastructure [#2 global caching] and maximizes the hardware so the crunching happens on the parts of the infrastructure that currently has the least "crunching" occurring [#3 parallel distributed processing].

So now maybe you are getting the problem and solution so here is how to implement it.

Global Caching with Memcached http://www.danga.com/memcached/. "memcached is a high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.".

What this means is two fold. Instead of your application querying the database for information it firsts checks the cache to see if that information is available. What makes this more powerful than using some static variable or custom solution is that memcached is a server that runs on EVERY machine that an instance of your application is running on. So, if server 3 pulls information from the database and adds it to the cache it is a "global cache" that the memcached server replicates for ALL instances/servers to make use of. This is extremely powerful because now every instance of your application is benefiting when all parts of the infrastructure are being used. In this scenario now you have un-compartmentalized "crunching" to no longer have to repeat some "crunch" of information to get to a result that another instance/server has already gotten to for their request/response.

Now this is a HUGE reduction to what often stresses a system but in of itself will not reduce the process to the degree that we are trying to get to because "at the end of the day" that "crunching" still has to occur. The crunching in the memcached implementation will still happen (hopefully a-synchronously once you find that it is not in your global cache and you have to "crunch" =8^0 ).

Now you need to crunch because your data is not memcached or perhaps you have to crunch for some other reason (that is what software does, right?) Just moving this to happen in the background off and onto another process provides no benefit within a multi-server environment.

A la "distributing the processing" and "executing it in parallel" which is where Gearman comes in http://gearman.org/. "Gearman provides a generic application framework to farm out work to other machines or processes that are better suited to do the work. It allows you to do work in parallel, to load balance processing, and to call functions between languages. It can be used in a variety of applications, from high-availability web sites to the transport of database replication events. In other words, it is the nervous system for how distributed processing communicates."

Both memcached and Gearman are servers with a great following with multi-language client implementation support. They are written in C so that they will execute better than if the had to deal with an interpreter or virtual machine. They might be over kill but if you find yourself with bottlenecks I hope you think about your design and internal architecture of the system before you throw more hardware at your problem (especially now that this can be done with a few clicks to launch an instance).

/*
Joe Stein
http://www.linkedin.com/in/charmalloc
*/

Ruby on Rails with OAuth for integrating TripIt

Here is an example of utilizing OAuth with TripIt using Ruby on Rails.

OAuth is an open protocol to allow secure API authorization in a standard method for desktop, mobile and web applications. http://en.wikipedia.org/wiki/OAuth

TripIt is an interesting social networking application for your travel itinerary http://www.tripit.com/

All the examples below are to be run in "irb" but work fine in your rails app. You need to figure where to store the variables I pass in (this is up to you and how your app is setup of course).

First things first... you need a TripIt developer account http://www.tripit.com/developer. Make sure you add an application (call it what you like but you need to submit it so that you get an "API Key" and an "API Secret".

For our example (since I do not want to give you mine NOR should you give out yours to others) I will use "api_key_shhhh" and "api_secret_shhhh" as the values that you will getting from TripIt.

Now before we get started make sure you go into your regular user account on TriptIt and a add your self a trip (or more). This example will list trips so you need them to see the XML we are going to query through the API.

lets get "oauth" installed now

gem install oauth

Ok, now to the code (all of this is for irb but you can have it work in your rails app, no probelm).

gem 'oauth'
require 'oauth/consumer'

api_key = "api_key_shhhh"
api_secret = "api_secret_shhhh"

@consumer=OAuth::Consumer.new api_key,api_secret,{:site=>"https://api.tripit.com"}

@request_token=@consumer.get_request_token
@request_token.secret

#ok now in your rails app you want to redirect and create (dynamically) the URL we are creating by hand which we will copy and paste it.

OK NOW THIS IS IMPORTANT. Do not use @request_token.authorize_url because the URL is wrong. In your rails app you should dynamically create what we are about to-do by hand (concatenate yourself silly). ALSO, The URL that you put into your setting when creating the application... if it was localhost (or blank) this will not work but have no fear there is simple workaround by overriding in the URL.

There are a few important parts of the URL and you need to take the @request_token.secret value which for this example let me call it XXXXXXXXXXXXXXXX

Put this in your browser now https://www.tripit.com/oauth/authorize?oauth_token=XXXXXXXXXXXXXXXX&consumer_key=api_key_shhhh&oauth_callback=http%3A%2F%2Fwww.yahoo.com

TripIt will now ask the user if it is alright for the application (in the real rails app redirect_to the concatenated URL you made) you just created to access their account (in our example this should be your account you are granting your application access to). Now notice that oauth_callback. In "real world" rails app that should be YOUR application to accept the user back in to-do stuff which TripIt will redirect to when done. All of the URL has to be URL encoded and "consumer_key" is that first value you get from TripIt (NOT THE SECRET) when you submit your application.

Ok, now you are just about done.

TripIt trusts you and now you just have to save that trust to use later in your app.

To save that trust (back to your terminal irb picking up where we left off)

@access_token=@request_token.get_access_token
@access_token.token
@access_token.secret

NOW SAVE the token and secrete from the access token (where ever you like for THIS user).

Ok, last step now that the user has authorized now you want to keep using that authorization from that users to-do TripIt actions (they would get annoyed if you had to keep asking them because you skipped this step) for that user through the API.

copy both access_token.token and @access_token.secret you are about to need them

exit irb

now go back to irb so you can see it all still working fine.

gem 'oauth'
require 'oauth/consumer'

API_KEY = "api_key_shhhh"
API_SECRET = "api_secret_shhhh"

ACCESS_TOKEN = "@access_token.token"
ACCESS_SECRET = "@access_token.secret"

@consumer=OAuth::Consumer.new API_KEY,API_SECRET,{:site=>"https://api.tripit.com"} #, :consumer_key=>"consumer key working"}

@access_token = OAuth::AccessToken.new(@consumer, ACCESS_TOKEN, ACCESS_SECRET)
puts @access_token.get('/v1/list/trip')

and here you go now with your XML back from TripIt per their spec http://groups.google.com/group/api_tripit/web/tripit-api-documentation---v1

Make sure "@access_token.token" and "@access_token.secret" are the values you saw in irb and copied before you closed it as that variable is GONE.

/*
Joe Stein
http://www.linkedin.com/in/charmalloc
*/

Cyber Security in Government, ONLY JUST NOW?

So it looks like the Department of Homeland Security will be moving to create a more secure infrastructure (or something) to our government facilities that use "computers" by starting to hire cyber security analysts http://www.cnn.com/2009/POLITICS/10/02/dhs.cybersecurity.jobs/index.html?iref=newssearch.

What concerns me most about this is that 1,000 people seem to be a mad rush now to have something in place which I feel should have been there all the time. Is the NSA not cutting it or working nicely with the DHS? What role is the DHS looking to play now in an industry full of consultants and information technology folk within organizations fighting the good fight.

Let me be the first to label these folks as "Blue Hat" (since Red & White are taken and Blue seems to make sense here... )



/*
Joe Stein
http://www.linkedin.com/in/charmalloc
*/