Oracle’s Jump Into the Big Data Realm

Posted by Topper on 10/3/2011
Posted in: General, Hadoop, Oracle. Leave a Comment

Many of you may have seen that Oracle officially announced their new Big Data product offerings today. Included in that list is the Apache distribution of Hadoop, Oracle Loader for Hadoop, Oracle Data Integrator Adapter for Hadoop, the Oracle NoSQL Database, and Oracle’s R Enterprise. On the Oracle big data pages it seems like there is some confusion as to whether R will run a custom Oracle version or an Open Source version. Knowing Oracle it may be both. They have been working on a version of R within the Oracle database and might be augmenting it with an open source version on the Hadoop appliance. Lets cover what is known so far about the different components.

Apache Distribution of Hadoop
I’m actually really surprised here.  Larry Ellison has acquired a Mike Olson company previously (Sleepy Cat) and Mike now runs Cloudera.  I would have bet good money that a partnership would have been formed.  My guess is that Oracle looked at the management tools Cloudera provides and determined that it would be too hard to integrate into their Enterprise Manager product or that Cloudera’s price was too high.  They are leading the Hadoop support market right now and have a really good future.  Hadoop comes with a very large open source ecosystem around it. This solution should be great for both Hadoop and Oracle.

Oracle Loader for Hadoop
From what I have heard this is a map reduce job that will format the resulting data set file into an Oracle Data Pump file to be loaded directly into the database.  There has been the tools SQOOP to suck data out of the database, but now we have the other side of the coin.  Hadoop is great, but joins are still a bit problematic and the BI tools around it don’t match what can be done in the traditional database world.  The loader should help companies figure out what the balance should be.

Oracle Data Integrator Adapter for Hadoop
The Data Integrator Adapter is similar to the Oracle Loader for Hadoop, but it extends Oracle’s Data Integrator product to be able to execute and manage Hadoop jobs as part of an ETL process. It is well known that Hadoop can crunch and count numbers faster than the Oracle Database in many cases.  This allows the ETL process to offload the heavy number crunching and then use the Loader to put the data into the Oracle database when complete.

Oracle NoSQL Database
For a long time it has seemed like Oracle was neglecting the Berkeley DB product and not making giant leaps forward.  Berkeley has always been a fantastic product for key value stores.  In fact many of the major key value stores today are underpinned with Berkeley.  It looks like Oracle has updated the product to bring many of the missing distributed features into the new product.  It will be interesting to see how the new Times Ten Database, NoSQL Database, Hadoop, and ExaData components work together in the Oracle BI tools over time.

Oracle R Enterprise
R has long been the language of choice for the statistics community.  It isn’t clear if Oracle will be using the open source R-project.org distribution or has released their own.  My guess is both.  R-project will be deployed with the Big Data appliance and Oracle’s R within the database.  This should make a bunch of the big data number crunchers from the SAS world happy.

I have been told that Larry and TK have given the green light to go full force into NoSQL.  If that isn’t justification that it’s a “real thing” that is here to stay I don’t know what is.  Oracle has and will continue to invest significant resources into Big Data.

This makes the team here at UberEther happy to see the largest software vendor come to the table and support the Hadoop community.  It relieves some of our worries in developing out new log aggregation and risk adjustable access control product to the market. We know our product will be able to run on predefined hardware platforms for our largest customers and we can easily load the data into their legacy tools to reuse their existing investments.  We still have a lot of work to do but if you’re interested in hearing some more while we’re out here at Open World contact us and we’d be more than happy to show you what we’ve been working on.

Taking a Techcation, Want Some Ideas

Posted by Topper on 3/2/2011
Posted in: General. Tagged: , , . 2 comments

So Jake over at the App Lab has been known to take many a staycation in his time.  I’m notorious for never taking any of my vacation (been at my 180 hour cap for close to a year). So I’m going to start a new trend taking a “techcation.”  My goal is to take a week away from the office to play with a different technology every day and blogging my progress at the end of each day.  My target week is that of March 13, so I have 2 weeks to prepare.  There’s been a bunch of different things on my desk and in my mind that I’ve been meaning to find time to play with, but as with everyone, works been taking all my time lately.  So I’m going to stay home and play with something new every day.  So far here is my list of things I’m thinking about:

  1. Arduino something (Thanks Chris)
  2. Hadoop 101
  3. Cassandra 101
  4. MongoDB 101
  5. Android 101

This is a very tentative list.  Anything else people think I should be looking at?  I’m looking for something that I can get a basic knowledge and have a working “something” up and running within 8 hours.  Feel free to hit the comments with suggestions.  As much as I love the identity world I’m hoping to stay far far away for the week.  I’d love to see this idea take off, even better would be if people started pairing up or doing it in small groups.

I might drop one of the days to work on the 280Z or my truck if its nice.  I’ve had two pairs of speakers sitting on the floor of my office to go into each vehicle for about 6 months now.  I’ve also got a bedslide for the truck thats been taking up garage space for over a year.  I’m sure my wife would appreciate that one.

The Next OpenID Platform

Posted by Topper on 2/14/2011
Posted in: Identity Management, Personal, Technology. Tagged: , , . 4 comments

We’re repeatedly seeing more and more accounts of accounts being hacked and passwords being stolen across the web.  Take a look at the recent Gawker hack and all their hashed passwords being posted to torrent sites.  People should also realize that there are many lists out there that take millions of common passwords and run them through all the standard hashing algorithms.  This allows a hacker to quickly search through a list of hashed passwords and come up with the actual password you’re using.  How many of us use this same password across multiple sites, or a very similar variant of the same password.

A few years ago I envisioned what the next generation OpenID platform might look like.  I’ve shared this with multiple people over the years and they all ask me why I haven’t built it yet.  I honestly just haven’t made the time so I feel like its time to give someone else the opportunity.  OpenID is great in that it allows you tohave a single strong password to authenticate you against multiple sites.  Many of you will have used Facebook Connect to go to websites and this is a very similar technology  Some of the things I noodle in the attached diagram TheNextGenOpenIDPlatform include stronger hardware based authentication devices, delegated account access, digital personas, linking of devices that can be remotely de-authorized, an analytics dashboard to find out who is using your identity and attributes, and synchronizing your attributes across all the places you go.

If there are any VCs listening I’m more than happy to start an angel round of investing and start building, I already have the team I’d use ready to go. 😉

Oracle IRM and LDAP Accounts

Posted by Topper on 2/14/2011
Posted in: General. 3 comments

One of the great features of Oracle IRM 11g is being able to automatically link your users from LDAP.  This way you don’t have to manage the user’s in two places or write any custom synchronization code between them.  The LDAP integration is done through the providers WebLogic.  A word of warning about this integration is that you need to have your LDAP provider setup in WebLogic before logging into IRM for the first time.  You also need to have your LDAP provider as the first item in the list before logging in.

My reason for typing this out is because it burned me for a few hours trying to figure out what was going on.  The first user that logs into IRM is set to be the administrator.  This person creates the contexts, roles, etc. and assigns all the privileges.  For most implementations people normally use the built in weblogic user that is created during installation.  This is where I went terribly wrong. IRM binds the GUID of this user to the IRM database repository.  This is obviously much stronger than binding just the username or the DN of the user but also can cause crinkled skulls when trying to debug.

So, I logged into the server as weblogic, got the tabs and pages I expected so I figured I would setup my LDAP provider. I went into WebLogic and created the provider.  In a development environment I normally set the internal provider first and then the LDAP provider second.  Even though I take a hit in performance, in a development environment I prevent myself from being locked out of the server. I was now able to authenticate but IRM wasn’t letting me into the interface.  After talking to the dev team they told me the LDAP provider had to be the first one in the list as thats all they look at when authenticating the user.  No problem pop it up to the top of the list.

Now I can authenticate into IRM and get into the irm_rights pages, but I don’t have any of the other tabs to manage the server.  The GUID that my weblogic user is tied to is now second on the providers list and the GUID of the weblogic user in my LDAP server doesn’t match the local GUID. Shit, so now my administrator user can’t be reached because he’s second in the provider list and I can’t set any other administrators because if I move that provider back to the top of the list the LDAP users don’t appear since IRM only looks at the first one.

Lessons learned and a reinstall to fix.

Installing Firefox 4 Beta on OSX

Posted by Topper on 2/9/2011
Posted in: General. 22 comments

So today I decided to switch over to the mainline Firefox 4.0 Beta from the Minefield dailies I’ve been using.  I started to copy the new Firefox.app directory over the existing one in my Applications folder and immediately was met with a:
‘The operation can’t be completed because the item “libsmime3.dylib” is in use.’
Well shit, whats holding onto that lib.  Turns out that it was my Cisco VPN Anywhere Agent, aka vpnagentd.  So I go ahead and do a:
‘sudo killall vpnagentd’
Ugh and of course it restarts automatically before I can copy the files over.  So what now.  Oh yeah, it runs as a daemon so I need to use my old friend launchctl to unload it.  The command for this is:
sudo launchctl unload /Library/LaunchDaemons/com.cisco.anyconnect.vpnagentd.plist
w00t! Now the files copy over with no problem.  Now I need to put AnyConnect back into place.  This can be done with:
sudo launchctl load /Library/LaunchDaemons/com.cisco.anyconnect.vpnagentd.plist
That should do it, Firefox 4 Beta up online and working again.

That was 15 minutes of my life I’ll never get back. Hopefully this post saves you 10.

Twitter Updates for the Week of 2011-01-09

Posted by Topper on 1/9/2011
Posted in: Personal. Leave a Comment
  • @squarism Thought you'd enjoy this list, enjoy the first day at your new gig http://is.gd/k1CZH #
  • I was getting so much more done before I realized that Thunderbird had crashed on me #
  • I would bet that Goldman made the investment in Facebook to get unfiltered access to their data to use in their trading platform. #
  • Effin heat is out at home #
  • @squarism just explained yak shaving to someone, you're already becoming a legend #
  • Anyone know a SharePoint expert I could ping some ideas off of? #
  • @cyberslate Trying to figure out how the LDAPProvider works and we're just getting everyone denied in reply to cyberslate #
  • @JimmySky Thx man. in reply to JimmySky #
  • A week of 4-5 hour of sleep a night and I've never felt better #
  • @oraclenerd so now you're calling me fat? cc/ @dannorris @jkuramot @ktjustice in reply to oraclenerd #
  • @jkuramot http://t.co/zQTHWV3 cc/ @dannorris in reply to jkuramot #
  • Dear Angry Birds, I have your game on at least three different devices. It would be nice if you could sync my progress between them #
  • Anyone have any suggestions on what I should watch/play with on my 6 hour flight to SFO this week? #
  • Thinking I'm going to setup an access point in my house for myself, wife and mother in law are hogging all the good bandwidth #yesiamselfish #

Twitter Updates for the Week of 2011-01-02

Posted by Topper on 1/2/2011
Posted in: Personal. Leave a Comment

Twitter Updates for the Week of 2010-12-26

Posted by Topper on 12/26/2010
Posted in: Personal. Leave a Comment
  • I love listening to people who are confused because they don't understand encryption #
  • Any of my Oracle pals have any suggestions on how to setup security groups on Amazon EC2 for the Oracle DB? #
  • @brad_tumy @mathewbutler I've figured out my issue, forgot I had iptables enabled in reply to brad_tumy #
  • @mathewbutler give me a couple weeks, setting up a big demo right now and this is my first time playing with it in reply to mathewbutler #
  • RT @dogfishbeer: Thanks for the love! RT @jonfitzsimon: Please RT for a second season of Brew Masters! @discovery @dogfishbeer #BrewMasters #
  • @jkuramot What type of man makes a snow angel barefoot in a tshirt? Sadly you looked sober doing it too. in reply to jkuramot #
  • Dear UPS, its 4:14PM and my package is still in Louisville, get that flight off the ground #
  • Hey UPS, glad I paid extra to get this gift by Christmas and now you rescheduled it to be delievered on the 27th #

Twitter Updates for the Week of 2010-12-19

Posted by Topper on 12/19/2010
Posted in: Personal. Leave a Comment
  • @squarism I remember you having this idea about a year ago http://is.gd/iE3Qs #
  • @tripit I'm happy you can link my account to gmail, but blocking my trip access until I do so sucks, largely because your side is broken #
  • Btw @tripit signing in and out removed the nag screen #
  • Reviewing my 23andme results, definitely interesting, more interested in seeing how my daughters come in #
  • After 3 years of fighting traveling, I think I've lost. St Louis this week, redwood for one week and London for 2 in January. #fb #
  • Doing some really cool shit right now and collaborating with a great group of guys, this is why I work where I do #

Twitter Updates for the Week of 2010-12-12

Posted by Topper on 12/12/2010
Posted in: Personal. 1 comment
  • What should have been a couple hour wiring project had taken me all day and it's still not done. Maybe tomorrow morning I'll get it back. #
  • How the hell does U of Michigan get a bowl game this year. #playingD3TeamsDoesntCount #fb #
  • @jkuramot Congrats man, just wait until she starts kicking you IRL in reply to jkuramot #
  • @ckras you must use the google platform? Or just for POC? in reply to ckras #
  • Everyone needs a Giraffe in their home http://www.dailydeals.net/ #fb #
  • @dannorris +1 empanada por favor #
  • For some reason it sounds like someone was mucking with the balance on Sublimes What I Got #
  • @Lanyrd, it's cool that you let me sign in with Twitter, but I don't see why I need to authorize you to update my stream. #
  • Wondering how an Oracle guy can justify to the powers that be to go to the Strata conference #
  • @Lanyrd I know you promise to not use it for evil, but I don't know you yet so I can't trust you #
  • @simonw Thanks Simon, I'll be hanging in the shadows until the update, looking for to it, looks like a great service in reply to simonw #
  • Trying to figure out how to secure a federated comment system ala Discus but privately stored data at corporate endpoints #
  • @astaley The more pressing question is what is more important, the skills or the tickets? Sadly I think its the tickets lately in reply to astaley #
  • Traveling too much over the next few months, just put TripIt back on my phone #