ODTUG Day 3-4

Posted by Topper on 7/4/2006
Posted in: Oracle. Leave a Comment

After a late night last night I was definitely dragging this morning getting to the conference, I came in about 10 and caught up on email until lunch time. I caught the BI / Data Warehousing Panel discussion, it was good with where people had been, but due to poor attendence there really weren’t any ground breaking questions. I was amazed to see how many Oracle BI employees were at the conference and had made it to this session, I swear half the room had come from Oracle. Its good to see them all coming out and supporting the BI development community at conferences like this.

I had to sneak out early to prepare for my “Quick Web Development with jDeveloper” presentation. The presentation went well, its actually a presentation put together by own of the owners (Brad Brown). I think it went over well, one thing I noticed at the conference were a lot of Reports and Forms developers looking for the next thing and direction on those tools. I think this presentation helped a lot of people to realize that jDeveloper is nothing to be afraid of and how easy of a tool it is to create basic forms and reports.

After my presentation I spent a good half an hour talking with people about jDev and where I though Oracle was going with it. Lastly, I went back and did booth duty until 5 when I met Dan Norris an ex-TUSCer and Shaun for dinner. It was good to catch up with Dan, he was one of the first people I went out on a project with when I joined TUSC. He’s always good for some database war stories and to bounce new ideas off of.

The last day of ODTUG started much like the rest of them, on the Metro at 8 and to the conference by 9. I caught up on some email and client work that I’d been putting off and then went off to Dan’s presentation on “Why you need a technical review board.” This was a presentation based on a client that Dan and I both worked on together and is really something that I believe should be part of every organization. No matter how formal or informal it is, a TRB should really help drive a company technically while controlling the environment from being completely out of control. I also served as Dan’s moderator for the talk and got a “nifty” ODTUG hat for it.

After Dan’s presentation I headed over to Andre Beland’s presentation on the “Oracle 10gR2 Rules Manager” based on the name I had hoped this was the new rules manager that links in with BPEL, but it was the database version of the rules engine. There was definitely some interesting content in there on creating and linking rules, but I think the most fun was watching Mark Rittman try and get his change data capture process up and running again before his presentation.

After a client call I caught Mark Rittman’s “Building an Effective Data Warehousing Architecture”, it was a good overview presentation on building an effective warehousing environment, but what I enjoyed the most was the change data capture and new Oracle Warehouse Builder profiling pieces. CDC is something that I’ve always built custom scripts or used MVs to manage (I could never get streams working flawlessly) but CDC seems to help solve a lot of those problems. With the 10.1.2 release of the DB a lot of the overhead from triggers on tables is replaced by reading the redo logs. The profiling piece is a nice add-on to the Warehouse Builder tool, I struggle with its $15K per CPU price tag, but it definitely helps automate some of the profiling I’ve traditionally done with scripts and SQL statements. But with that price tag I don’t think my scripts will be collecting dust anytime soon. I checked out of Marks talk early so that I could prepare for my “Changing the World with Oracle Web Services” presentation, of course it was right as Mark was getting to some of the really good stuff.

Being the last presentation time for the conference I thought the room would be empty, but it was anything but, the room was packed. I saw many of the same faces from my “Quick Web Development” presentation so it must not have gone that badly (honestly, I felt flat for that one for some reason). The presentation went very well and I received a lot of compliments on how I cleared up some of the haze for people my breaking it down into easy to understand parts. This was another presentation I did for Brad Brown, but modified and went off on lots of tangents of my own. I got a lot of ribbing for drinking a RedBull during the presentation, but I honestly think it helped me concentrate.

Well thats all for another ODTUG…sorry these got finished so late, I’ve been starting a new project with a client thats very aggressive and didn’t get a chance to finish these up. I’m hoping to have some more interactive articles up shortly and a big announcement on this site I’ve been working on.

ODTUG Day 2

Posted by Topper on 7/4/2006
Posted in: Oracle. Leave a Comment

I got into the conference early today, being a Monday I was banking on the Metro to be a little slow and it take over an hour to get in. Much to my surprise, they were right on time. Which was good because it gave me some time to go over my presentations and catch up on client stuff before the day began. As I was taking advantage of the free Wifi at the conference (I love that all of them have it now) I ran into Mark Rittman in person finally. I’ve traded emails and blog comments with Mark for a while now and it was good to finally meet him in person. We had nodded to each other in the BI Customer Advisory Council, but I never got a chance to go over and talk to him. I also ran into one of my TUSC colleagues Shaun O’Brien and had a good chat catching up with him on what he was doing at a client and trying to bring him back to work on a project with me.  As I sat with Shaun I also got a change to see Aydin Gencler again, one of the BI product managers who I haven’t seen since Oracle World in 04 and got to catch up with him and some of the things coming from the Oracle BI world.

After lunch, I went to Dan Vlamis’ talk on Real World BI Case Studies, I was hoping for some more indepth from the trenches type talks, but the rest of the room was looking for a much higher level. Dan being the great presenter he is catered to the majority and threw in some real world at the end. It gave me some good time to try and work out some of the details on a new site I’m hoping to have up soon and get some of the integration done.

After that I did some booth duty and talked to a lot of people working on some cool projects out there. Thats one of the things I like about ODTUG over the other conferences, it seems to be more of the geeks in the trenches that want to talk about what they are doing. Rather than CIOs and managers that roam the halls of Oracle World, I think we need some more of these type of conferences. I’m thinking about putting one together kindfa like O’Reilly’s Foocamp or Barcamp out in the Valley if anyone’s interested let me know, I’m definitely serious about doing this.

I ended the day with dinner and drinks with some of the other TUSC people out here this week, had some good conversations about where the company is going and how we are going to get there. I’m definitely happy to be with these guys and happy to be part of the team driving us to the next level.

ODTUG Day 1

Posted by Topper on 7/4/2006
Posted in: Oracle. Leave a Comment

My proof for calling this day 1 is that the wireless internet access for the conference actually went up and is working today. Today I spent in the BI Customer Advisory Council. Its an invite only event for people in the Oracle BI space. I got lucky enough to be added to the CAB when I was working at Polk and they were gracious enough to let me continue to be a member when I joined TUSC.

A key note that I got out the meeting was that BI is officially considered the 4th leg of Oracle with the Database, Applications and Middleware. With the purchase of the Siebel suite Oracle is definitely taking BI seriously and coming back with a vengence.

Some future / current features coming in the products
Discoverer workbooks are going to be a source for XML Publisher
XML publisher to get pixel perfect rendering, seems to be one of the next logical replacements for reports (if we didn’t see that already)
In the Siebel suite there are APIs to access metadata layer, control users and role membership. If I get a chance to get this up and running it would be great to integrate this with the Oracle Identity Manager (Xcellerate) product for provisioning.
Right now the Siebel product doesn’t do Oracle OLAP (or at least easily) but the new Answers Plus product will integrate this in.
There is the idea of a “Common Enterprise Information Model” within the new Oracle BI Suite that allows you to physically or virtually connect and expose data to any and all of the Oracle BI products. This is a great concept that very much brings a Hibernate environment to the Oracle BI stack. Now it won’t care what my data sources are, Oracle DB, Excel spreadsheets, web service data feeds, it won’t matter. Will this replace classic datawarehousing? I doubt it…at least for now with current technology.
There is the concept of an iBot, that triggers to run data at a scheduled time. Its much more flexible than the current scheduling built into Discoverer that we see today which would make customers happy. There are some cool features where based on results of the report or query you can send messages to the dashboard, send a page, or kickoff BPEL process.
Most importantly to me, the Linux release of the Oracle BI EE stack is coming around the September timeframe.

That was all, a very short day back on the Metro and in bed before everything starts tomorrow.

ODTUG Day -1

Posted by Topper on 7/4/2006
Posted in: General, Oracle. Leave a Comment

Well, I’m calling it day -1 because the festivitites that is ODTUG isn’t supposed to start until Sunday, but Paul Dorsey and Dulcian put together an Architects conference with a lot good people and good topics. I started out getting up a little late, its been one of those weeks from you know where and worked until 2am and left about 9:00. I love conferences in DC, it lets me jump on the metro 2 blocks from my house and not have to worry about traffic. So I get out of the metro and should have made a right instead of a left, ended up walking around for half an hour realizing I must of made a wrong turn. Called thehotel and realized if I had looked to the right when I left the metro I would have been staring at it. Oops, I’ve only been living in DC full time for 8 weeks, I don’t know where everything is yet (and probably never will). So I finally get in at the end of David Schleis’ AJAX talk, by the look of people it looks as if it was very good, no one was sleeping and everyone was paying attention. Next was David Wendelken’s .NET talk, I’m not going to lie, I wasn’t very interested and found everything to be a high level overview of .NET and really didn’t dive deep in anything specific in working with .NET and Oracle. Instead I spent most of that time building an updated J2EE Development Virtual Machine that includes the newest versions of Oracle XE, Oracle Applications Express, and jDeveloper 10.1.3. Next was Frank Nymphius’ talk on JSF, a very good overview of ADF and JSF. Talked about how easy it was to switch out their backend beans from TopLink to facades, I’d like to see the numbers of how long it took on that one ;-). Next up was John King talking about Oracle and JDBC, Some nice things around the batch processing and auto commit that can be turned off at the conncetion level. I think he should have better stressed the piece on callable statements and bind variables as there are some major security and performance gains there. And then lunch…

Right after lunch was Scott Ambler. Scott is in charge of Agile Development for IBM. The talk focused on Database Refactoring. Great guy with presentations the way I like them, to the point and very blunt. Calling out the obvious problems with current database models, and going into depth with real examples!! Open an honest about performance hits when your moving things and refactoring databases, especially when you use triggers to support it. Compares it to Sun’s Java deprication strategy for the database. Maybe its the RedBull I drank after lunch, but this talk really hit home with me. It seems to be that with all the projects I hit there are always problems with refactoring data models, I’ve suggested the same methods in the past. Its nice to see someone else as crazy as I am. One thing that he didn’t talk about was making sure your logging the usage of your “pre-factored” tables (the old deprecated ones). Scott discussed using triggers and such to allow old and new applications to synchronize tables and setting hard dates when the old tables are going away. If you take this approach you need to audit the “pre-factored” tables to see usage actively go down and when your nearing deprication time to know who you have to go “talk to” to go change their code and make sure the deprication will continue on schedule. When talking about configuration management, “If you don’t need to lock it down, then why are you bothering to do it.” It amazes me how many companies are so anal about locking down their J2EE code, but completely ignore data models and scripts. Database regression testing, unit testing for databases. I like this guy…database encapsulation with TopLink or Hibernate. Applications are evolutionary, and why shouldn’t data models be? Invest in regression tests to validate corporate data, regression test your stored procedures, need real strategies to fix endemic problems with data sources, data sources shouldn’t be the anchor around your team’s neck, cross train your developers to be database guys and vice versa, raise the bar for data management!!! Great talk, its not easy to do in practice, but if you can even take away some of the thoughts from his talk your organization will be much more flexible in their development and deployment strategies. Its already been done for years in the J2EE/Java arena, why not push the same concepts into the database world.
More Info:
http://www.agiledata.com
http://www.ambysoft.com/scottAmbler.html

Next up was Roland Berg, on the people factor of Web Architecture. One of the big points was that the keys to architecture that we really don’t understand today are People, Time and Motivation. Very true, thats why sometimes when theres the impossble project that no one believes you can get done, the team finishes early, and the project that should be done in 2 weeks takes 4 months (I’ve been on both sides of the coin there). Some good discussion on who leads at different parts of the project and who needs to have control on the direction:
Logical Model: Architects (Leads), Designers (supporting role), Engineers (supporting role), Stakeholders (as reviewers).
Physical Model: Designers, Engineers (supporting role), Architects (reveiw, preserve vision), Stakeholders (as reviewers). This is where technology selection belongs.
Implementation: Developers, Designers, Architects, Stakeholders. “Having the right people at the right place and at the right time is critical to the success of a project.”
A key point:
“The architecture must take into account the people involved at all phases of the project to be effective.”
I like to think this is one of the reasons that the teams l’ve put together through projects have succeeded

The last talk before the panel was Peter Koletzke on Web Application Communications. Honestly, it was a low level talk on the HTP request and response cycle and the components of those messages. I felt it was one of those “Well DUH!” talks and too high level for a Web Architecture Symposium. I checked out mentally and completed my proposals for Oracle World papers. Onto the round table…

The round table seemed to be a very much us vs.them talk. On one side was the group that wanted to put everything into the database, aligning with Paul Dorsey and his thick database concept. The other camp was the OO developer, SOA, new world camp as I like to call them. The big worry seems to be with just about everyone, how do I keep myself safe from changing technology while still remaining flextible. Its an age old fight that never really has an end. The thick database camp wants to put everything in the database because its one place to maintain all of the business rules and at the bottom of every stack. This may be fine and dandy but really doesnt leave things to be flexible, not everything runs through a database and not everything can live in the database. A question came up about where to put error checking for data being pushed and pulled from the database and other interfaces. My answer to the question is BOTH. As a rule of thumb I never trust anyone, no matter if its an external source or an internal service. Things always change, people always forget who’s using a service, and things arent always checked. Its a good rule of thumb to trust no one. ;)Thats all for today, tomorrow the BI Customer Advisory Council….

Haven’t been living up to my hopes…

Posted by Topper on 6/20/2006
Posted in: Discoverer, General, Oracle. Leave a Comment

As I said in my last post I was hoping to give daily updates, well I’ve been having too much fun meeting new people and catching up with friends.  I’ve been keeping track of everything and have posts ready to go, I just haven’t had time to give them the once over before posting them.  Lots of great stuff at the conference so far, BI its just plain exciting right now, hopefully Oracle’s customers will be able to afford it ($225K a proc, ouch!!!).  The JSF / Ajax / SOA presentations have been great.  I love Oracle right now, the only problem is trying to keep up.  Also, look for a site that I’ve promised for a long time to come online soon.  I think I’m getting final clearance soon to go ahead with it, keep your eyes peeled.  I think its going to be a lot of fun for everyone.

ODTUG this week

Posted by Topper on 6/15/2006
Posted in: General, Oracle, Technology. Leave a Comment

Well its time for another conference, I’ll be presenting at ODTUG this week and at the conference all week. TUSC recently opened an office in DC and I moved out here to help support it. So if anyone wants to catch me I’ll be working the TUSC booth or flying around. I’ll be one of the guys completely out of place in a suit the whole time.  UPDATE: I guess someone from TUSC is reading my blog, the day after I posted this an email came out saying this was now a business casual conference 😀

Saturday, June 17 8:30 – 5:00 Web Architecture Symposium
  9:00 – 4:00 Hands on Training
Sunday, June 18 1:00 – 4:00 Discoverer CAB – Wilson AB
 
 Monday, June 19 11:15 – 12:15 XML Publisher – Mark Rittman – Wilson C
  12:30 – 1:30 TUSC Booth
  1:30 – 2:30 BI Real World Cases – Dan Vlamis – Wilson C
  2:45 – 3:45 Successful Dimensional Modeling – Washington 6
  4:00 – 5:30 TUSC Booth
Tuesday, June 20 8:30 – 9:30 Why OWB – Jean Pierre – Washington 6
  9:45 – 10:45 ADF Recipes – Duncan Mills – Washington 5
  10:45-1:30 TUSC Booth
  1:30 – 2:30 BI DW Panel – Washington 3
  2:45 – 3:45 Quick Web Development – Matt Topper – Washington 4
  4:00 – 5:00 Climb to the OLAP Summit – Keith Laker – Wilson C
Wednesday, June 21 9:45 – 10:45 Why you need a TRB – Dan Norris – Washington 5
  11:00 – 12:00 Ajax and JSF – Frank Nimphius – Washington 3
  1:00 – 2:00 Effective Data Warehouse Architecture – Rittman – Washington 6
  2:15 – 3:15 Changing the World with SOA – Matt Topper – Washington 4

I’m really looking forward to this conference, there is a lot of new things coming out in the BI/DW space and it’ll be good to catch up with some friends. Hopefully I’ll get a chance to meet some new ones too. My goal is to blog some daily updates and give a recap. We’ll see how that works out.

Oh a good laugh!!!

Posted by Topper on 6/6/2006
Posted in: General. Leave a Comment

For those of you that know me IRL (in real life) you know that I’m the self proclaimed king of ADD and come up with more ideas than fort knox has gold bars. I consistently come up with new ideas and being a tech guy is my way to make them come to life. All my major project successes come from when people tell me “Those things will never work together” or “Theres no way to get that done, you know your job is on the line if you don’t”. This is me, this is where I thrive, major pressure, major issues and 1000 miles a minute. So when I ran across this blog entry today I had to laugh:

Of course with my ADD problem I copied the text of it and not the actual URL to the blog, so here it is:

These people come up with ideas and love incubating them but may or may not try to run them longer term. They:

  • generate an idea a minute
  • have a major case of ADD
  • are easily distracted by shiny objects
  • would rather generate 1 good and idea and 19 bad ones than just 1 good one
  • are always thinking about the next thing
  • are only excited by the possibility of what could be, not what is
  • are more philosophical and theoretical
  • probably shouldn’t run the companies they start for more than a few months, as they will frustrate everyone around them and get bored themselves
  • are really fun at cocktail parties
  • say things like “I thought of auctions online way before eBay!”

    • Sound like anyone you know, well except the whole “probably shouldn’t run the companies they start for more than a few months” I’m still friends with most of the people I’ve worked with over the years, well except those who couldn’t keep up or those I frustrated ;-). So if anyone wants to pay me boat loads of money to build my ideas or even a little money I’m all ears. Or if you have a cool project where it’ll never get done in time or things will never work together…bring it on.

    Update: I found the link to the original blog, http://onlyonce.blogs.com/onlyonce/2006/06/what_kind_of_en.html The article I read actually was stripped content from the original article (theres that ADD again). So based on the full article I actually think I might fit into that rare category…

    Comparison of the Oracle Spatial and Locator Features

    Posted by Topper on 5/11/2006
    Posted in: General, MapViewer / GIS, Oracle. 4 comments

    This is a quick analysis I wrote up for a customer, but I figured other people would get a benefit out of it as well.

    The Oracle database engine has an extremely robust feature set that allows for the analyis of GIS data within the database. Within the Oracle suite there are two products to perform spatial analysis in the database. One is called Oracle Locator, while the other is called Oracle Spatial. Oracle Locator allows for basic GIS analysis. While Oracle Spatial will allow for more advanced analysis, including creating new spatial data and transforming data that already exists. Currently, Oracle Locator is a free component of every database that Oracle sells; this includes OracleXE, Oracle Standard Edition and Oracle Enterprise Edition. Below you will find a list of all the features you can perform with each product and some examples of what can be done with that feature.  

    Oracle Locator

    Functionality

    Feature Name

    Example Use

    Support for All Spatial Geometry

    Points, Point Clusters, Lines, Line Strings, Compound Line Strings, Polygons, Polygons with Holes, Compound Polygons, Arc Strings, Circles, Rectangles

    All the counties in the United States
    Locations of all corporate sites
    Locations of all customers
    All of the roads in the United States

    Spatial Indexing

     

    R-Tree indexing to speed up query performance – selecting only the data needed, not an entire layer

    Spatial Operators

    SDO_FILTER

    SDO_RELATE

    SDO_NN

    SDO_NN_DISTANCE

    SDO_WITHIN_DISTANCE

    SDO_GEOM.SDO_DISTANCE

    The closest 100 customers to a site
    All customers that live in a particular building
    All of the stores in the shopping mall
    All of the Elvis impersonators in Nashville

    Two Tiered Queries

     

    Filter the data, first on the geometry elements that satisfy the condition, and then based on the relational facts (i.e. first find all customers within 5 miles, then find the ones that have spent more than $5,000 this year)

    Open Standards

     

    Certified with the Open Geospatial Consortium and OpenGIS standards
    Move away from PostGIS to Oracle
    Produce a standardized mapping service for customers

    Long Transaction / Workspace Management

     

    Create new “What If?” scenarios
    What if this location moves 5 miles west?
    What if this location is sold? Where will customers go?
    What if I combine the Northeast and mid-Atlantic regions?
    Load new datasets without affecting production queries and switch over seamlessly

     
     

    Oracle Spatial

     

    Functionality

    Feature Name

    Example Use

    Length and area based calculations

    Length / Area

    What is the average size of site locations?
    How long is a segment of road?
    How much area is covered by each sales representative?

    Generation of new geometries

    Buffer, Centroid, Convex Hull

    Geometry Intersection, Geometry Union

    Creating regions from current properties
    Creating 5 mile boundaries around each site to ensure market share

    Spatial Analytic Functions

     

    Determining what percentage of customers are within 5 miles of locations
    Determine the most effective place to put a new site, based on current customer data

    Identify which communities are growing and where to acquire new land for opportunities

    Find the highest income areas to launch for new executive line of products
    How far is the competition from the current customer base?

    Coordinate system transformations

    SDO_CS.TRANFORM

    SDO_CS.TRANSFORM_LAYER

    Translate data from localized projections to a national projection
    Convert Universal Transverse Mercator data into longitude and latitude points

    Raster image support

     

    Include aerial images with the application to assist in problem analysis
    Why do people drive 20 miles to this store when there is one 10 miles away? Because there is a mountain in the way
    View how land has changed over a period of time and determine where people are moving

    Network datamodels

     

    Can I make a left hand turn at this intersection?
    Where are the gas lines on this property?

    Geocoding

     

    Standardize addresses based on postal information
    Correct addresses with common misspellings or alternative street names
    I have a GPS location, who owns the property there?

    Routing

     

    How do I get to the nearest gas station from my current position?
    How long will it take me to get there?
    The street is under construction, what other alternatives do I have?
    Should I take the bus, a cab or the metro from the airport? How long is it going to take me and what does it cost?

    Like this article? Digg it!

    Analysis of Geocoder.us vs Free Navteq Data for San Francisco Starbucks

    Posted by Topper on 5/9/2006
    Posted in: MapViewer / GIS, Oracle. 3 comments

    As promised (I know a couple days late, but I had a brainfart on some of the non-codable data) here is the analysis of the Geocoder.us data vs. the Free San Francisco data from Navteq. The data I used to prove this was 75 records for the Starbucks with a San Francisco address.

    The first thing to discuss is the extra time it took for the Geocoder.us data. When I geocoded the data against the Navteq data that was loaded inside of the Oracle database it took me approximately 1:15 seconds to geocode the 75 records on my laptop. The same data through the Geocoder.us csv webservice took almost 10x as long, it was ~10 minutes to took to code the same 75 rows. OK, definitely a major hit there, the latency was due to the UTL_HTTP call I was making out of the database was taking 12 seconds per record to return. The Geocoder.us website discusses how to setup your own local Geocoder.us server, maybe in the future I will see what the difference in time is using their interface locally.

    On a good note, there were 3 addresses that the Navteq data would not recognize, but the Geocoder.us data recognized all but one row. The one row that it did not recognize had ‘ONE’ for the street number instead of the actual number 1. When replacing the number for the string in the data, the site was successfully coded. Simple enough to fix, but still a limitation.

    OK, now onto the data. The Navteq data was coded down to 12 decimal places, while the Geocoder.us data only gets coded down to 5 decimal places. This might not seem like a big deal, but take this into account.

    decimals degrees miles-statute feet inches
    0 1 69 364320 4371840
    1 0.1 6.9 36432 437184
    2 0.01 0.69 3643.2 43718.4
    3 0.001 0.069 364.32 4371.84
    4 0.0001 0.0069 36.432 437.184
    5 0.00001 0.00069 3.6432 43.7184
    6 0.000001 0.000069 0.36432 4.37184

    So basically we are talking a difference in less than inches. While this might not be a big deal for the address is off by 4″ (I think we’ll be able to find it) but if it was a missle defense system, we may have some issues that need to be discussed. Since normal GPS devices are only accurate to about 15 feet on the best days this pretty darn good for free.

    The first problem we run into is that the Geocoder data from Navteq is in SRID 8307 format, aka Longitude/Latitude (WGS 84) and the Geocoder.us data is in SRID 8265 format aka Longitude / Latitude (NAD83). So for most people this means nothing, but its actually very important. Remember back in elementary / middle school where they showed all the different maps of the world; one where greenland was really big, one where there were rips in the map, etc. Well this all comes back to us when we are talking about Coordinate Systems in Oracle Spatial. All the different types of map projections; coordinate, catesian, geodetic, projected, geodetic datum, and authalic sphere are all present in the Oracle Spatial database predefined for our use. These projections follow the defined standard projections defined by the OpenGIS consortium ( http://www.opengeospatial.org/specs/?page=specs ).

    Very simply I wrote a quick SQL statement that uses the built in SDO_CS package to transform the data from 8265 to 8307 and then calculates the distance between the two points in meters.
    select /*+ordered*/
    o.store_name
    , o.location navteq_location
    , g.location geocoder_location
    , sdo_geom.sdo_distance ( o.location, sdo_cs.transform(g.location, 8307), .005, 'unit=METER') distance
    from sf_starbacks o,
    sf_starbucks_geocoder_us g
    where g.store_name=o.store_name
    order by distance;

    Below is the results table that was returned from the function.

    Store Name Distance in Meters
    SF Courtyard Marriott Lobby 2.71659553129724
    Spear Street 3.34121058899212
    455 Market 3.55328348605654
    Market & Fell – San Francisco 3.8057058105241
    1750 Divisadero Street 4.69942861653962
    99 Jackson 4.70218233521226
    Sansome 5.11814739909792
    California & Battery – SF 5.29755965611037
    123 Battery 6.05824239531821
    505 Sansome Street 6.08548442574047
    Laurel Village 6.3455649117665
    3727 Buchanan – San Francisco 7.21260972749949
    Kearny @ Bush 7.22469004513458
    123 Mission Street 7.4188927049375
    1231 Market Street 7.42953327611775
    4094 18th St. 8.08740578188483
    3rd & Howard 8.64205159112492
    701 Battery 9.9626753714261
    Grant & Bush – San Francisco 10.0439995350524
    1800 Irving Street 10.0918780314643
    398 Market St. 10.0937804181894
    425 Battery – San Francisco 10.2121900661491
    333 Market St. 10.3904072176329
    340 Mission 10.4766571905475
    Hills Plaza 10.5820256764645
    50 California St. 10.812256443162
    Fillmore & O'Farrell (UCO) 11.235605600453
    Masonic @ Fulton – S.F. 11.2570582622603
    901 Market St. 11.383558585046
    565 Clay St. 11.500375430594
    675 Portola – Miraloma 12.0143457537109
    4th & Brannan – WFB 12.0215273447908
    Chestnut 12.7235318198783
    199 Fremont @ Howard – SF 13.2578337955333
    Union Street 14.4825612550203
    390 Stockton @ Sutter (Union Sq) 15.419906040731
    36 Second Street 15.6140523231353
    74 New Montgomery 15.9682235898887
    Safeway-San Francisco #1490 16.1190250104597
    Safeway-San Francisco #2606 16.1745772800324
    Mariposa & Bryant 16.2510104423628
    44 Montgomery @ Market St. 16.4293094494693
    King & 4th Street – San Francisco 16.5061673580469
    Levi's Plaza @ Sansome 17.097438240954
    Fillmore 17.2944166057407
    Kansas & 16th St. – San Francisco 17.4991251104624
    9th & Howard 17.6327431473884
    Sony Metreon SF (UCO) 18.213309935106
    120 4th Street 18.3932339959993
    27 Drumm Street 19.067431537748
    Polk Street 19.2677065095131
    201 Powell Street – San Francisco 19.2714989034239
    Beach & Hyde – San Francisco 19.3904276298555
    Irving Street 19.5731611233583
    Van Ness & California – WFB 19.5861907980684
    15 Sutter St. 19.9180139893338
    Grand Central Market – Mollie Stone 21.2170151879307
    Albertsons – San Francisco #7146 21.217250792467
    Jones @ Jefferson – San Francisco 21.3270789286301
    Cyril Magnin @ O`Farrell – Nikko 21.4961567925423
    24th & Noe 22.2641624424109
    Geary & Taylor – San Francisco 24.0074384547142
    350 Parnassus 25.5867539051804
    555 California St. 27.5157348868899
    Safeway – San Francisco #667 28.7814007353958
    Bush & Van Ness – S.F. 28.9662827730563
    100 West Portal/Vicente 31.8815087423281
    4th & Market – S.F. 49.9397278381422
    Church & Market – S.F. 71.3778706089687
    Safeway-San Franscisco #1507 74.3198277064713
    Stonestown Galleria 76.3800406139539
    Albertsons – San Francisco #7128 182.468554131262
    5455 Geary Blvd. – WFB N/A
    Albertsons – San Francisco #7137 N/A

    Out of the 75 Starbuck stores in the table, 3 could not be compared because one of the two distances did not have a geocoded value for the address. The minimum distance difference was 2.71 meters and the maximum was 182.47 meters. The average distance difference was 19.44 meters. So on average the difference is alright, 20 meters average is fine if your trying to find a stores location on the street, but to do anything that requires more precision than driving directions you’d probably want to be assured that your data was more accurate. Even in the worst case it is only off by a tenth of a mile. Which data is more correct? Within the industry its typically accepted to be the Navteq data. Navteq spends many dollars a year to validate their data and make sure its the most accurate around.

    The Yahoo uses both Navteq and TeleAtlas data for their geocoder. Maybe I should run some analysis against their service to see what comes out…anyone interested?

    Adding a new disk to a VMWare Virtual Machine in Linux

    Posted by Topper on 5/7/2006
    Posted in: General, Technology, VMWare. 212 comments

    I’ve been using VMWare for a while now and I always get asked some common questions about it. One of those is how to add a new virtual disk to a Linux virtual machine. So in response to that, here are the steps to adding a new SCSI based virtual disk to a CentOS Linux virtual machine. The steps for adding a disk to a Windows machine is very much the same except you would use the Disk Management utility from the Control Panel.

    Step 1: Open virtual machine settings
    Select your virtual machine, as you can see from the photo I selected the Infrastructure virtual machine. Next press the “Edit virtual machine settings’ to open the Virtual Machine Settings dialog.
    Select your Virtual Machine

    Step 2: Add new hardware
    From the “Virtual Machine Settings” dialog select the “Add…” button at the bottom of the screen. From this dialog you can also modify how much memory you dedicate to the machine when it boots.
    Virtual Machine Settings Dialog - /> Select the Add… button” src=”https://matttopper.com/images/blog/adding_disk_to_vmware/2.jpg” /></p> <p>Step 3: Start hardware wizard<br /> Next we will walk through the “Add Hardware Wizard” the process makes it very simple to add additional hardware to a predefined virtual machine. From this screen we want to select the “Next >” button.<br /> <img decoding=

    Step 4: Select new hard disk
    From this screen we can see the many types of hardware we can add to a virtual machine. You can emulate just about any piece of hardware that one can expect in a modern operating system. It definitely makes testing with different configurations and devices much easier. For our example we want to select “Hard Disk” and then select the “Next >” button.
    Select Hard Disk from the Add Hardware Wizard

    Step 5: Create the virtual disk
    In the next screen we see the three options for adding a new disk. We can “Create a new virtual disk”, this will create a brand new disk on the guest operating system. The second option, “Use an existing virtual disk”, allows you to mount a disk from another virtual machine. I like to do this with my “source” drive. I have one virtual disk that I’ve made that has all the Oracle and Linux CDs on it, that way I can just mount it to the machine I need when I have to do a new install instead of copying the binaries I need across disks, its definitely a big time saver. The last option is to “Use a physical disk”, this allows you to mount a local physical disk to the operating system. This option is akin to NFS mounting a drive to a virtual machine. To add a new disk we select the “Create a new virtual disk” option and select the “Next >” button.
    Create a New Virtual Disk from the Add Hardware Wizard

    Step 6: Select type of disk
    Next we want to select the type of disk. I’ve been using VMWare for a long time and agree that the recommended Virtual Disk Type should be SCSI. I don’t know why, but I’ve had much better success with the SCSI virtual disks than the IDE ones. So in this step we want to select “SCSI (Recommended)” and the “Next >” button.
    Select Disk Type from Add Hardware Wizard

    Step 7: Set disk size and options
    Now we want to set the size of the disk we are creating. One of the nice features of VMWare is that you don’t have to allocate all of the disk when you create it. So if you create a 40 GB disk it doesn’t have to take it all right away, the disk will grow as your virtual machine needs it. I will say this is a big performance hit you take when the disk has to extend, but for most applications its OK. Also, I will warn that if the virtual disk grows and there is no physical disk left on the host operating system you will see a catastrophic failure and in most cases both the host and guest operating systems lock up and become unusable. (Don’t say I didn’t warn you) Lastly, you can split the files into 2GB sizes, while this isn’t necessary, it just makes all the disks much easier to manage and move around. For this step we want to set our disk size (12 GB in this case), I chose not to allocate the disk space right now (the machine has a 300 GB drive and has only 20 GB on it) and Split disk into 2 GB files.
    Specify Disk Capacity from the Add Hardware Wizard

    Step 8: Name the disk file
    This is actually pretty simple in that you decide what you want to physically call the disk and where to put it. .vmdk is the extension for VMWare virtual disks. After we name the disk we can select the “Finish” button which adds the disk to the virtual machine.
    Name the Virtual Disk File from the Add Hardware Wizard

    Step 9: Ensure new disk exists
    So now we can see that the new disk has been added to the “Virtual Machine Settings” within the selected virtual machine. From here the disk acts just like it would if you added a new disk to a standalone server. So we select the “OK” button to continue.
    Review Virtual Machine Settings for the newly created disk

    Step 10: Boot the virtual machine
    From here we just start the virtual machine like we would normally, either by selecting the button on the toolbar or selecting the “Start this virtual machine” link.
    Start the Virtual Machine

    Step 11: Virtual machine start up
    The machine boots normally as it would any other time.
    Booting Virtual Machine...

    Step 12: Create the Partition
    After we’ve logged in and accessed a terminal window as root (or another user with root/sudo privs) we first want to run fdisk on the newly created drive. In Linux the first SCSI drive is sda, the second sdb, the third sdc, etc. since this was the second SCSI drive we added to the system, the device is known as /dev/sdb
    The first command we want to run is fdisk /dev/sdb (NOTE: Thanks to everyone that caught my typo here) this utility works very much like the DOS utility of the old days and allows you to create and manage partitions. To create a new partition we enter the command n to create a new partition. This is going to be a primary partition p, and the first partition number 1. Because I want this disk to consume the full 12 GB I specified earlier we start at the first cylinder and end it at the last cylinder. We then want to write the partition table with the new partition we have just created so we enter the command w which writes the new table and exits fdisk.
    Create new partition with fdisk

    Step 13: Format the partition
    Now that we’ve create the partition, we now want to format the first with the new file system. I’ve decided to use ext3 filesystem for this disk, ext3 provides all the features of the classic ext2 file system plus journaling which helps to prevent disk corruption in the event of an improper shutdown and speeds up the recovery process. For a good overview of Linux standard file systems check out this article: http://linux.org.mt/article/filesystems So, to format the new partition we enter the command mkfs -t ext3 /dev/sdb1. This command makes a new files system with the type t ext3 on the /dev/sdb1 partition, this is the first partition on the sdb disk.
    Create new filesystem in the virtual machine

    Step 14: Create the mount point
    Determine where you want to add the new virtual disk you’ve created. I like to create a partition specifically for all the software I install after the basic Linux install called /software to do that we run mkdir /software, just a simple make directory command. Once that is complete we then want to mount the newly created partition. Because we haven’t added the partition to the /etc/fstab yet we have to mount it manually. To do that we run mount -t ext3 /dev/sdb1 /software. To break down this command we run mount with the ext3 filesystem type, the partition /dev/sdb1 to the directory /software. Pretty simple and straight forward. To check that the partition is properly mounted we run df -k which shows us the mounted partitions and the amount of available space.
    Create new mount point and mount the new drive

    Step 15: Open the fstab file
    The fstab file holds all of the used disks and partitions, and determines how they are supposed to be used by the operating system. So we edit the file to add the newly created partition
    https://matttopper.com/images/blog/adding_disk_to_vmware/15.jpg

    Step 16: Modify the fstab for the new partition
    After we open the fstab file in the previous step we add the following line:
    /dev/sdb1 /software ext3 defaults 1 1
    The first column is the partition name, the second is the default mount point, the third is the filesystem type. The fourth is the mount options, in this case I used default which mounts the drive rw, suid, dev, exec, auto, nouser and asynchronous. The 5th and 6th options are for the dump and fsck options. If dump is set to 1 the filesystem is marked to be backed up, if you are going to have sensitive material on the drive its a good idea to set it to 1. If fsck is set to greater than 1, then the operating system uses the number to determine in what order fsck should be run during start up. If it is set to 0 it will be ignored such as in the case of a cdrom drive since its a solid state disk. For more information on the fstab file check out this article: http://www.tuxfiles.org/linuxhelp/fstab.html
    Lastly, we write and quit the file with the :wq command.
    Editing of the fstab

    So now that the fstab has been written the drive will be mounted and unmounted when the machine is either started or shutdown. So there you have it, the quick and dirty process for adding a brand new disk to a virtual machine. Until next time…