Allan Hirt is a well know High Availability writer and practitioner and SQL Server Clustering MVP or a Clustering MVP at least.
Greg Low: Introducing Show 52 with guest Allan Hirt.
Our guest today is Allan Hirt, Allan is a well know High Availability writer and practitioner and SQL Server Clustering MVP or a Clustering MVP at least. Welcome Allan!
Allan Hirt: Thanks Greg.
Greg Low: So as I do with everyone, the first they are on the show, I will get you to tell us, how did you ever come to be involved in SQL Server at all?
Allan Hirt: For me it was actually it was more straightforward than for most people. I was in college and I was actually interning at Sybase. Actually a company that Sybase ultimately bought, so I would be using databases in some form since 1992, and after college I didn’t go work for Sybase but I did work for a company doing QA on a database based application. I sort of always I guess in my professional life had something to do with databases and one way or another.
Greg Low: What made you migrate towards High Availability as an area to focus on?
Allan Hirt: I think I have always been more of a platform IT kind of guy overall. I never loved programming and what a lot of people do today, for example like performance tuning. It doesn’t mean I can’t do it, it is just not me. I am more of a back end kind of guy, admin ops. It sort of fits my need. What really pushed me over the edge actually was I didn’t know at the time, but I actually played with the Wolf pack stuff which is the clustering, pre for those who don’t know, this stuff came before Windows got it right. There is an incident after I bought, well after I joined Microsoft and I was working for Microsoft for a while with this particular customer where they had a really bad availability issues which sort of bought it on for this for me and snowballed from there, at this point was almost 15 years ago.
Greg Low: Yes, that’s great. Well listen SQL Server 2012 or I think the marketing team there like to call it SQL Server twenty twelve, basically there are a lot of changes in around the high availability area. In general each version seems to make a big effort towards increasing the availability in the product. Before we get into the main areas of high availability, what I do want tackle is quick discussion on Windows Server Core and the ability to deploy on that? Because again that is a substantial change in 2012 and just your thoughts around that and info for the folk and what you can and can’t do.
Allan Hirt: Sure, so a quick recap for those who don’t know what Windows Server Core is think of your typical SQL Server box but without having any kind of user interface. Basically there are a DOS or Powershell prompt and faster to get. For most people that is really scary because a lot of Admins out there, not just SQL Server hasn’t seen a command prompt since DOS.
Greg Low: Yes.
Allan Hirt: I mean once we kind of gone gui, most people haven’t looked back. So now we have in essence Windows telling us command line is not such a bad thing and it is not. It never has been, but you know most of us have gotten so use to using gui tools like SSMS that the thought of having to type commands on the box is scary. Now having said that, it is something that Windows type folks or Systems type folks have been asking from Microsoft for a long time. Not necessarily DBAs mind you because they want a Linux like experience I guess or UNIX like experience for Windows where you could deploy things on the box that does not have a gui. If you really think about it, this is where I think people forget where what we come from. Back in the days when we deploy things like Oracle and Sybase on say SLX, you just never have a gui. I mean you are configuring things like TNSnames.ORA by editing a file. For even the database world, we didn’t always have a gui. One of the things when it started to happen, let me give a little bit of history how Windows got here. Back in Windows Server 2008, Microsoft introduced Server Core. The problem was at the time, it had no .NET support meaning that outside of Active Directory and a file server you pretty much weren’t putting anything on it. There was no adoption, so clearly something had to happen or otherwise it would become really niche. What happened over time is that Microsoft put minimal .NET code in there to be able to run servers. This is why SQL Server can now actually run up but it is not everything. There are certain things for example like Engine that runs on there but you are not going to get SSMS. That is totally fine. The real issue is for folks that are finding in adoption, is they are scared of not having a gui when they get on the box.
Greg Low: Yes. I suppose we should discuss is why do they care if they have a gui or not? What is the advantage of not having one?
Allan Hirt: Right, so very excellent question. The one thing that I hear the most is the least important one is oh I will get better performance with less stuff running. If you think back we don’t have a gui because a lot of services may not run. People think that this automatically means better performance. It may mean some better performance but to me that it is another 1 or 2% it is negligible. I don’t think you are going to see night and day between running a SQL Server instance on Server core versus Server Full. I reserve the right to be wrong there but I haven’t seen it yet.
Greg Low: Look even I suppose I see the temptation to run other things on the same box is dramatically reduced because there is a lot more things run on there anyway.
Allan Hirt: Right, and especially where the big benefits come in are security and really better configuration. Now better configuration is relative. Now we talk about a security for a second, how many times have we heard the stories out there where people are logged on somewhere and they forget what box are on and they go and issue a command and you know that is that? One of the reason is better security is that this will get us off our reliance. I see this all the time at customer sites, of customers feeling the need to logon to a production box just to send one SMS or look at something.
Greg Low: Yes.
Allan Hirt: I mean, we all know inherently that we should be administering boxes remotely. If you have to get onto a box and all you see is a DOS prompt, it is going to give you cause. For good reason, your reason is to go on there and fire something to go and kill it. Now it doesn’t mean to say that you can’t take of it remotely but I think it will get us back to the core of what we always needed to do from an admin perspective is not be on the production box if you don’t have to be.
Greg Low: Yes. So security there, I was going to say the biggest noise I hear is usually around patching and availability.
Allan Hirt: Yes that is actually the biggest one. So I actually I did a thing for TechEd last year and I am actually pulling it up right now, where I did a test on Windows Server 2008 R2 at the time and installed a box with four gui and then a box just a Server core and installed SQL Server 2012 on there. All stock nothing weird, anything additional. Once I ran Windows update, what the differences would be, the funny thing about it though is there was actually pretty significant difference. Microsoft has some places up like 70% that is a bit high you know. I was a little bit skeptical, but at the time what I really saw was really anywhere about a third to almost 40% - 45% difference in the number of patches that Microsoft recommended for me or said were more important. That was significant. So when most of IT has these days very limited windowing for deploying patches say Solaris on a weekend, reducing the number of patches by even just a third could mean significant time savings. Forget the fact that you are not putting additional patches on your box that you probably don’t need to begin with.
Greg Low: Yes it is the whole argument that I think you don’t need to be patching Internet Explorer and things like that. And having that cause reboots to the Server, I mean overall things just going to be up more often.
Allan Hirt: Oh absolutely. For me that is arguably the number one reason to deploy Server core. But the resistance I am seeing more often than not these days with customers is it is not just small ones. Small ones I can understand because they are resource constraint and you know they have one or two admins both of who are probably supposed to be jack of all trades. Even the larger companies where I had one very, very large multi-national customer tell me that the reason they are not doing Server Core because there admins would screw up more by typing commands than they would by using a gui.
Greg Low: Yes.
Allan Hirt: That is the unfortunate reality because the benefits at least a third of the patching savings and you know more security should trump some of the other stuff, because it is not that hard to learn. I mean nobody has asked me to write complex Powershell scripts. Although as aside, you really should learn Powershell, that is something else I want to mention too. First of all let’s just clarify we are talking about SQL Server 2012 here.
Greg Low: Yes.
Allan Hirt: They didn’t port SQL core compatibility to SQL Server 2008 or 2008 R2.
Greg Low: Yes.
Allan Hirt: But one of the biggest changes to Windows Server 2012 was that unlike the previous versions, where there was some Powershell stuff. Windows Server 2012 has 2400 plus commands lits for Powershell. Meaning everything that had a DOS equivalent now has a Powershell equivalent. That is pretty significant.
Greg Low: Now that is good, and so what do people need to know in terms of installing it? So if we tackle that, so in terms of the installation experience how is that different for an existing person use to installing SQL Server what are they going to find when they deploy on core?
Allan Hirt: Well first of all you can’t use the gui. If they are used to doing gui based installs. I apologise but that is not going to be your experience. So you have to use SQL Server’s command line experience or running a configuration file then calling it from steup. Either way if you are use to doing that now you and in your current environment and you are using the gui. You will have no issues transitioning to installing on Server Core. If you are not use to doing that, maybe there may be a bit of a hiccup, but one thing you can do to sort of get there now is on your gui based machine for example is that at the very end of SQL Server setup on the Confirmation information page you will notice at the very bottom, the little one that says “Configuration Files Here”. That you can say is a basis and see how SQL formats it and take important bits and create your own or just even just modify that.
Greg Low: So the experience basically is very similar to doing an unintended install on the current systems?
Allan Hirt: Absolutely, the only real differences are what you can install which is all clearly laid out on Books On Line. Some of that may be a bit of trial and error for folks getting their scripts right, but once you do you are pretty much set.
Greg Low: Yes, so in terms of components obviously the database engine is happy. What about Analysis Services?
Allan Hirt: The core of Analysis Services I believe should work and I have the list in front of me.
Greg Low: I would of thought so.
Allan Hirt: Let me get to the more assertaric stuff. When you a kind of looking at it and going look, probably not.
Greg Low: Well, I would think for example in Reporting Services the engines going to run but the portal isn’t right. That is what I am sort of saying.
Allan Hirt: Right.
Greg Low: There is nothing to stopping you. The portal is going to be somewhere else, all be it with a separate license but basically it is going to be the things that are going to be the back-end engine components are the ones they are going to run on the core.
Allan Hirt: Right, you have a very good point though, because this will also force people to think about their licensing schemes. If they are used to minimizing by putting a bunch of stuff on a single box or a single cluster. This now also makes you rethinking your licensing, because you have heard also changed quite a bit since SQL Server 2012 which is also given a lot of customers quite a bit of cause.
Greg Low: Yes. The other think I suppose I should mention as well, for those that want to try this obviously they could spin it up in a virtual machine and try it. The other thing that they will be pleasantly surprised is how quickly Server Core itself installs compared to the other operating system.
Allan Hirt: Oh absolutely, I mean in general the Windows experience for installing the OS has gotten much quicker but installing it is pretty much minutes. We are not talking hours and hours of going out to dinner and coming back. It is pretty amazing!
Greg Low: The other thing I found is that after it was installed the initial Windows update, that process was again so much shorter. Where whenever I installed an operating system nowadays it seems like I am going to be sitting there not so long for the operating system to install but it is going to be the endless updates that come down after I try to finish installing it.
Allan Hirt: Exactly. The one thing that you know is just a subtle difference between the SQL Server install experience in general Server Core not on Windows Server 2008 R2 say versus Windows Server 2012 is that a significant portion in my experience of the SQL Server install time on Windows Server 2008 R2 is installing .NET 4.
Greg Low: Yes.
Allan Hirt: Whereas on Windows Server 2012, .NET 4.5 is already built into the OS so the SQL Server is itself actually quicker on Windows Server 2012.
Greg Low: Are there any notable things that you, I will probably do a show separately on this but notable things from Windows Server 2012 that you think are worth noting at this point?
Allan Hirt: There are a few, a lot of their investments were made in for example in the Hyper V area. So we can have up to 64 nodes of clustering which is essentially a clustering things because mainly we put in for the Hyper V folks they can have I think it was 4000 VMs or something like that. There is a cluster aware updating so Windows can now help you patch your clusters automatically and move things and reboots to minimize some of the interference there. SQL Server doesn’t directly support it yet with its patching so it will be interesting to see if it will ever support that for passive cluster instances of SQL. It is sometimes a bit of a hassle.
They have improved things like check disk now it can potentially run in seconds. What else? There are a few other things.
Greg Low: I think the other big one I was thinking of was SMB 3.
Allan Hirt: Yes, so SMB 3 will be really interesting for SQL because in 2012 also supports deploying your databases on checks.
Greg Low: Yes I suppose for those that just are not across that, so SMB is the protocol that is used for basically disk sharing and Windows ever since the year dot. Basically from there SMB 3, basically is dramatically faster and the other big thing is that in the case things like SQL Server we can now use it as a shared disk sub system instead of necessarily having to use something like i-scuzzi.
Allan Hirt: Right, but the problem now we have is we are shifting what we need to make available or what you are assuming your internal guys can make available for you. Now you need to make sure they are putting on a highly available SMB share that will give you the performance need and your network needs to be rock solid. Those are pretty standard things anyway, things that a DBA doesn’t necessarily think about but is ultimately is going to be a responsible before it can access it the box.
Greg Low: Yes, that’s great. So listen where I wanted to head is a little bit now is just the general high availability improvements in SQL Server 2012 and I suppose the AlwaysOn word has reappeared. It appeared initially in 2005 to describe some of the technologies there but they are all very much under the AlwaysOn banner in this version. So for perhaps what we have had previously is a sort of clustering and mirroring, so maybe if we start with where mirroring has evolved and then we will look at where clustering has evolved.
Allan Hirt: Sure, AlwaysOn has been used for a while, for certification and for store vendors too for SQL. So it has had a very stormy past. What really Microsoft has done in SQL Server 2012 is that they had this brand new feature is AlwaysOn Availability Groups and features Availability Groups it was not AlwaysOn and it sort of taken elements of what we know today as database mirroring and elements of what we know today as failover clustering. It is essentially combines them into a single feature.
Greg Low: Actually that is something we should probably make clear too. It is a good point, I hear people refer to it just as AlwaysOn but understanding there are two parts to AlwaysOn; there is AlwaysOn Availability Groups and AlwaysOn can refer to fail over clustering changes as well.
Allan Hirt: Also depending on who you talk to, the AlwaysOn monitor which is a marketing umbrella also refers to anything even remotely that has to do with High Availability including DBCC commands that are online. We have had to base this on twitter and blogs and even the Microsoft folks I talk too. You can talk to 5 of them 3 of them will have a different opinion.
Greg Low: Yes, so specifically at this point, we are talking about AlwaysOn Availability.
Allan Hirt: Correct, so what Microsoft has done here, this is database availability so following on in the tradition of database mirroring here. Now with database mirroring we had a few limitations. The feature was introduced in 2005 SP1 and it has worked well for a lot of people. So they didn’t want to change it in a way that would make unfamiliar right? As a lot of things sometimes happens, one of the benefits of database mirroring is that unlike log shipping for example which is based on transaction logs you take your point in time transaction log backups copying them elsewhere and applying them. There is going to be bigger delta of data needing potentially larger amounts of data loss in your High Availability and disaster recovery solution. With database mirroring it is at a per transaction level so whether you are doing it synchronously or asynchronously your secondary server will be a shorter delta or potentially even if it is synchronous it is completely caught up at the same point as your primary which is obviously a better scenario. Something like log shipping ever provided for us.
Greg Low: Yes.
Allan Hirt: Availability Groups continues on from that, and takes a lot of the gusts in database mirroring but what they changed was some of the mechanisms and some of the enhanced a few things that were more limited. For example in database mirroring you could have single mirror, you could add a primary and a secondary that was it. Now with Availability Groups we have the ability to have more than one ratio if we want. So we can have one primary replica and up to four secondaries for a total of five. Which is really nice, and another thing they have done in that is one of the biggest gripes most people have with almost every SQL Server availability solution in the ability to use a secondary for say read only purposes.
Greg Low: Yes.
Allan Hirt: Where that stems from is a lot of management types look at it and going we have got this server sitting here running SQL. It is not doing anything, we want it to do something. Let’s take licensing out of this equation for a second and let’s talk about that. That I understand because you are looking at it and going well it is just sitting here idol, we want to have every server in our farm and our data center. I can understand that but also remember you are putting this in mainly for availability solution. So if you hamper that availability solution then the purpose of using that secondary machine makes no sense. Having said that to have enough capacity on that secondary all of a sudden assume you can get over the licensing that will cost. By the way I have also heard people complain about the fact that if you are going to be using a readable secondary replica with availability groups that people that likes it. Here is the sad reality. This would have been true in any version of SQL if you are using the secondary for reporting. If you are using log shipping or whatever.
Greg Low: Yes, in mirroring if I had a purely passive partner then in that case then the thing is I didn’t have a SQL Server license cost still I have Windows licensing cost. But the minute I made it readable via snapshots or something like that suddenly I have a licensable secondary instead. That aspect hasn’t changed. One of the bigger differences here is that with mirroring as you say can only have a single and people often wanted local High Availability and more remote sort of disaster recovery and at the very minimal people wanted sort of a local synchronous replica and they wanted a remote asynchronous replica and that’s probably to me the biggest change is that you can do that, plus the ability that you can push out to multiple replicas if you want to.
Allan Hirt: Well that is one bigger chance for example to achieve what you were just talking about. To have a local synchronous and non local asynchronous. General like database mirroring plus log shipping.
Greg Low: Yes.
Allan Hirt: Now you can have it all in one feature. There is a couple of other improvements for example for the readable secondary it is near real time. It is not like say database mirroring where you have to create a snapshot and keep recreating snapshot which are essential a few a time.
Greg Low: Yes that was a fairly painful process, I remember we had to setup something that endlessly created snapshots and also cleaned up the old ones when no one was connected to them and then the client had to look into a table first up to find what to connect to. So it wasn’t a trivial thing to have fairly up to date readable secondaries on mirroring. It is really an easy story.
Allan Hirt: Another big change and this is sort of where we start to get to bring clustering into the picture. Let’s get something straight up, remember that when we talked clustering there are couple of different things. There is Windows Server failover clustering which is what SQL Server Failover Clustering is built on and what I talking about here is context of availability groups is Windows Server Failover Cluster. An availability group does require that the underlying servers participate on a Windows Failover Cluster. Now this doesn’t mean that you actually have to cluster SQL itself. You can but you don’t have to. The reason for this is twofold really. One if you are using database mirroring in the past and were getting synchronous database mirroring and you wanted to have automatic failover you had something on the witness. Essentially the witness was the core mechanism built into server failover clustering. So what they essentially done, so instead of coding your own you were now using a standard process in Windows to deal with core.
So that is number, the second benefit of having the Windows Failover Clustering feature underneath the covers is a brand new functionality called the Listener. You might see if referred to in some Microsoft documents as VNN or Virtual Network Name. Which is really actual name of the resource in the cluster which is neither here or there? What the listener is for those of you familiar with standard SQL Server Failover Clustering instances know that when you install it you have to give it a unique domain name. And does it different underlying nodes and that unique name is what you connect to, regardless of what node it is connecting to underneath the covers.
The listener is essentially works just like DAC. Where you can create a listener so that regardless of where your availability groups sit underneath the covers on whatever server. You connect through this listener and you keep the name in your connection string. That is really huge so the thing that seems like database mirroring or log shipping or even say replication. You are doing things that create DNS alias if you are doing any number of things to extract to extract that server name change underneath covers. Now we have a standard mechanism built into the feature that allows us to have one way to connect in even though it is database level availability. It is not instance level availability.
Greg Low: So I suppose one of the summaries here, the thing we are saying is that we are mirroring before didn’t have dependency on the underlying clustering in Windows. Now we have a dependency from AlwaysOn Availability Groups in SQL Server. They have a dependency on Windows Clustering not on the SQL Server Clustering but one of the big things they have tackled there is how does the client find the database to connect to. Now we have a concept of a virtual name that they connect to which automatically re-routes them to where they need to be connected to rather than the client having to know how to do that.
Allan Hirt: Yes and you can use not only read-write connections on the primary but also only connections to readable secondary. Literally it is the universal type.
Greg Low: Yes that is actually a really interesting point as well. So where we are saying we have a readable secondary and a client connects to that it is not only re-directing the people that connected to the primary it is how we re-direct the people that connect to the secondary as well. It also takes care of that.
Allan Hirt: Absolutely not, one of the thing that I am sure Greg would agree with me on is having this Windows Failover Cluster underneath does make the planning touch more complex than just say going to deploy database mirroring. That is all tack able for a lack of a better word.
Greg Low: So as you were saying we don’t need a witness anymore so that goes away however we need to satisfy the quorum requirements by using clustering but the problem is that you are sort of alluding to. Is not everyone can migrate necessarily to clustering. We had mirroring before, where it was cross domain mirroring where people were in servers and different domains and even unrelated you could set up mirroring with certificates to do the authentication. One other big things you have to think about when going to these availability groups is it all has to be a domain.
Allan Hirt: Well not only the domain, in the same domain it can’t be trusted domain. SQL Server 2012 fully supports multi subnet functionality Windows. You can still stand multiple subnet with no issue is it absolutely correct now you go into a scenario where everything is domain based and it may not work with some people. The other thing we should mention that officially database mirroring considered deprecated in SQL 2012.
Greg Low: Yes it is still there but it is not an upgrade blocker, so if you are on mirroring there will be a solution in 2012 that works for you but may or may not be availability groups.
Allan Hirt: Exactly the thing is over time you are going to have to think about what you are going to do right now if you look in BOL it says if you are in Standard consider log shipping, if you are in Enterprise consider Availability Groups. If you considered obviously log shipping is not like database mirroring.
Greg Low: No that is what we had 2000 Standard, sorry not Standard I meant to say Enterprise. In 2005 that I thought one of the best things in the box was mirroring in Standard edition so that was an issue.
Allan Hirt: Yes you are going to have to really think about its feature and the one thing you are going to have to do as Greg said. Database mirroring isn’t going to go away tomorrow SQL Server generally two releases after it is announced that it will probably disappear. So you have got time depending on your deployment cycles but don’t be complacent either. Having said that, just because availability groups isn’t in Standard doesn’t mean that Microsoft won’t change their mind and put it in even in limited form in a future version. It is just that for SQL Server 2012 as of now, that is all there is. I have no insider information that they are going to make a change like that but if you look at the.
Greg Low: It is the track record.
Allan Hirt: Yes so if you look at the history they have done stuff like that.
Greg Low: Yes features come in the higher level editions and then slowly migrate their way down in general.
Allan Hirt: Yes I know a lot of people were up in arms about that but don’t get into panic just yet. In five years if they haven’t done that they are probably making a statement.
Greg Low: Look the other thing that we should mention in terms of Availbility Groups. Apart from what they consider as it being an enterprise version of mirroring now are the availability groups. The other one is around backups.
Allan Hirt: Actually there is another one besides backups that there are three things. Besides it having usable near real time replicas you can do certain maintenance functions like writing DBCCs and making backups all be it copy only full and key log backups no different.
Greg Low: Yes that’s worth highlighting too. The backup options, we can do backups on these secondaries however it is not full database backups so that is not differential backups. We can do copy backups which are like a full but don’t change where we are up to in the logging and we can also do. The interesting one is the idea that you can do transaction log backups on the secondaries.
Allan Hirt: Right and you won’t break the LSN chain.
Greg Low: Yes so there is a single log chain across all of the backups that are involved, no matter which server.
Allan Hirt: The only real challenge we have is because backups are considered local. MSDB information for backup history is not replicated around so you need to make sure that you are copying the backups to a central location and if you have to restore that you can have all the pieces in the puzzle in the same place.
Greg Low: It is one of the challenges, the thing to think about is I think if you are doing backups on a variety of servers when you go to restore you need to be thinking long and hard about how am I going to find all those pieces that need to go into the one spot to be able to restore. I noticed also they have helped you with that with the Restore Wizard now within Management Studio as well.
Allan Hirt: Yes, Microsoft have done some really nice things. You know like the Restore Wizard. For me that is just really, really terrific. Even the average DBA can get some benefit. I have written scripts to that stuff in the past to interrogate my data history. Even if it is gui you know version.
Greg Low: So if I want to restore to 9:59 exactly which backup files do I need? Yes that is gold having that.
Allan Hirt: One thing to mention that another significant change from database mirroring is that we can have more than one database in Availability Groups.
Greg Low: Yes.
Allan Hirt: Which is actually pretty significant, so how many times have you been getting at a point in the application and it doesn’t have a single database. It has got two maybe three databases and the question then becomes then how do I get these over to a DR or a HA solution at the same time? Well with Availabilty Groups assuming there is databases all live in the same instance, primary databases, they can be in the same availability groups. The caveat here is they are not kept in sync either. The synchronizing mechanisms are at the individual database level but they can failover at the same time.
Greg Low: Yes this is a significant change, mirroring was completely unsuitable for multi database applications an app that hits multiple databases. I have often gone into servers that have 15 databases mirrored and you will find 9 of them over here and 6 of them over there and so on. You simply don’t necessarily end up with them all on the same server at any one point in time. Now this gives us the ability I suppose where the name of the Availability Group comes from was able to give the group itself a name and make it be a group of databases SQL Server tries to fail them over as a set and also worth noting that listener. That’s right could not find part of that same naming and is associated with that set as well.
Allan Hirt: Let’s say you have two different databases, if you are going to certain other instances, you can do it synchronously there that you can’t have everything synchronous everywhere. Synchronous and asynchronous replica pairs within Availability Groups so let’s say you have for example three instances of SQL and A is your primary and B is your main HA and C is just going to be let’s just say a readable secondary. You can have that readable secondary be instance be asynchronous where your HA wants to be asynchronous. So that is also kind of nice as well.
Greg Low: So even though you can have four replicas, we can have up to two of them can be synchronous.
Allan Hirt: Exactly. So there is a lot of flexibility here, obviously we are over simplifying and make it easier to talk about. But there is a lot that goes into planning this, I don’t want to over simplify and make it seem like just easier than have it totally done and totally reliable. There is plenty of thinking that goes into it to make sure that happens.
Greg Low: The other one, just before we take a break, I suppose we need to touch on is licensing and the thing to keep in mind is that every one of these servers that we put into use as we said come a licensable server. Even though they might look at a piece of hardware and say hey I can make this usable and do my backups over here this could be a very expensive backup device.
Allan Hirt: Well, again depending who you talk to for licensing it may give us a different story. Because really what Greg and I are talking about, Microsoft have made a shift away from per processor licensing it is now core for Enterprise in SQL Server 2012. There is no per CAL licensing per Enterprise edition in SQL Server 2012. So if you have got a two processor machine that has 8 cores for each processor you are looking at 16 core licenses. Well single licensing is I think is $7500 roughly US or there about or maybe less than that I forget. It can add up pretty significantly because how many people are now buying two or four core systems because they just don’t exist.
This is where a lot of people where I sort of mentioned earlier, had a little pause with SQL Server is that the pricing model for some. For some people it may not be that significant different but for other people it could be a very big difference especially if you are going to take advantage of things like a readable replica where that server now absolutely needs to be license and you now paying essentially double even if it is the same hardware spec.
Greg Low: Yes it is something that does really need to be considered. Even though there is a nice city about being able to move a backup onto another server entirely or some of the DBCC checks and so, you really have to think long and hard about whether you can justify license for doing that.
Allan Hirt: Absolutely, you know it is probably the number one reason I am hearing a lot of grumblings from some of my clients that for certain solutions going forward they may actually consider non SQL Server platforms in the future. Now having said that I don’t see people making wholesale migrations that something SQL Server is currently in.
Greg Low: No.
Allan Hirt: Changing things but this is sort of something that everybody who is thinking about deploying SQL Server 2012 needs to really think about because let’s say in the grand scheme in deploying hardware is probably going to be your cheapest expense. Licensing support will be your biggest cost. So as your DBA that is not going to be something you are always going to be thinking about when you are thinking about future uses of SQL. The cost map it is going to cost somebody something and you are really going to need to balance your company and understand the implications to some degree of doing something like a readable replica because ultimately if they can’t use that they are going to look at you for some other solution and make something happen. But also what I am also seeing in terms of trends in the Enterprise space even though this change is significant and I don’t want to underscore the fact that it is. Most times, in fact every time I have done some calculations it is still cheaper than an equivalent Oracle solution. It is just that SQL Server is now closer in price to Oracle than it was before.
Greg Low: Yes indeed and as anyway as we say licensing is not something you can kind of ignore in this, this is something you will have to think about long and hard when you are working out which of the nodes is going to be used in which way. But look that is a good point to take a break and then we will come back talk about the failover clustering changes.
So welcome back, first of all is there life outside SQL Server?
Allan Hirt: Believe it or not there is. I have always really since third grade here in the US I am not sure if that maps to Australia, I have been playing Bass since third grade. Music tends to be a fairly significant; it is not really a hobby because I would certainly love to do it for real but you know just other things just conspired. I look at the day job which is doing the SQL Server stuff and everything that I do and being able to help me pay for my music habit.
Greg Low: Yes it is very often not the other way around. I have a shared love of playing bass and I played in bands in another life in the 1980s in particular. It is not something that I want to be using as having a dependency on income shall we say.
Allan Hirt: Well I mean it is all about luck. Just look at what we do, it is all about luck to be able to get to be able to strike out on our own sometime and sometime is it skill. It is all really that who you know. I have been pretty fortunate over the years. When I am home for example although I am on the road a lot I play in a local big band every few weeks when I am around. I sit and sub for other bass players when they are out and I am in the midst of recording a new album and I have done a few jazz albums in the past. This actually a big bad album that I have written and arranged. For me there is definitely life outside work and just a general rule, people need to have a life outside work if you let it consume you then you are really going to be an unhappy person later in life. You are never going to wish you had spent more time, gaining extra 1% performance out of your database queries.
Greg Low: Yes I think it always important everybody has a passion and those sorts of passions often continue long after the work finishes maybe.
Allan Hirt: Yes absolutely I have been lucky in the sense that I knew early on that music was going to stick with me but on the other hand I also knew that a steady paying income. I tried to find something that I liked and I do like what I do but I know far too many people and I see them all the time walking around constantly look at their phones. On the weekend and it is sort of the new pager right?
Greg Low: Yes.
Allan Hirt: That is one of the reasons that I find a way but everybody thinks I am nuts but I don’t do email on my phone. I hate texting so I really don’t even do that so there are a lot of things I don’t do to make sure I have a bit of work life separation. I think that is crucial for a lot of people too. Setting your boundaries with wherever you work and one of your kind with one of their kind. I am not saying.
Greg Low: Actually I should mention that I did manage to run into Allan and spent a bit of time with him in Melbourne again when you visited recently. I have to say you were one of the few people who weren’t constantly being pestered by his phone the whole time.
Allan Hirt: Yes you know it is funny. I have always had this weird mentality; I am not saying that I am a gadget guy. It is sort of in my DNA, my gadget guys stuff shows up in other ways. It is not necessarily like the latest and greatest phone. Like musical related stuff like headphones. Having said that I am always constantly amazed when I am out and about you see a couple or you see a family and they are sitting at the dinner table and they are not talking to each other and they are looking at their phone. I am thinking I do technology all day and sometimes and all day not just 9 to 5 but because what I do, it could be odd hours. When I am out and about and doing stuff the last thing I want to be doing is being tethered to email. I kind of want to be enjoying myself whether it is color facing or out with a friend or family member. Enjoying my surroundings if I am somewhere where I am not normally. Tokyo or somewhere in Europe or somewhere in Asia. I mean if I am worried about my Facebook status or who tweeted something I think my priorities are a little off.
Greg Low: Actually it is the best thing I have seen on that recently, I saw a photo where it had four they might have been in their late teens sitting at a table like a restaurant each one of them was on their own phone and none of the four of them were looking at each other or talking to each other and I though oh that is such a pity.
Allan Hirt: I actually saw that in real life, about a year or two ago I went to visit him and they were sitting in front of us. A little to the left assume where four friends, they looked like they came together. They were all looking down at their phone before the movie started, probably talking to each other not actually verbally talking to each other and that boggles my mind. If they are sitting right next to each other but they are not saying anything.
Greg Low: I suppose that is the modern equivalent is that people use to joke about is the fact that within their house people would be in different places and they would just instant message to each other to say heh it is time for dinner now. This is the modern equivalent, that you could be all sitting around a table and texting each other.
Allan Hirt: You know the problem I found with it, if you can understand is maybe get off my lawn old man syndrome. I think we have lost of touch of civility. Common courtesy, I can’t tell you how many times people looking down at their phones and walking and running into you and driving and not paying attention. All of a sudden you will see somebody slow down when they are driving. The disclaimer here in your state says you shouldn’t be using your phone, then don’t but having said that where you will be driving and you see somebody slow down. You have to slam on the brakes because the person in front of you and you need to pass them and you realize they answered a phone call or texting of something. I mean people have forgotten all sense of de quorum because they are attached to a device.
Greg Low: There you go, so look anyway so I suppose the thing is if people aren’t use to clustering much too we should make a clear distinction between Windows Clustering and SQL Server Clustering.
Allan Hirt: Absolutely, one more thing I am going to bring into a little bit later it is related to availability groups to clear something up as well. So I didn’t mention this earlier but if you think of a lot of things from life like a stack for example to be able to walk you have to be able to crawl first. To be able to install a clustered instance of SQL Server or SCI which is instance level protection. You need to have clustered Windows first.
Greg Low: I suppose I should get you to define SCIs.
Allan Hirt: That is a failover clustering instance.
Greg Low: Yes that’s good, just playing acronym police.
Allan Hirt: Yes absolutely. It is very easily to get into acronym hell with Microsoft very quickly. When you are deploying an SCI of SQL Server you can’t just install that way. You have to have clustered the underlying servers which are known as nodes first within Windows so it is a stack. You have your hardware, you install Windows, you cluster Windows and then you can cluster you instances of SQL. So if you think of it that way it is a much easier concept. Now a cluster instance of SQL Server needs some form of shared storage as Greg mentioned earlier. SQL Server 2012 we can now use things like SMB shares so your shared storage may just be a share somewhere but it is still the same concept as if you say used a SAN. You are having it just beyond a share somewhere.
Greg Low: So traditionally these have been fiber channel scuzzy or SANs or possibly i-scuzzy.
Allan Hirt: Right and in the good old days, there were DAS which is Direct Attached Storage.
Greg Low: Yes.
Allan Hirt: Now here is where I want to make a distinction with availability groups. So we mention that availability groups require a Windows Server Failover Cluster. This again doesn’t mean that you have a clustering instance of SQL Server to participate in it. Although clustered SCI can participate but having said that the biggest instance between SCI and AG in their use of Windows Failover Cluster is that and AG has no shared storage requirement what so ever. So you can have essentially a local install of SQL Server on what is a cluster node participating in a Windows Failover cluster. With no shared storage what so ever. That is how availability groups work.
Greg Low: Yes.
Allan Hirt: I want to make that point, that is one of the biggest points of confusion whenever I teach it or talk about it. I know when we talk about it; it is hard to see other points. Trust me when I say that with availability groups it is not the same exact way you would think of what we are going to be talking about now it is now clustered instances. That have a shared storage requirement, that is one of the benefits and reasons a lot of people now are looking at availability groups over failover clustering instances because it doesn’t have a shared storage requirement.
Greg Low: Yes because things like for example where we were talking. I suppose the main thing to think about is what we are protecting as well is in the case of availability groups or we are protecting a group of databases. In mirroring we are protecting a database in the case of failover clustering we are talking about protecting the server but the difference is in terms of the copies of the database in mirroring or in availability groups each node has its own copy of the database. In the case of clustering we are talking about a single copy of the data being shared my multiple servers because we are protecting against the server itself failing.
Allan Hirt: Right and so at that point it should be pointed out that your data could be a single point of failure if you lose your underlying storage whatever it may be.
Greg Low: Yes.
Allan Hirt: Having said that, that is also having a single copy is a benefit that Greg has pointed out. Which means that you have a copy of your data in every place? So if you have a three terabyte database. To cluster three terabytes you need to have it at every place left in availability group plus whatever backup storage costs could be potentially much higher from the purely space utilization from availability groups than it will be within traditional SCI. So it is all about tradeoffs.
Greg Low: So in terms of the requirements, so in the case of failover clustering for SQL Server, we need an underlying Windows Cluster. Are there any specific requirements around that cluster because gain keeping in mind the thing we are saying is that Windows SQL Server clustering doesn’t necessarily support Windows Server Clustering has supported over the years.
Allan Hirt: With really SQL 2012 that gap has the most part been closed. A good example is geographically dispersed cluster some people call it geo cluster that is not really the right term or multi-subnet failover cluster insert your favourite preferred term here. In Windows Server 2008 Microsoft introduced a way to do that natively. SQL for a long time through SQL Server 2008 R2 required a VLAN or virtual LAN setup to basically be able to do that. Whereas now in SQL Server 2012 we can natively use what is in Windows to span multiple sub nets for a cluster.
Traditionally things like that there were a gap now there isn’t really. Windows Server 2012 and 64 nodes support, Microsoft have come out with SQL Server yeah can support a 64 node cluster. It is not a hard coded number in SQL although I will tell you; you don’t want to support a 64 node failover cluster.
Greg Low: No indeed.
Allan Hirt: Even if it is possible it is something I would never ever recommend unless you are a total mass cist. I can’t really think of too many features in Windows Server Failover Cluster outside maybe the new cluster we are updating that SQL for the most part can take advantage of.
Greg Low: Yes closing that gap substantially, yes certainly stretch clusters was one of the core things. Also one of the things around, we have had over the last few versions things like majority node sets have appeared. This is where instead of having shared disk providing the quorum we have been able. Which was a single point of failure we have been able to move to just having a vote amongst the nodes and not having to have that. So there are different mechanisms we can use for quorum now days. I suppose what we should talk about briefly is disks and mount points.
Allan Hirt: Well right. So one thing that an SCI requires is shared storage. SQL Server required a drive letter and drive letter can only be used a single SCI. Just say you have a drive S which maps to disk carved out of your backend. Drive G and you have two instances of SQL, as one disk goes to another. Obviously there are only 27 letters in the alphabet.
Greg Low: So this has been a natural limit on the number instances that can be installed.
Allan Hirt: Absolutely so really starting with Windows Server 2008 after a while it has been a few good years. SQL Server also supported a mount points. What a mount point is, is you still have a name drive letters but you can make that essentially really small. What you then do, when you carve out your disk on your storage you don’t give it a drive letter but you essentially attach this main drive letters in programming. You are telling ok mister drive I am going to access you via drive in a folder named Allan but really underneath the covers you are a real drive but basically I am just pointing to you. That is a mount point, so you can have many more instances of SQL Server if you use mount points and it is pretty neat. It works really well, it is a traditional concept that a lot of DBAs deal with. I am seeing more and more usage at customer sites just things that make management much easier.
Greg Low: Yes so they are probably the main changes, we still have to have a working cluster. Probably the other big change was 2008 where we got the cluster validation tooling.
Allan Hirt: Right, Microsoft don’t talk about that, so we were talking about requirements for clustering. Microsoft changed a lot over the years. Back in the wonderful old days of clustering where things were much, much harder than they are overall are now. There were things which were more or less known as hardware compatibility list which became the Windows Server Hardware Compatibility List (HCL). The bottom line was that when you were going for clustering of SQL Server or exchange or whatever you were trying to do you entirely solution forms had to be in the list. Then say they released a new driver for your HA card or new firmware for your storage the vendor had to test it and recertify it with Microsoft and maintain their own list that you now had to check against.
That was a pain for everyone. We the implementer, things are to make sure that whatever the customer is buying had all that stuff and that they may not necessarily know. The vendor can maintain and test all this stuff. And the people just trying to sell you hardware and for everybody this was a pain.
Greg Low: Yes and mind you we should highlight that the reason to have that is if you wanted to be supportable.
Allan Hirt: Absolutely, it made supportability a nightmare because the first question that Microsoft support asks you is your solution on the HCL and you have to prove that. Microsoft realized this was a pain for them but for everyone else. In Windows Server 2008 they introduced something called Cluster Validation, there is a requirement for supportability but what it is, is before you even cluster Windows you run this process and in Windows Server 2008 R2 there is a Powershell command list. There is a gui in Windows Server 2008 as well. Basically it goes through components like disk and network. Even down to things like do these nodes have the same patches. I once was at a customer and ran that validation and each node had different set of patches. Basically it does a report and tells you I will remark you grade. There are some warnings but I think this configuration is suitable. Check the warning or raise a red flag that says no go, fix some stuff and rerun the validation and then hopefully it will be fine.
What this changes is that no longer do you have to go about this and no longer you have to worry about anything as long as it passes the validation. From a Microsoft support standpoint, you have a supportable cluster, that means you can technically have one server be Dell, one server be HP as long as the Windows configuration is the same you passed. Microsoft doesn’t care about the different number of processes or memory. It is just actual Windows configuration and things like that. This is a pretty significant change for not only the Windows folks but also for the DBAs as well.
SQL Server built into its setup a check to check that validation was good. If validation was bad it will pop up a message saying basically you can’t continue until you fix your cluster. That is significant because in the past there were a lot of folks that implemented a clustered instance of SQL didn’t test the end of line cluster that the guidelines that gave them.
If it failed over somewhere you couldn’t because of some underlying issue because be it storage networking whatever it was. We think that sometimes you had to tear down and build it right back up. So that meant down time. Now most of that stuff has gone away because most of your issues have been resolved up front. The reliability and easy deployment of you clusters has significantly gone up since the release of SQL Server 2008.
Greg Low: Yes so the hardware compatibility list was a big change in 2008, I suppose the other one that were very notable there was the change for anybody coming from an earlier version. Instead of installing into one node and then installing all the other nodes from that from 2008 we went around to each node individually to install.
Allan Hirt: Right, that is something at first I will say I was critical of. Let us give you an understanding why this happens. In SQL Server 2000 and 2005 you basically ran setup be it for an instance or for patching or service pack. You go out and then patch all your nodes, the problem was in those rare events where something happened on one of the remote nodes and the process didn’t complete, you may have been stuck with a broken cluster because you don’t know where it fails and how it fails. In getting back from that was never easy or always possible. So to change that what they have essentially done is say node, by node instance from install meaning you install what is known as a first node install and that gets your instance up and running and the cluster. On your additional nodes you run an add node to join that instance to that first node install that you did so you have a four node cluster and you have two instances of SQL that’s two first node installs and six add nodes. That seems like a lot of installs but in essence it is not. It is two really big ones, the first time you install on the other nodes. Just say you have installed SSMS you will install that again. After that it is essentially waiting down the buying areas for that particular instance on the boxes. So the installs are pretty quick.
What that provides you localized failure? If the install goes ary on the third node install when you try and join it to a cluster. That is fine you can isolate it and deal with it, even if it you need to reinstall it on the box and there is nothing on the box even if it is running in production while you are sorting out this issue. Versus having it by and potentially have them scratch your entire install. Why you use a little bit flexibility or you can use running a single install and you can run your trouble shooting abilities and piece of mind by doing it one by one.
Greg Low: Yes I gather from talking to the guys from that team, the reliability of installs when they made that change typically went through the roof.
Allan Hirt: Absolutely, I have seen this at customers. We have had failures but we could troubleshoot them and diagnose them. Not a lot of fuss because I am not worried about the other stuff, I am only worried about what is going on this particular node.
Greg Low: I gather the other thing that of course is getting harder, is the security guys are trying to tie things down more and more all the time and it actually makes it harder to install on one box and reach out and do things on one node anyway. I gather this was also a driving force that was making them realize that they literally had to do things node by node.
Allan Hirt: Maybe. I never heard that through them. It is a possibility. I think that may have been anecdotal customer feedback.
Greg Low: No I was actually hearing that directly from guys who was writing the code, they were telling me it is getting harder and harder from one node to sort and reach out and reliably install on another node anyway.
Allan Hirt: I actually agree with it, I just never heard it directly from them. Another issue sometimes you had was for example. That really came into play was I was able to access install bi-nodes, because sometimes if you have on say node one and you didn’t share it properly when you hit node two it could not do the install because it didn’t have access to see things. So this again sort of proves your point. If you do a local, you know.
Greg Low: Yes it was straightforward.
Allan Hirt: Another significant actually came from SQL 2012 is the new slip streaming methods which to me.
Greg Low: In fact we should define slip streaming and mention that as well.
Allan Hirt: Yes what slip streaming is, Windows has had it for a while where basically if you looked at your install media. You noticed that you can download let’s say Windows Server 2008 R2 with SP1 integrated in MS SDN in your license that is slip streaming. Where your major update is included with the package, in package I mean in the case the OS SQL Server.
SQL Server prior to this 2012 introduced a method in 2008 where you could basically run a couple of robo copy commands from batch files and integrate updates into SQL Server you can do from the Service Pack or Service Pack plus CU but you have to do it manually yourself and they didn’t ship it that way and every time let’s say things change like you are introduced a Service Pack. Not the most ideal scenario.
Greg Low: I suppose acronym police, I should point out CU is cumulative update.
Allan Hirt: Yes sorry about that. That is the release that SQL Server is roughly every two months that the update is update to a supported branch of SQL Server.
Greg Low: Yes so between service packs.
Allan Hirt: So what they did in 2012 and simplified the model and basically. It happens in two ways one, essentially the jist of it setup is shall we say where this is the best thing to describe it. If you run setup you can apply single checks that can be installed. What that means that setup by default, will go out on the internet and critical environment that is not what I want.
I can control that behavior in a line in a change file or I can default setup for the install media and say update equals false. I can also give you this, I just have a single line, it is called Update Source and I then say download SQL Server 2010 Service Pack 1 which was released a couple of months ago. We are recording this January 13 and I think it was released in October 2012. You can then take the executable, the downloaded executable and stick it on a folder or share somewhere. Point update source to it say C:\SP1, you set update source = C:\SP1 and then during an install it will detect oh heh I see Service Pack 1. I am going to install this as part of your install you don’t have to unpack anything, you don’t have to copy things into Integration Services yourself. What this means is that, if you have a generic folder, let’s just say called C:\SP, is it Service Pack 1 or Service Pack 2? Just swap out the executables and what it will do is you can have it dynamically updating installation media with the latest updates that you need in a very non threatening, easy to update manner. It is awesome!
It is one of my favourite features of 2012 actually but it is not something that will make a marketing set.
Greg Low: Other notable changes in clustering in 2012. I suppose the Failure Detection changes?
Allan Hirt: Yes, there are a couple of significant changes. The first is in the past there is a standard mechanism in Windows Failover Clustering that SQL Server leverages. Two processes called Is alive and Looks live. One is basically when a cluster says are you there and the other one is what we call something that is written by the application in this case SQL Server that checks to make sure that it is up and running. In the past it has basically been a simple look at select versus type of a thing where it is just running a simple query. It doesn’t say anything about the health of the databases or the health of the instances other than the fact that this is up and running and can connect in.
Greg Low: Yes the idea that you can do a SELECT @@ version almost nothing in the server needs to be running before you can do that but your user databases might not be available at all.
Allan Hirt: Exactly, so this is where people will customize that query and the answer is no. In SQL 2012 you can still customize the query what they have essentially done is that you have a stored procedure called SP Server Diagnostic which you run on even if you are not running clustering but this process now runs stored procedure and it basically gives you back a score. For all intensive purposes it says I am health or I am not healthy. Based on the failure condition level that is set and it will determine how sensitive it is to that score. If it detects a problem it will then obviously potentially trigger a Failover.
The intelligence around ‘Am I health?’ is much better in SQL 2012 and by the way the Availability Group feature also takes advantage of this. For its health detection as well, so it is not just Failover Clustering that uses this stuff it is also Availability Groups. Your instances in an unhealthy state and you have automatic Failover set in Availability Groups that could happen as a result of this.
Another actually significant improvements in SQL Server 2012 Failover clustering is the ability to have a local TempDB.
Greg Low: Yes.
Allan Hirt: So with TempDB, with TempDB we always have had to put it on shared storage but it is SQL Server scratch base so that is really an expensive use of SAN especially if you don’t have the IOPs to begin with. I don’t know about you Greg, but I have a lot of customers that use FusionIO or some local. How we performing solution for things like TempDB that maybe not be cheap but is much cheaper than SAN it gives them the IOPs they need.
Greg Low: Yes in fact it is usually the ones I see locally hasn’t been the price of the SANS in storage it is more that they want to chase the performance they could get from Direct Attached Storage in TempDB while using things like FusionIO or even local SSDs or something along those lines. They are chasing the performance of that, not necessarily the cost is not an issue but usually TempDBs are not that huge and they are usually chasing the performance so they can get that locally.
Allan Hirt: Oh 100% agree but the bigger issue, is that say you want to do a multi-site failover cluster standing subnets were you have to mirror that storage. Mirroring TempDB is really expensive so what they have done in SQL Server 2012 in an clustered instance is you now have the ability to have TempDB actually be created locally. It is the only database you can do that for. You can’t do it for any of the user databases or the system databases. So you can take advantage of these highly performing solutions and on the other side when it fails over it will just use the same solution from there. The only caveat is unfortunately during the setup the add node process it doesn’t create the directory structures. So you have to manually create the directory structures with the right permissions.
Outside of that it is a pretty fantastic solution that lets you leverage the hardware or things like FusionIO to give you that performance with TempDB with clustered instances fully supported out of the box.
Greg Low: Yes. No that is good, as I said it was a common requirement and people were saying the TempDB gets built every time anyway why does it have to be on shared storage.
Allan Hirt: Absolutely, this is one of things for all its +s and -s people have their criticisms of Microsoft. Microsoft really does listen, because a solution like this being officially supported comes directly from customers asking Microsoft for this. They wouldn’t do this otherwise, I can tell you this because there are so many people doing this inside the solution it was a no brainer for them to do it.
Greg Low: Yes. So we have got those, anything else you think notable with changes in clustering itself SQL Server?
Allan Hirt: The only other one I would say related to clustering is they have essentially a new throttle where you can tell SQL Server to throttle IO to speed up the recovery time in a failover.
Greg Low: Yes.
Allan Hirt: But that is not something I would recommend out of the box but that is brand new in SQL Server 2012 as well. But that it really about it, they pretty much kept clustering the same with these minor improvements in terms of number of improvements but the improvements they made are actually pretty significant in terms of their impact.
Greg Low: Yes that is great, listen so thanks for that Allan. Where will people come across you in future, or what is coming up for you?
Allan Hirt: Let’s see, I am doing some SQL Saturday in the States. I will be in Vancouver Canada and Mountain View. The Boston one here locally, maybe I will be at TechEd in May or June here in the States. I mean I am around I do webcasts, I can always be reached at my website http://www.sqlha.com
or at the blog sqlha.com/blog. I am on twitter as @sqlha, I am not that hard to find around the web.
Greg Low: I like your twitter avatar it has a nice bass guitar.
Allan Hirt: Yes actually the headstock to the T bass I own since 1988.
Greg Low: Excellent, well listen thank you so much for your time today Allan and we will talk to you again soon.
Allan Hirt: Great thanks for having me Greg and G’day everyone!