Richard Waymire is the program management architect for Visual Studio Team Edition for Database Professionals. He’s been with Microsoft for eight and a half years working on SQL Server for the majority of his career. He’s a contributing editor to SQL Server magazine and the author of several books on SQL Server.
Greg Low: Introducing show 20 with Richard Waymire. Hi folks. Welcome. Sorry for delay in posting shows. Made lots of changes: moved from Brisbane to Melbourne and got married. New wife Mai is SQL Server DBA so keeps it in the family. Lots of emails asking if we’re doing any more shows, and yes we’re doing heaps of them. In fact the first of those is here now, so enjoy.
Our guest this evening is Richard Waymire. Richard is the program management architect for Visual Studio Team Edition for Database Professionals. He’s been with Microsoft for eight and a half years working on SQL Server for the majority of his career. He’s a contributing editor to SQL Server magazine and the author of several books on SQL Server. So, welcome Richard.
Richard Waymire: Hi.
Greg Low: I might ask how you got involved with SQL Server in the first place?
Richard Waymire: I was working on mainframe systems with DB2 and was needed to work on a side database project. This was back when we were doing NT3.1 and it took a large number of floppies to even get the operating system going. They needed somebody to get started on that, somebody with a database background, and I said sure I would give it a try. Promptly got hired away by another company that was getting much more serious about SQL Server and worked on 4.2 and 6.0 and early beta work with 6.5, then took off for a training and consulting firm. Liked it so much I joined Microsoft and started working on SQL Server 7.0.
Greg Low: How did you enjoy the training? I first came across you in the training area years ago. Did you enjoy doing training in those days?
Richard Waymire: Loved it. Loved talking to real customers, solving real problems. You see lots of aspects of the product. There’s a great irony in loving a product and working on its development team but you only get to see a tiny sliver of a product as big as SQL Server. Each person on the dev team typically knows a small area very, very well. In training, one of the luxuries you get is being forced to see the entire product as a whole.
Greg Low: One of the things I really enjoy about training is that people will ask you just anything and it’s amazing that every time I teach a class I feel like I learn a lot more things than the people that are in the class.
Richard Waymire: Absolutely. Sometimes they’re about windows interaction, sometimes networking, core SQL, or replication. Things that come up are all over the map and that’s what makes it exciting and fresh every time.
Greg Low: Yeah. And so, the move to Microsoft itself?
Richard Waymire: Well, that was a straightforward call for me. An opportunity to be part of the team of my absolute favorite product. Help design the software and help fix the issues that I’d wanted addressed, and work with a fantastic bunch of people that I’d met through my training connections. I’d come onto campus or meet people at events. I wanted to work with these folks.
Greg Low: What areas were you mostly involved with within the product group?
Richard Waymire: Core engine stuff, security, high availability, fail over clustering, worked an awful lot on security capabilities of 2000 and early 2005, then the management tool suite. Everything from command line SQL DMO/SMO stuff through the management tools SQL Server management studio, the profiling tools, query editing tools. Little bit all over. Even a stint in data warehousing working on DTS/SSIS and analysis services. Been all over the product.
Greg Low: What’s your feeling about where analysis services is positioned? Is it mainstreamed yet in the industry?
Richard Waymire: People thought that we would run into a brick wall with relational, that we’d get to a certain size and that the servers would be un-queriable. At this point I don’t think we’ve hit that wall. So analysis services can fill a great niche. If you need quick answers or have predefined queries that you need the answers to on a regular basis, or really complex calculations you can get nearly instantaneous answers from analysis services. Things like KPI’s where you just want indicators: “Am I in the red or green?” on my business and then I can drill into the details to figure out why the numbers are strung up that way. That’s a harder thing to do on a relational solution. Or I should say on a pure relational solution since analysis solutions can be backed up by a relational database as well. But people are now implementing 50 or 100 terabyte solutions and they’re no longer panicked. I remember it wasn’t long ago when people said we couldn’t do one terabyte.
Greg Low: Yes.
Richard Waymire: Today I bought myself a new USB 2.0 hard drive and its 500 gig and you can chain them together. There’s 1 terabyte USB drives now. The size of drives and stored data has gone up yet somehow the database products manage to scale just fine.
Greg Low: I have customers with single tables that are up in six and seven terabytes now. Sizes that I would not have even considered possible some time ago. As long as well designed and queried appropriately they perform quite well.
Richard Waymire: Don’t want to imply that everybody can do a ten terabyte table. It’s not a trivial exercise. But typically setting up analysis services will involve data cleansing, careful design to understand what measures they’re after so it’s a non-trivial exercise either way.
Greg Low: One thing I’ve been interested in with analysis services is that, previously, mainstream Microsoft products had not incorporated analysis services but that seems to be changing.
Richard Waymire: Absolutely. For instance you’ll even see team foundation server leveraging analysis services.
Greg Low: Is that a fundamental shift to the point where Microsoft starts shipping its own products based on analysis services that might make it more mainstream
Richard Waymire: There’s a recognition that when you have a product that stores an enormous amount of data about your business, well what are the things you want to do with that data? You want to report on it and figure out if things are going well or not. Are you developing software in an efficient manner? Are you getting the results you’re looking for? And with analysis services, if you know the kind of data you want to see and the source of that data then suddenly a product like analysis services makes a lot of sense. You combine that with reporting services and a relational engine and you get a pretty powerful solution that can be built in short order out of the box in an automated fashion for the most part.
Greg Low: From there you’ve been moved to the Data Dude team recently. How did that move take place?
Richard Waymire: I meandered over here. My friend was development manager over here and I talked to him about what he was working on and he got me very excited about the product and I joined the team. Took it from there. In about a year we took some very early prototypes and turned them into shipping code.
Greg Low: I recall being in a software design review in May, 2004 and it was still very much PowerPoint-ware at that stage.
Richard Waymire: Team got funded late summer/early fall of 2005. I joined over the holiday break of 2006 so came onboard fully in January, 2006. By TechEd we had a working community technical preview.
Greg Low: Thinking about it, it must have been 2005, the software design review, not 2004. So yeah, it was even tighter than that timeframe.
Richard Waymire: The team wasn’t really staffed until December 2005, early January, 2006. Then we cranked out the software as fast as we could. We tried a few things we hadn’t tried before in my experience at Microsoft. We did scrum and agile. Had daily scrum meetings with the team. Dev, test, program management were all reporting daily status, making adjustments and coordinating our feature set. We were working in four week sprints. The idea was to crank it out, make adjustments, and ship it to the community as quickly as possible for real customer feedback.
Greg Low: So you were using an MSF agile type process?
Richard Waymire: Absolutely. It’s gone very well for us and we’ve been very happy with the results. It’s an adjustment and a different way of building software from what a lot of us were used to. Our manager had done it before.
Greg Low: Given that your previous work hadn’t been like that, what was the driving force?
Richard Waymire: Funny the difference one person can make. Our product unit manager wanted to do it that way and felt it was best for rapid development. Given that it was a V1 product and we wanted fast turn around and lots of customer feedback, he drove that vision and motivated the group to try it out. Quite literally it was the evangelical approach of one guy to do it the way he wanted to do it.
Greg Low: There’s been talk in the blogosphere because Microsoft has been posting job ads for new developer oriented things for SQL Server. What’s interesting is in the job ads they were talking about end of the year/early next year completion times for projects that hadn’t started yet. Is this something we’re going to see more and more of?
Richard Waymire: I know we got a lot of attention from Microsoft management going “How did you do that?” How did we get such a quick turn around on the project? So we’re doing a post-mortem on what went well, poorly, and what can be done better next time. Trying to be introspective on what needs to change but also focused on what we did a good job at. Then we’re taking those best practices and sharing them on a wider scale.
Greg Low: What were the main things you did differently to other groups you’ve been in?
Richard Waymire: The biggest thing has been the rapidity of adjustments. Because we’re meeting daily, by the end of each week we knew what would be harder or easier than we thought, or what would require more integration than we thought. Able to coordinate those changes much more quickly. Didn’t have a couple of months lag if things weren’t on track. We knew within a day or two if things weren’t on track. Because you’re talking to each other every single day, every morning. You get a lot of interaction. When everybody’s reporting their status you might hear, “Oh you’re doing X? Oh, I need to get my part done.” Or a tester might say, “Oh you’re up to that now. I didn’t realize that was coming. I’ll make sure that the test environment is ready to go for you.” Or somebody would say, “Oh why are you working on that? I’m waiting for this other thing.” And you would say, “I didn’t know you were blocked, I’ll get right on that.” All these interactions are happening so frequently you don’t get delays like on other software projects. Also just realizing it’s ok to re-open questions about feature X if it turns out to be a 60 hour development project, not 18 hours. You get 12 hours into it and realize you’re nowhere near done so you ask about the feature’s importance. Because you’re talking to customers, you can ask them “What do you think about dropping this, or adding that?” We got fantastic amounts of direct customer feedback on this project.
Greg Low: Next thing is to recap the features you got into the product and done. What would you describe as the main contribution of the product?
Richard Waymire: Biggest thing is redefinition of where the truth is of your production schema. The folks that are adopting the product earlier are telling us that this is something they’ve wanted for a long time but hasn’t been possible without a lot of manual work on their part. Now they will have a historical reference of what schema and what versions of that schema are in production. And they have an easy way to replicate a test environment of that production solution and then write tests and figure out when they make changes if it will break existing code. All of that in the same environment that they write the rest of their software. We’re also pulling the DBA’s in as well and so they’re getting involved earlier in the software development process for SQL. The developers need them to get the initial project started because they won’t have the rights to see the production system to get a baseline built because they know they’re going to have to deploy these systems out since the DBA’s are integrating directly into their project solutions. Much closer interaction being enabled.
Greg Low: With the environments you’ve seen. How many will use it stand-alone and how many more will use it in conjunction with TFS as part of a wider deployment of VSTS?
Richard Waymire: The value of the project is in integration with things like Team Foundation Server. Let’s imagine I’ve got my schema project and I’ve written my data generation plan to generate some test data for a schema, and then I create a unit test so that I can test my stored procedures and functions. If I have Team Test then I can integrate that with other tests I have for my C# or VB or C++ projects. If I have Team Developer I can combine it with all other code quality checks I have for my source code. If I have Team Foundation Server and I run a unit test that fails I can right click on the failed test and log a bug right in Visual Studio. I can insert and cut and paste all the stuff right there from my failure and track the work flow without a new tool. It’s all in the same environment with me.
Greg Low: How many groups internally are using it, or is it mostly just external?
Richard Waymire: We’re getting very aggressive at migrating the company into Team Foundation Server. Changing over our systems, keeping our source there, migrating our bug tracking from our old product studio databases into Team Foundation Server so a lot of groups are making that move. Obviously the Team Suite Products are doing it, Visual Studios, several other groups in Microsoft. We’re getting good internal adoption. Proving we can scale it up and meet the needs of a large business.
Greg Low: What about the DB Pro side of things in terms of internal groups?
Richard Waymire: I’m not sure I understood the question?
Greg Low: Sorry, the Database Professional Edition. Are there many internal folk that the new product helps?
Richard Waymire: We’re getting good internal adoption from people who have casual projects and want to version it and keep up information without having to do a full database restore. Also IT groups using the testing feature so if they want to test out production data but if they’re not allowed to see the production data for legal or privacy reasons now they have generation tools that will give them samples. Other software groups depend on SQL Server…can’t name names. There are other groups that ship with SQL Server underneath the covers. A lot of teams at Microsoft are now adopting DB Pro to keep track of their schemas and build their test environment. So there’s fantastic internal adoption as well as external.
Greg Low: One key thing in the product as well as the first level of refactoring so with a rename/refactor, what’s your feeling on where that might head from here? That looks like a beginning start at refactoring.
Richard Waymire: Absolutely. We know that there are a number of different kinds of refactoring and you can imagine that’s a key value-ad of the product that we’ll be focusing on in our next release. You can imagine the ability to refactor across your entire solution. So if I had a WinForms application in C# and it made some database calls and I had some of those calls in strings, as well as having the database project as part of the solution, now let’s say I want to rename a call on a table. You can imagine how slick it would be to refactor that and have it find all the references not only in the database, stored procedures, and triggers, but also in your application code. It will give you options to update everything simultaneously and run unit tests to verify your rename didn’t break anything. There are other types of refactoring you can imagine but a lot of those are on our radar for V2. Unclear exactly how much we’ll get done, but it is a key focus area for us.
Greg Low: With the V2 are you again aiming for a fairly short iteration?
Richard Waymire: Don’t know if our schedule’s been announced. We’re definitely having a very short turn-around for our first service pack. It will be out in the first half of year. Then you’ll also see a significant number of power toys released frequently as add-ons to V1 that you can play with at your own risk.
Greg Low: Ah, tell us about power toys. That’s one I haven’t heard discussed.
Richard Waymire: Just things that are little capabilities in the product that we would have liked to add or the developer had an idea for but we didn’t have enough time, or couldn’t adequately test. For those we’ll get them in shape enough and then release onto the Web and then you can download them and run them on top of DB Pro and they’ll give you extra capabilities and features you might not have had otherwise. So as a developer or program manager or tester comes up with an idea for a feature ad-on that they couldn’t get into the main product, we’ll release those. You can think of those as the directions we might be headed as well. We plan on releasing those as regularly as possible to keep people seeing value adds to the product and to enhance their experience.
Greg Low: Now that you had a shortened initial cycle I’m wondering conceptually if continuous short cycles is the overall direction?
Richard Waymire: Unclear what’s going to happen. The short iteration of V1 was to build the V1 product, get excited, get it in marketplace but then we’re also part of Team Suite. So one of the very first things customers ask us is how come we’re not automatically set up when I set up Team Suite? So for customers who get us through Team Suite, they will want better integration across the board, to roll out and be part of the Team Suite set-up. When the next Team Suite release comes out they will expect us to be part of that. We have to balance that we are one of the core components of Team Suite and a rapid release schedule and I don’t know how that will go and how fast that turnaround will be but our goal is frequent updates.
Greg Low: Who is the Richard apart from the one we see at Microsoft?
Richard Waymire: I like to go hiking. I’m just looking at the pictures I’ve got from hiking up in the mountains around here. One of the things we’re fortunate to have here in Redmond, Washington is lots of mountains. In the summer and fall you can find fantastic hiking up by mountain lakes where you’re the only person for 20 miles. I love getting out there and doing that. I like reading. Play the occasional computer game but sometimes it’s nice to get away from the computers. Spending time with my kids, I like fishing with them.
Greg Low: The fishing there, is that fresh water fishing in the mountain streams?
Richard Waymire: More lake fishing. I like going after trout, mostly. Rainbow trout is a favorite. Last year a friend introduced me to fly fishing and I could get addicted to that.
Greg Low: Ahhh…So where do you do fly fishing there?
Richard Waymire: Standing in a river with your waders on and just sort of getting the technique down of landing the fly on the water so the fish think it’s an insect that’s landed instead of your fishing line. It’s so much fun getting a bite and reeling the fish in. Just catch and release, it’s just the enjoyment of being out there. It’s 50 percent just enjoying nature and 50 percent the fun of catching the fish.
Greg Low: Fishing is a passion of mine as well. In fact our local DP, Chuck Sterling, you know Chuck as well I’m sure, one of the things I love on his blog is his fishing report.
Richard Waymire: I didn’t realize that, I thought he was into scuba diving.
Greg Low: He’s very much into scuba diving as well, but since he’s been at the Gold Coast I’ve seen more where he’s quite involved with fishing. On his blog he intersperses technical content with fishing content.
Richard Waymire: Sounds like the next time I’m in Auz, we should have a fishing trip.
Greg Low: Indeed. I just moved to Melbourne myself and I’m still finding fishing places. One of the places I loved just North of Brisbane is a beach called Castaways. It’s an unpatrolled beach with few people. The thing I love is dawn/pre-dawn and the amount of life in the water. A few bait fish, then bigger ones, and sometimes dolphins. It’s an amazing sight. People who arrive late in the morning probably look and say, “Yeah it’s nice.” But this entire show occurred early in the morning that they did not see.
Richard Waymire: I know exactly. I get that a lot. Not just the fish but all the wildlife. I tend to do that in the mountains. We’ve got ocean here, but it’s the North Pacific…colder.
Greg Low: A few more things on the product. The TSQL editor, what’s your feeling on that? There’s the endless discussion about IntelliSense.
Richard Waymire: A couple of features you’ve obviously got to do for V2 and that’s one of them. Then becomes a question of how much IntelliSense? There are some other query technologies coming out of Microsoft soon. These new query language techniques always start with the “from” clause, not the “select” clause and the reason is: how much easier is IntelliSense if you already know what tables you’re selecting from? If you just say “select” and what do you expect next? It’s an interesting problem. Certainly if you’ve already got the from clause there you can start figuring it out, but if you want write good SQL that’s not overly complicated it gets cumbersome. So we’ve got challenges but one V1 guidepost is to give people an SQL experience that is familiar if they’ve been used to, say, C#. One of our goals is to bring the SQL editing experience up to par with C# or VB experience. You’ll expect code snippets, color coding of key words, IntelliSense, prompting for syntax, and instant syntax feedback. Those are key areas to look at. It’s getting to the point where the query editor in Visual Studio is as good as the query editor from the SQL Server team so you aren’t forced to switch tools for simple query results. That was one of the V1 goals. We got close and I think we will continue rapid improvement and get to a superior SQL editing experience. So if you’re working on our product and you need to make a live connection via server explorer you have a fantastic query tool, and when you’re working offline you have a fantastic editing experience on those offline SQL files. Definitely a V2 goal.
Greg Low: Another one was schema compare and how far you got in that? Lots of objects in SQL Server 2005, so a fairly big challenges in making that work?
Richard Waymire: Biggest challenge in schema compare is that things are just hard to figure out from the SQL system table if you’re not SQL Server. We’ve had our challenges in figuring out what do things mean and when they come out of the systems table, how do they work? What’s an efficient way to reverse engineer a schema? Sometimes it’s really hard to know what matters. You have set options for example and you worry about all of them, but the SQL systems table can only store the state of a couple of them. Most only change run time performance, not schema. Things like that may not be well documented. We have been asked to schema compare a pre-defined label in Team Foundation server of a previous version of the project.
Greg Low: Once you’ve got different versions of the same project you want to compare them.
Richard Waymire: Exactly. The schema compare tools will probably be trying to figure out how to go back in time. Key questions this project should solve: tell me about the history of this project/what version I’m on/previous versions. Behind the scenes will be fun tricks, how to use synch to a previous version in the background and compare against it. That’s why we’re here. We have really good coverage of SQL 2005 and then a new release of SQL on the horizon…
Greg Low: When you were doing it you had a version that was already out to work with. You now have a moving target to try and keep the product up to date with upcoming versions as well.
Richard Waymire: The SQL team is excited about our product and they’re helping us keep up. We have source access, great contacts over there; they want to see us succeed. If people are doing a better job of managing their SQL Servers they’re going to think highly of the product. People who use DB Pro and feel like they understand their schemas, versioning, and a great deployment story, a great test experience, will like SQL more and maybe choose it over Oracle, DB2, etc. and choose it for their next project. What’s good for us is good for them and vice versa.
Greg Low: What about the data compare? What’s its main role? We’ve used it more for look-up table data and more part of the schema than user data.
Richard Waymire: Yeah. I think that’s its most appropriate use. Ironically, we’ve haven’t gotten much feedback on what people want for a data compare product. Think about replicated catalogue systems, it would be a good way to get integrity checks. If you have ten SQL servers with, theoretically, the same data, you can check that. So depending on which website you hit in the web array you don’t get a different price on your product. There are some uses like that but we haven’t gotten a lot of feedback, other than a lot of people using it for look-up table use.
Greg Low: I tend to see that data as separate from user data even though it’s data in the database. It tends to be almost more schema in nature.
Richard Waymire: Yes, at what point should we be storing the data for look-up tables as part of your DB project? Does that make any sense? Weird line that some data is almost meta-data. We’ve been taking a hard line that we store schema and meta-data, but not data. But some data really is meta-data and we’re trying to figure out how that makes sense.
Greg Low: We used to have a configuration application that applied schema changes, but to me it’s the data that we would have also modified using that same tool. Unrelated to user data, we would have added new look-up values, maybe our own system tables for example and those sort of entries. Simple things like last invoice number issued that might have needed to be stored somewhere but the idea that that row needed to be in a table somewhere had more to do with the schema than user data.
Richard Waymire: It’s an open question. We’re very early on the V2 planning.
Greg Low: Unit testing was nice to see in the product. In the DB area people have not been very big on unit testing and I think that’s because it’s been so hard to do. Mostly because in terms of unit tests every time you want the system in a known state and also typically you want a large amount of data. The two things don’t go together well. In SQL Server 2005 I like database snapshots. Will there be more automation of using snapshots for testing?
Richard Waymire: We haven’t thought hard about unit testing. Post it on the forums, that idea with snapshots. We’re working with the Visual Studio Team test group to figure out integration points. Remember Ewan Garten? He’s on Team Test group so we’ve got a strong friend over there. We’re trying to keep the baseline established. Should we allow database restores? V1 idea is to deploy your schema, data generation, and then run your unit test but there are other approaches. There are other kinds of unit tests you can integrate with outside of the database. A key selling point of the product is having ability to see if the changes you’re making are going to break what you’ve got. Clearly there’s more to improve there: debug failures, diagnose them.
Greg Low: Data generation, I like that fact that it was very extensible. June of last year, extensibility was what I was pushing. In early product phase, it’s important to have extensibility points built in all over. The product might not do what people are after and the ability to extend it is critical. I like the ability to build data generators. Trying to get a community effort to build lots of standard data generators and having a repository to go to find those. Nice to have simple things: state name generators, local phone number generators, good to have a repository for those types of things. Putting that up on the site soon. I really liked that you could add your own assertions in the unit testing. Really nice late addition.
Richard Waymire: One publically committed feature is to support other products besides SQL Server. We’ll have to approach it by extensibility.
Greg Low: How tied do you see the product to 2005, support for 2000, or other databases?
Richard Waymire: For foreseeable future support for SQL Server is what we’ll focus on. We’ll try to enable third parties like Oracle or other databases to plug into our framework and give you functionality. But SQL Server is our first customer because, well, we’re Microsoft.
Greg Low: Noting the approach they took with ADO version two, they have a meta-data layer where you can pull back schema details but what I liked is it doesn’t have a hard coded list of the different object types. The object type list you retrieve from the provider as well. For example, in the current product you might have nodes in the solution explorer that say “stored procs” but in Oracle you might want a node that says “sequences” but that may have no purpose in another database product.
Richard Waymire: Clearly we’ll have to have something similar conceptually. Even between 2000 and 2005 you had to have different project types, different lists of objects, and therefore different dependency tracking. So to extend to other database products it’s even more the case.
Greg Low: That really great. Finishing up then, what’s coming up in your world? Any places we’ll see you speaking? Things happening?
Richard Waymire: Just got back from a month out of the office. Went on a road show to New York City, North Carolina and did some launch events for DB Pro. Took some vacation, my mom ended up needing surgery so I was out longer than expected. But I know there are a number of shows in the U.S. that I will be participating in. I know in spring as we do our V2 planning we’ll be touching as many customers as possible to figure out if the ideas we’re coming up with people would buy and that we’re solving real problems in the real world. Definitely getting out but don’t have any specific travel plans at this point.
Greg Low: That’s great. Thank you for your time, Richard, just outstanding.
Richard Waymire: Fantastic talking to you as always, Greg.
Greg Low: Thank you.