Managing Tech

The more I play with Mac OS X, the more I love it.  This is a surprising new adventure for me as I’ve never really given the Mac OS much thought until recently.  I’ve long been a Windows / Linux guy.  It’s been over 20 years that I’ve been creating solutions for Windows and once open source & the Internet took off, I switched to Linux for the server work that I was involved in.

Windows has a rich API but as more and more of the world moves to linux based systems (including bsd, solaris and mac os x), it’s becoming more and more of an island.  On the linux side, there is a lot to like but it’s always been a little rough around the edges.  The IDEs available are no match for Visual Studio and some utilities (like Xen) are a  pain to get working.  Still, for server work, Linux has a lot going for it.  First and foremost, the large selection of open source libraries.  Every now and then I lust over some of the very cool features in Solaris (like DTrace and ZFS) but the community was too small and many libraries did not have support for it.

Given the small installed base of Macs, especially on the server side, I never seriously considered the Mac OS X.   Then a funny thing happened.  I was investigating an idea fo the iPhone and was forced to get a Mac to do development.   And the more I used the Mac and learned about Mac OS X, the more I liked it.   This was a real surprise to me.

This morning, I’ve spent some time learning about what’s new in Snow Leopard (the latest version of OS X) and I’m impressed.  Apple is doing a lot of cool things.  First of all, OS X is sufficiently close to Linux that I could use it as my development enviroment.  And recently, Apple added DTrace (so no need to consider Solaris anymore).  In Snow Leopard, Apple is laying the foundation for easier multi-threading.  They’ve include something called Grand Central Dispatch and Blocks.  While all OSs support multi-threading, coding for it is a fairly advanced topic.   The best part is that Apple has released the source for Grand Central Dispatch as open source.  Let’s hope that this gets ported to Linux in short order.  Another cool technologies is OpenCL.  This allows you to create code that can run on the system GPU.  Traditionally, to harness the power available in today’s GPUs required chip specific coding.  OpenCL is generic and can even run on the CPU if it makes sense.

Now there are some things that I’m not sure I’m crazy about.  Objective-C for one.  No other OS has much support for this so anything created in Objective-C is OS X only.  I’m a cross-platform kind of guy so this is not a path that makes sense for me.  And some of the new features in Snow Leopard are OS X specific right now.  I like that Apple is willing to open source some of the key items though (like Grand Central Dispatch).  While my focus on software APIs, Mac OS X is a surprisingly nice client OS. I love how little extra software I’ve had to add. On a PC, the first thing I do on a new machine is load up a large number of extra software packages.  It’s easily several hours worth of work.  Other than added XCode (the Mac IDE), an out of the box Mac is pretty good to go. And Time Machine is very cool.  Even though I”ve had several hard disk crashes over the years, I don’t really do enough backups.  Time Machine is something every OS should have.  I’m surprised that Apple is the 1st company to get it right.

So today, I still use a PC as my main development enviroment.  But I’m at the point where I could easily see myself switching to the Mac as my primary computer.  Hats off to Apple for creating an OS that even a hard-code PC coder could love.

One of the hassles of being a geek is that you actually get your hands dirty with technology.  Usually that’s a good thing.  But sometimes it isn’t.  Yesterday at 5am, I started getting SMS messages from our mail server indicating that services were failing.   I had no idea why this was happening.

Now if I wasn’t a geek, I would not have a server to manage and whom ever provides our web/email would take care of the problem.  But I am a geek and want the flexibility to do things that don’t come in a standard solution from a web/email provider.  So that means I had to be the one to figure why I was getting SMSs at 5am.

It turns out an email account got compromised and spammers were hammering our server with SPAM they were sending.  Thank goodness my hosting company (www.powervps.com) has great support.  They were able to identify the problem and help me fix it.

Now I’m not really sure how spammers got the password for one of our email accounts.  We use IMAP and SMTP AUTH, both of which send the password unencrypted but only a sniffer at the ISP would be able to grab that.  And I assume most (if not all) ISPs protect against this.  Anyways, I decided that we needed to get all email clients using TLS and SSL.  Turns out TLS was already enabled on the server and all I had to do was add a SSL cert for IMAP.  So, hopefully we are protected now.

Along the way, someone suggested I should also look at SPF and domainkeys.  Now they really don’t have anything to do with the issue but they are good things to implement anyways.  I had already added SPF records to our DNS but was not familiar with domainkeys.  So I spent saturday morning tackling this.

Turns out domainkeys is not that hard to implement as I’m on a host with cpanel, which supports domainkeys.  The only complication is that I have our DNS hosted at a DNS hosting provider (so I can get redundancy).  So on our hosted server, I just used /usr/local/cpanel/bin/domain_keys_install account to generate a private/public key and make it available to exim.  And then I took the entry added to the DNS file in /var/named and added it to our external DNS provider.  So now, between the SPF and domainkeys, we should not have much, if any, email rejected.

All in all a satisfying couple of hours.

It turns out spammers had gotten one of the passwords for an email account and where using the server to send a ton of SPAM.   Now, if I wasn’t a geek, I would have just gotten our email from a service provider and just have to

The last couple of months I’ve been playing with a new hosting company called slicehost.  For those of us that are constantly in need for a server to just a few hours or a few days, slicehost is a god send. While I have been using  various virtual hosting companies for things like our website, most of the various development has been done using a server here in our office.  You don’t want to know how many times I’ve installed centos or windows over the years.   Well, slicehost makes that a thing of the past.    With slicehost, you have a web control panel and you can create and delete servers as required.  And since you pay a daily rate, it’s not a problem to create a server, use it for a few hours and then delete it.  And slicehost supports a fair number of linux distributions so if you want to test an app you wrote on a number of distros, its easy to do.

Another nice thing about slicehost is that you can upsize or downsize your slicehost at will.  So if you start with a small slice and find that the load is too heavy, its trivial to upgrade the slice to a large size.

Also, I should note that I find slicehost a good compliment for Amazon EC2.  I’m using EC2 for a service that we are working on.  EC2 is good for scalability and robustness.  But it also has a starting price of about $75/month.  Slicehost starts at $20/month.    So our serious stuff ends up on EC2 and we use slicehost for various research projects or ‘whenever you need a quick server’.  The two really are complimentary.

Having gotten rid of the time that we waste installed the OS (again and again), there is still another area for improvement:  that’s server configuration.   Once you have your server, you still need to install the software that you want to use.  For those of you used to cpanel (or other related solutions), this might be a bit of a surprise to you.   Both EC2 and Slicehost give you a plain vanilla server.  No apache or mysql.   For those of us in the software development area, this is perfect.  I do a lot of work in the area of VoIP and I don’t want apache on most of my servers.   Also, our subversion server don’t need mysql or any other software (except bugzilla).

But sometimes getting everything installed can take a long time.  We had one server configuration that took several hours to build.  That was because several items had to be compiled, patched and built.   The solution to this time waster is obviously scripting.

As we move to using these ‘cloud computing’ solutions, we are turning to more and more server scripting.  There has been some debate as to what we should use but for now we are starting with the basics.  All of our servers are setup using bash scripts.   This fits nicely with a 3rd-party monitoring solution that we use.   Bash scripting is not pretty but its amazing how you can do pretty much everything you can do with a normal programming language.  As a guy who has traditionally done everything in C/C++, I was quite surprised how powerful bash is.

So in the end, we are are now able to create servers on demand and using a script, have them setup and configured the way we want in minutes.  Life is good!

We’ve been hosting our svn repository in the Toronto office.  We did it because we thought it was the safest place to keep our important source code.  Yet, over the years, our office DSL connection has been out at least once ever couple of months.  That ends up being a real hassle for the team in India.  Also, the upload speed is pretty poor (768K) so big checkouts in India can take quite some time.

Yesterday, we had another outage and decided it was time to move svn to a hosted solution.  I had tried once before but the svn dump is over 15GB and the upload was going to take several days and interfere with checkin/checkouts during business hours.

So here I am again figuring out what to do.  This time I’m going to create separate dumps for each 1000 revisions.  Then I’ll upload those to the hosted server slowly when the indian office is closed.  Once all the dumps are on the hosted solution, I can load them all and finally move the team over to the new server.

Since I was already doing admin work, I decided to also cleanup the tree.  I’m going to do that by splitting the repository in sevearl.  One for each team/product.  You need to use a dump and svndumpfilter to do that.  Overall, not a hard process but during the planning I ran into a couple of issues.  First, the dump can not be made with –delta.  For small repositories, thats not a problem.  For larger ones, a full backup without deltas can be huge.  Also, the documentation for svndumpfilter only mentions how to include or exclude one folder. It took a bit of searching on the net to find out that you can include more than one directory after the include or exclude.

So now, we are finally ready to start moving forward with the move to a new hosted server.  Let the fun begin!!

One of the big challenges with outsourcing work is that the development team is very removed from the business people.  Sometimes when the businses staff are worked up about something, it’s hard for the techncial staff on the other side of the planet to really understand the issue.  Most of them have never met a customer face to f ace or how a customer uses the software.

Traditionally, this has been handled by having a senior technical person liason with the business staff.  While this is manageable, it does not generate the level of creativity that you have with direct interaction.  As a result, not all types of work are perfect to be outsources or if you are, you have to figure out a way to deal with this issue.

The outsourcing countries are trying to figure out how to solve this issue.  In India, there is a group of people that are fostering startups.  Startups, by their very nature, need to be tuned to what customers want.  It’s been a slow, tough road, but India is starting to see some decent startups.  It’s still early days though and most software companies are not ready for new product development with some assistance to be in-tune with what end users really want.  I figure it will be a decade before there is a large vibrant set of companies that are in-tune enough to create their own products that are strong enough to compete on the world markets (before people flame me, yes I know there are some companies today).

Today, I’m in review mode. I’ve been going through some of the team’s projects and providing feedback.  Maybe it takes a tech to belive it but software really is like art.  Even if it functionally works, it can be beautiful or utter crap.

I’m the kind of guy who is a perfectionist.  I like things to be structured properly, named probably, commented properly, etc, etc.  Considering how mature our industry is, you would think that there was great concensus on what is beautiful and that everyone was onboard with the ‘right way’ to code.

It amazes me how many young developers who seem to not really care about code quality.  It seems that most have to learn the hard way why it’s so important to really care about the quality of the code that they write.  Most are just focused on getting the functionality coded and then claiming success.  Very few actually create code the 1st time around that is really stellar.  Now having said that, almost all developers can be taught very quickly why its important.

Well, for a guy that has a lot of opinions on things and is curious by nature, it’s about time that I got a soapbox to share my thoughts with the world.