Professional Development/Deployment Strategies

ontheway

I know I probably shouldn't make assumptions but my target audience for this question is ideally professional software developers who work either solo or in teams on software projects.

Question: Let's say you are working on a web application using whatever technologies, how do you handle upgrades to a live application?

E.g. I would imagine that you would develop an application on a test server (a cloned environment of a production server), and then port it over to the production server upon release, am I right so far? Then the client would come along and start using the production release populating it with live user data, etc. Then what do you do when you find a bug? or if you want to add new functionality? Do you simply go back to your copy of the test server, modify it, test it, and overwrite what is on the production server?

Now in the above I make the assumptions that backups are made prior to these "overwriting" updates.

What I am really looking for is to create a good environment that promotes a quality, agile software development process, from IDE to release, specifically in the context of web application development.

Can anyone share stories of how they have seen it done successfully?

I understand that downtime is often inevitable during maintenance/update tasks, but how could this be minimized/eliminated?. I feel like I'm asking an age old question, but this seems to be something that I find very hard to get a clear answer on.

[edit]Also I read before that many large companies still perform development and testing using live or copies of live customer data. I assume that this is illegal in terms of the Data Protection Act[/edit]

Sparks

In the last two shops I worked in where this was done, there were two approaches:
1. Do the code changes on your local PC and then after 1800h, ftp everything to the live site and test, hoping no-one decided to work on the live server without telling you.

2. The job gets logged into the ticket tracking software by the client-facing team, checked to see if it's a real bug or approved feature request by the team manager, assigned to a developer who codes on his local pc using a copy of the live database for test data if needed, notes get entered into the dev team's wiki, the code changes get checked in to the VCS and then checked out onto the test server, the ticket goes to the tester who checks the work and then (if it passes), it's checked out onto the live site during that day's update window (or a custom window if it's been prearranged with the client) and final checks are done and the deployer closes out the ticket on our side and passes it over to the client for final signoff (the client had limited access to the ticketing system for this purpose).

Do I need to point out the obvious, that method 1 is braindead and caused enormous amounts of hassle for the schmuck who got assigned to deploy everything every evening (yours truly) and that the second is a far better way to do things?

Sparks

ontheway wrote: »

[edit]Also I read before that many large companies still perform development and testing using live or copies of live customer data. I assume that this is illegal in terms of the Data Protection Act[/edit]

No, it's not illegal. The developers are all under NDA and are employees of the company, so the issue never arises. Were the developers to unethically use the information they have access to under the NDA and employment contract, they'd be prosecuted individually, the company would have done its due diligence.

Now if the testing was ever done on a public server, that'd be a different kettle of fish.

amen

missing a few steps
1: developer is assigned task, writes and test on his on machine. Generates test plans (maybe part of spec) and check into code reposoitory
2: Build done on a build machine(along with other tasks) and released to internal test server
3: tested by internal test. Any bugs and the whole process restarts.
(Some regression testing also performed)
4: Once the testing complete and no issues release to clients Test site.
5: Client test and signs off. Any issues repeat above.
6: Once all issues identified it is applied to Live.

Normally the live update is on the same day/date each month.

Giblet

Typically we have a development site which is used by developers. After we're happy with our release, it moves up to our test environment. This is for internal testers. Any bugs brought up are then fixed in the dev server, and pushed to test. When testing has passed, we move up to User Acceptance Testing, which is usually a combination of other testers and the client using the site as a user would, adding real world content which may be pulled up to the live server on launch. Any additional expectations by the client and other problems are brought up and the cycle begins again. After everything has passed UAT, we move to production, usually on a Sunday or early week.

MOH

ontheway wrote: »

[edit]Also I read before that many large companies still perform development and testing using live or copies of live customer data. I assume that this is illegal in terms of the Data Protection Act[/edit]

Depending on the size of the comapny, and exactly what you're testing, you need to make sure that your tests never get confused with live: e.g. if you're testing part of an application which automatically generates letters which are printed in a different department, be very sure those letters don't accidentally get printed and picked up by someone who doesn't realise it's a test and posts them out. It's a good idea to sanitise some of the customer data such as the main address fields, just in case.

kenbrady

Tomcat Application

Build the war file on local test machine.

Use the Tomcat Web Application Manager to undeploy the old war and update the new war, takes about 30 seconds.

We do updates/bug fixes every Sunday, if we find a big bug during work hours we sometimes sneek an update in.

Sparks

We had UAT available amen, but oddly enough, only a small handful of clients ever used it - most felt that us doing the testing was what they were paying for.

Mind you, when they came back two weeks after a life deploy to say "you know, that's not what we wanted actually, even though we signe doff on the spec", it was a tad fustrating...

DublinWriter

ontheway wrote: »

I understand that downtime is often inevitable during maintenance/update tasks, but how could this be minimized/eliminated?. I feel like I'm asking an age old question, but this seems to be something that I find very hard to get a clear answer on.

The whole discipline is called 'Change Management' and is an art in itself, there have been loads of books published on it in the past.

A classic old-school approach I've seen used in the past is a three-stage approach where you have Development, Test and Live environments.

Anything goes in Dev. Once changes are ready for testing they go into the Test environment for UAT. Once past that stage, PM and/or manager sign-off is required to migrate amendments from test to live.

There should be contingency measures in place if you need to row back quickly to the previous live version.

Version Management is a subset of Change Management - the important thing to remember is that you should always retain the complete source code, DB scripts, libraries etc for each historic release.

Agile brings a few new challenges to the overall approach, especially in the test phase.

Sparks

What I'd like to see - and what Agile of all kinds appears to ignore or treat as a minor implementation hiccup - is how the heck you do CM on a database. Because it sounds so easy, but when you go to do it, it turns into a monster of a headache.

DublinWriter

Sparks wrote: »

What I'd like to see - and what Agile of all kinds appears to ignore or treat as a minor implementation hiccup - is how the heck you do CM on a database. Because it sounds so easy, but when you go to do it, it turns into a monster of a headache.

I was going to mention that. It's very easy to row-back from a code-base, but when you are making changes to what could be giga/terabytes of live data, it's a nightmare.

The recent issues with Boards are a classic example.

amen

I was going to mention that. It's very easy to row-back from a code-base, but when you are making changes to what could be giga/terabytes of live data, it's a nightmare.

The recent issues with Boards are a classic example.

test test and test again. If its a major upgrade take a recent backup copy of live and run your changes on that several times and simulate the load

All DB changes, SQL, Scheam, views etc should be in a CMS.
No reason not to be.

voxpop

The best advice I could give is to release small changes often. That way if something does go wrong its easy to either roll-back or patch.

As part of the release management here, we use continuum+maven2+artifactory for automated builds and release candidates. This allows you to automate much of the release process and means that you will reduce the human error aspect of your builds.

We have a dev environment for semi stable code/db changes (should already be tested by developer at this stage). Staging environment that is as close to production as possible and production. Any issues should be seen at the latest in the Staging environment. When a build is ready to be released it goes to staging for a time period, then, if no problems, production.

Without have a dedicated release management and test team, its very hard to get a full bullet proof release management process in place.

On our database side of things - all changes have to be scripted. Scripts applied are tracked with builds.

Eoin

Maybe read up on the change management section of the ITIL framework. Changes made to production systems tend to account for more P1 issues than hardware issues etc.

Civilian_Target

In our company, the UAT and Production environments are (in term of hardware) exact interchangeable clones. If you want to make something go live, load it in UAT, and then when you're ready, tell the load balancer to route all new production sessions to the UAT. And voilla, half an hour later, your UAT is now in production, you can take out your old production servers, and no-one notices a thing.

(But it is an expensive way of doing things)

ontheway

Thank you very much for all the comments above, that's a great help. It's been a while since I've heard the term "Change Management" and that summarizes my core concern at the moment.

With regards version control, are people using Git or is SVN pretty much the standard at the moment?

Can anyone recommend any good/recent books on Change Management, ideally in the context of web app projects. Google keeps bringing me to this book, but it looks a bit dated and the fact that it focuses so much on Microsoft tools hurts my head. All my favorite tools/platforms are open source based. I heard people mention the "Pragmatic Programmer" book before, but that was several years ago and I wonder if the concepts in that book are a bit dated now.

Many Thanks

stephenlane80

ontheway wrote: »

T
Can anyone recommend any good/recent books on Change Management, ideally in the context of web app projects.

Look at the change management process area from CMMI, it free:

http://www.sei.cmu.edu/reports/06tr008.pdf