Investment firm Knight Capital made headlines in 2012 as it lost USD 400 mn at NYSE in just 45 minutes. Well, an investment firm could lose money in the market you say…possibly a wrong investment decision you say. Well, if you did catch this news item then you would remember that it wasn’t a bad investment decision and there was no super crash in the market which occurred in those three quarters of an hour. What happened, and this has been revealed just a few days back, is that there was a defect in the code deployment process. The code being deployed was missed in one of the eight servers. No one did a cross check to see if the deployment had happened. No one realised that the old code had not been removed. There were no automatic checks or procedures to catch this as well. So, this buggy deployment caused Knight Capital to lose USD 10.32mn a minute.
Seriously? Does this actually happen in this day and age?
Evidently it does. What happened with Knight Capital was something like this. The SEC report filed says “When Knight used the Power Peg code previously, as child orders were executed, a cumulative quantity function counted the number of shares of the parent order that had been executed. This feature instructed the code to stop routing child orders after the parent order had been filled completely. In 2003, Knight ceased using the Power Peg functionality. In 2005, Knight moved the tracking of cumulative shares function in the Power Peg code to an earlier point in the SMARS code sequence. Knight did not retest the Power Peg code after moving the cumulative quantity function to determine whether Power Peg would still function correctly if called. … During the deployment of the new code, however, one of Knight’s technicians did not copy the new code to one of the eight SMARS computer servers. Knight did not have a second technician review this deployment and no one at Knight realized that the Power Peg code had not been removed from the eighth server, nor the new RLP code added. Knight had no written procedures that required such a review.” Good Lord! …if you wish, you can read the SEC report here.
This perhaps is a type of risk, rather difficult to have perceived and identied at the start of an effort whether it were to be projectised (is that a neologism?) or not. However, given that one deals with known risks (which constitute the bulk) and unknown risks (for which you buy insurance), it is really worth the time spent in planning for risk and ensuring it goes into your overall project plan.
The Guide to the Project Management Body of Knowledge (PMBOK) from PMI elaborates this out rather well. This also is an area which has seen modification and expansion across various editions. I was part of the group writing this section for the 3rd edition of the book and we brought in some detailing. The 4th edition has changes as well in terms of addition of “Plan Risk Management” as a sub process area. The Risk Analysis sub process was already broken between Qualitative and Quantitative Analysis. The rest remains the same and the steps are still Identification, Quantification, Develop (response) and Control. But, clearly any amount of elaboration in Risk Management is still less.
Usually, during a project risk analysis, people do go through the motions and do not either quantify the risk or at least not take it forward into the project management plan to have it impact the project budget. Why is that necessary? Here is an anecdote from my past.
This happened in a Consulting /IT services company that I used to work for, and this was a Consulting /IT services project for a large global hotel chain. For obvious reasons, I need to mask the name of the services company and also the name of the hotel chain. Anyways, the consultants started flying into the head quarters of this hotel company from all over the US to work and were flying back every Thursday evening. There was not one consultant which would use the city that the hotel was head quartered at as their home base. Why? Because the IT services company did not have an office in that city and perhaps not in a 300 mile radius either. No one realised this up front. The symptom that got caught was when travel bills /expenses started coming up and started tearing the expenses budget away. Realisation dawned that no one had really accounted for this level of travel required. Oops! A change request was filed in for allocation of fresh budget amounts (from the client) and was of course summarily rejected. The project went down the tube, of course. The P&L loss was in a fair number of millions, write offs required and a few senior heads rolled.
Given that over 70% of IT projects fail globally, just might be time for project managers globally to think about this key area.