Continued from Part III:
Prototyping Output - As agreed, the architecture team used their additional thirty calendar days to finish proofing out ideas, components, and integrations. The Senior Architect was happy with the results and believed the output would be quite sufficient to guide the teams for the future.
A challenge lay in the fact that prototype results proposed/suggested a bit more change than was originally projected. For example, upon further investigation the teams observed that most of their potential customer base used a particular operating system and same-vendor programming language other than what they had chosen for development, not to mention the fact that it appeared most customer companies, either purposefully or accidentally, were two to four major versions behind 'current' published versions of nearly everything. As a result, the architecture team proposed changing all development, test and end-state architecture environments accordingly in order to insure they were most compatible with, and appealing to, those customers valuing single-vendor lock-in in the interest of technology continuity (not to mention they represented the majority of potential revenue for BioRecCo).
Second to this, the prototype results suggested that integrating the hardware and software components would likely take more time than originally estimated due to several factors including, but not limited to, a few newly proofed out components, and the OS/programming language change implications to the original architecture proposal. It would take time to get corporate security approval to swap out the environments, get the programmers trained by the vendor on the new language and standards, and re-write the functional specification documents in a manner suitable for the programmers who would later read them.
All of these items in consideration, the Architect proposed to the Project Manager the most current known level of effort suggests a development window of six months from 'today', and approximately two months for testing thereafter. All in all, the Architect believes all of this work will be completed in eight months, but that he could pull in the window if the development staff size were increased to include an additional team of eight high-end consultants to pick up the slack that these changes will incur. Substantial conversation regarding money did not occur, though the Architect made it clear if the company wanted this job done right, he simply need to find 'the best' and get the job done.
Leadership Activities - With the help of the Architect, the Project Manager crafted a presentation for the Project Sponsor (aka Business Sponsor) in order to bring her up to speed on progress, lay out the definition of success for the project, and propose the required need for a change request 'requiring' a new timeline and more budget to meet the revised objectives. The Project Sponsor, caught by surprise, felt blind-sided, angry and disappointed in the Project Manager because of the large change request presented her. She vented, communicated the importance of not failing on this project, and reiterated to the Project Manager that this behavior was absolutely unacceptable.
Feeling defensive, the Project Manager returned the volley, communicated how incredibly complex the project really was and the fact that they had employed 'only the best' technologists in the industry to meet these goals. The PM further communicated that everyone had been working far more hours than they were billing for, had deferred vacations, and that if this team felt they needed more there was really no one who knew more about it that this team. She further communicated the number of tests written, documents delivered, prototype decisions, and number of requirements in the system (functional and non-functional merged together) to evidence money well spent to date.
The Project Sponsor followed-up with a single question: "When can I see something?" The Project Manager responded with: "Most of the documents are still pretty technical and the prototype work is really at the technical infrastructure level accessible mostly to the technical folks, so we'd like to wait until we have something you can see and touch. Right now, we're planning to show you the system when we publish the first draft in six months and we think you will really like it." The meeting ended and the Project Sponsor left feeling badly about the project and wondering if she had attached her career to a lead weight.
The Re-based Project - November 16, 2000 - The Project Manager was granted the revised timeline, environment structure swap, training, and staff augmentation on the conditions that all of these expenditures would be outlined in detail for the Project Sponsor to understand and approve prior to acquisition attempts.
Projection: Eight months from 'today' and with 66 existing people and the eight new 'high-end' consultants, the project would be production ready. The revised blended rate for all staff increased to $105/hr (because some other consultant rates were negotiated down) and the new infrastructure purchase costs came out to a negotiated low of US$2.5 million (which was new since the previous work was planned to occur on existing in-house and open-source solutions where possible). The Architect was pleased everything was coming together; the Project Manger was pleased to have this bullet already added to the resume; the consultants were pleased to know their contracts would be for another eight months (though likely longer since customer's always change their minds); and the employees were astounded at the ever-growing juggernaut upon which they were at first willing participants, but now began to question.
High-ceremony status meetings occurred with all stakeholders, participants, on-lookers, food and music on a regular monthly basis. The presentation, always delivered by the Project Manager and backed-up by the Architect and/or Project Sponsor, without fail showed:
- A Gantt chart;
- the Top 3 Risks and Issues;
- a Picture of the most current application architecture;
- a List of documents and requirements completed;
- the Number of screens created and;
- the Number of tests written and successfully executed.
Anytime actuals did not match planned estimates, one of the top three issues would end-up being 'Corrective actions necessary to bring actuals in line with planned LOE' and the Project Sponsor would then privately check with the Project Manager that all was well (to which the response would ordinarily be 'the Architect will look into it and make sure we get on-schedule').
February 16, 2001 - Three months into the re-based project and during the monthly status meeting, someone at the meeting asked if they could actually see the system work. The Project Manager immediately said 'Yes' and pulled up a PowerPoint presentation showing screen mock-ups the HF UI group had put together before the project was re-based. The Architect barely had a chance to process the question and resultant action before that same someone in the audience asked if they could actually see the application rather than seeing some documents that talk about the application. The Architect stood up from the back of the room and suggested that they did not have the application on an environment they could access and show during this particular meeting, but that he would arrange it and have a demo prepared for the following meeting in March. Immediately thereafter, the Project Manager volunteered to send out the PowerPoint deck they had just viewed so that everyone could see what the 'system looked like'.
Stream 1 - Development - To date, builds were happening as planned; wherever the specifications were not current, developers would check with the Architect for a direction; HF UI screen creation work happened at a high rate of speed and were scheduled to be 'hooked up' later in the project; and for all practical purposes, the hours actuals were within +/-25% of planned LOE at basically all times. All data points suggested a positive future and the Project Manager and Sponsor were pleased with what they knew. Privately, the Project Sponsor began talking with a CTO friend elsewhere in the parent company somewhat hoping that her discomfort could be allayed by something she yet did not understand.
A mild speed bump occurred over and over again during the builds where developers would have trouble building some of the 'stubborn' code elements, but no one really took the time to understand why until somewhere in month four of eight when there occurred disagreement over how to fix a couple of high severity defects. It was at that point that someone made the observation that there were still basically three operating systems, two different databases, and some language version variations collectively between the developers in sum. It later turned into a bit of an unhealthy conversation pitting off-site consultants against on-site full-time employees with regards to vendor-lock-in versus platform/technology agnostic development practices. This discussion further revealed that some developers were stubbing interfaces and others weren't, while the testers were actually writing tests in an environment with the wrong end-state integration points on software loads nearly three weeks behind 'current' at all times...and the hits just kept coming.
The Architect eventually called an 'all technical hands' conference call to lay down the ground rules for all people to work together. The results of the meeting required 'all people' to have their computers reconfigured to a particular development environment specification. A Senior Partner of one of the consulting firms later called the Architect to suggest alternatives and an undisclosed agreement was reached. Thereafter, the Architect brought the Project Manager up to speed, mentioned there had been a technical problem which required attention and it had since been resolved. Three days were lost for some of the off-site consultants who had to have their computers reconfigured and get their development environments rebuilt. Four of the consultants from a yet different consulting company somehow got caught in a Corporate Security dragnet and were not given permission to build their development environments locally (on their own machines) for another two days until all feathers were in place.
Stream 2 - Software Testing Tests were being written based upon the original business requirement documents. Where there was a delta between what they were reading and what they were hearing at lunch and at the water cooler, they err'd on the side of sticking to the business requirements document until they knew otherwise. They were taking a new load nearly every two to three weeks which the test manager simply felt was too quick but had no choice in the matter since he wanted to stay 'current'. The test group was required to send in daily status reports which eventually trickled up to the Architect, but no one noticed the testers had failed to note all component changes that had occurred during the final thirty days of prototyping.
Stream 3 - Leadership Meetings were happening in a consistent mechanized fashion. All of the math and messages sounded good and all were pleased. The Project Manager and Architect communicated things were all 'normal' for the typical software project and that due to the complexity of this project, it would be considered 'ordinary' to have bumps here and there. The Project Sponsor reduced her conversation time with the CTO because everything seemed to be heading along appropriately. The Project Sponsor did however ask if the milestones suggested more parts of the system were being delivered and were tangible to her...to which the Architect responded that "later milestones would be attributable to a tangible system; but today they were largely associated with lower-level technical deliveries".
May 16, 2001 - Now six months into the project, work has continued be reported 'behind schedule' at the individual status level. After a handful of Project Management initiated meetings telling the teams they are failing to do work as planned and need to correct (also known as 'flogging meetings' by many of the team members) it is becoming difficult to report positive results during the monthly status meetings. The Architect, after looking back through status reports for the last couple of weeks - due in part to the fact that the Architect himself has been relying upon the Project Management meetings for status - he discovers that developers simply are not getting their questions answered in a timely manner and therefore do not often know what to code with clarity. He calls a meeting with the senior developers to ask them why they are late and what can be done about it. They report that they often do not get answers to their questions from the requirement team and reiterate that for business reasons, they were told they could not talk to end-users of the system. As a result, many things upon which they are to deliver are stalled until questions get answered and that at this point they believe themselves to be about one month behind schedule. The solution, they believe, is to require business analysts/requirement people to answer their questions within one business day of receipt.
So, the Architect takes this information to the Project Manager who, relying upon the Gantt chart picture, is in disbelief. After the Project Manager processes the impact and utters some unsavories on the Architect, the Architect puts the Project Manager 'in her place' by communicating that the PM is not doing her job by failing to make all parties focus on helping development get their job done and that as far as the Architect is concerned, this time lag is due to lack of leadership on the part of the PM.
During a normally scheduled weekly status meeting, the Project Manager sits down with the Project Sponsor and communicates the needed schedule change couched in 'we have a problem with the requirement team responsiveness which is impacting our schedule'. Effectively communicated, the Project Sponsor runs much of the message down the flagpole to the Business Analysis Manager who then communicates that his team is not technical and does not understand everything being sent to his team. As a result, responsiveness is slow because they have to figure out what is being asked, then go do research on an answer prior to delivering a verdict. At this point, the Business Analysis Manager believes they are about two weeks behind answering development originated questions.
When all parts of the problem are explored, discussed and addressed, two months are added to the project schedule after considering one month for development plus 20% overage, coupled with 2 weeks for the requirements team plus another 20% overage calculation.
August 16, 2001 - Today is the originally planned 'done' date for development. Due to the February 16 challenges discovered, two additional months of development were added prior to the test window suggesting development 'done' should theoretically be on October 16, 2001. After that, testing had planned to take the expected two months, but now theorized they may be bumped out to three months because they were finding disparity between the design documentation and the actual software they were receiving. During the August status meeting with anyone interested, including the food, music and theatrics, someone again asked if they could see the application demonstrated. The Architect immediately stood up and said they had been distracted by the change in project scope and schedule and had simply overlooked the need. He logged into the test site to show what may be there. A pleasant portal framework with nice skin compositions and a predictable, repeatable user behavior could be discerned. Most of the icons did not go anywhere, there were no reports, users did not yet have to actually login and the majority of custom wizards were yet inaccessible. However, while driving through the system most of the conversations were something like... "And this is where you will be able to do this...and this is where you'll be able to do that..." and so on. When questioned, the Architect simply stated that much of the work was actually done and simply had not been delivered to the test group yet.
October 1, 2001 - The revised done date being today, development reports that they are done with all work and waiting on testing. Not seeing this coming, testing communicates that while the development team reports that they are done, the test group does not have code much more mature than at the spontaneous meeting demo two months back. In fact, even though the Business Requirements group was later required to respond to all development requests within one business day, all transmission occurred via email and there were no updates to requirement and design documents as a result. So, the test group believed themselves to be at a stand still until someone updated the business requirements and design documents of yore. Furthermore, the test group communicated an inability to actually take new software loads and predictably and repeatably get them installed and usable within a one day time period. The Architect suggested this issue being due to the fact that the test groups had not tested the installation process yet and a war ensued.
After much discussion, some of it expectedly unhealthy at this point, it came out that the development group had stopped building and executing unit tests back in May when the project architecture was changed and then again when all computers were later re-imaged according to standards. The reality is that most developers had been reporting this through their individual status reports, but it had not been recognized due to the fact that the Architect and Project Manager were processing fewer and fewer of the status reports appropriately by only looking at those things 'done' and not including those things 'not done'. The fact of the matter was...the definition of done had been reported according to when code was checked into the version repository; and how hard people had been working was being measured by the number of checkins into the version control tool across thirty day periods. Exacerbating the situation was the fact that test progress had to date been measured by the number of scripts written versus the number of scripts actually proving out something important to the user. Ironically, upon the advice of the CTO some weeks back, the Project Sponsor decided to start making spontaneous visits to the on-ground troops to see how things were going. Little did anyone realize what she would be walking into during this spontaneous meeting occurring in the hallway near a marker board.
The Project Sponsor asked three questions:
1. How do you know when you're done?
2. What do you have available to show me right now in relation to the aforementioned definition of done?
3. What will it take to correct the current state of affairs?
The Project Manager walked in during this period of time and immediately chose a side of the gameboard without understanding what was actually happening. Acting surprised at what was being stated, the Project Manager feigned being upset with the Architect and the Development team and the Project Sponsor ended up taking control of the meeting, giving assignments and deadlines.
Three days later the Architect submitted detailed thought on the current state of the project suggesting development was basically done and testing needed to begin which would likely take two to three months. The Architect reported that integration testing was really happening every time a new build was performed, but that the test group should really be the ones doing the integration testing as a second set of eyes.
On the same deadline, the Test Manager submitted detailed thought on the current state of software maturity, how much work they had done, how much appeared to require rework, how much they didn't know and even slipped in some unnecessary commentary on the performance of project leadership. Needed time was estimated and reported at three to four months with the highest probability being four months to rewrite the old, write some new, and actually get some testing done.
The Project Manager, using the project management tool and individual status reports arrived at a six month estimate which included remaining development, testing and implementation work thereafter.
The Project Sponsor then took all of this data over to her CTO friend to look it over and discuss next steps. The CTO expected the project to be a relatively good state of affairs due to the fact that it had been assigned a senior architect and had a number of high end consultants on the side. Upon research together, the data points suggested the following:
To date time elapsed: 20 months
To date expenditures:
Super Consultants: US$105/hr * 55hr/wk * 48 weeks = $2,217,600.00
Pre-existing Staff: US$105/hr * 40hr/wk * 48 weeks = $13,305,600.00
- Blended wages this period: $15,523,200.00
- Infrastructure this period: $2,500,000.00
- Total this period: $18,023,200.00
- Previous period totals: $3,771,200.00
- Project Totals to Date: $21,794,400.00
To date ROI:
- 10 documents
- 76 automated tests
- 254 defects reported, 76 reported fixed
- 470 screen mocks
- 340,000+/- lines of code, commented
- 250 business requirements generated
- 127 business requirements fully/partially accessible in the system
Super Consultants: US$105/hr * 55hr/wk * 22 weeks = $1,016,400.00
Pre-existing Staff: US$105/hr * 40hr/wk * 22 weeks = $6,098,400.00
Approximate final project costs: $28,909,200.00.
The final straw came when the CTO had another senior technical person take a look through the application, the code repository, the build output, tests and test results and suggest that he did not personally feel comfortable taking on this project due to lack of clarity relating 'what is done' to 'what is stated to be done'.
The CTO went with the Business Sponsor to the Vice President's office to notify him in advance that they planned to suspend this project until such time as a more cost and time efficient method of completing the project could be ascertained. Angry with the situation, but happy the Sponsor and CTO had the fortitude to say 'stop', the VP simply asked two questions and an assignment for now:
- What are the top 10 things we did not do correctly?
- How will we make sure we will not do them like this again?
- You have 10 business days to bring me this material and recommended next steps.