For years we used Waterfall methodologies and built amazing systems.
These methodologies should be tools to use when needed. Are there still
reasons to use these methodologies? Are agile approaches the new super
hammer?

Here is my take. We had reasons for big design up front. The reasons
varied from technical to territorial.

Reasons for BDUF: (No particular order)
1) Build times were submitted and it could be days before it was
loaded/compiled/ran, etc.
2) Debuggers where weak and it was more efficient to avoid bugs than
look for bugs.
3) Software management was emerging as a business and manage means
control.
4) Management flexed its new found power and said it will hold
developers accountable.
5) Developement needed a contract to be used in defense. (Development
new that all requirements are not known up front but management
believed that if more activities were performed then all of the
requirements could be discovered)
6) Software businesses emerge with many departments and with all of
these departments and people the business requires efficient use of the
human resources discounting the efficient development of code. This
means keeping marketing people busy at the expense of development. This
means keeping HR people busy at the expense of development. This means
keeping managers busy buzzy around at the expense of development. (Is
this enough to get a rise out of someone?)

In my humble opinion (IMHO) I feel that the reasons I list as (1) and
(2) are the most important. If there is something expensive to do (i.e.
build time) then do activities that are less expensive.

So in todays world when is BDUF useful. Suppose you will have only one
meeting with the customer. You had better get all you can get from that
meeting. Gather requirements. Create paper prototypes and present them
to entice further requirements from the customer. Because, what you
walk away with is what you have.
Can you think of another example?

Geoff

From:	Alan Gauld
Subject:	Re: When does BDUF and Waterfall methods work?
Date:	Wed, 12 Jan 2005 19:32:33 +0000 (UTC)

On 12 Jan 2005 08:24:34 -0800, "Gilligan"
wrote:
> For years we used Waterfall methodologies and built amazing systems.

And still do.

> Reasons for BDUF: (No particular order)
> 1) Build times were submitted and it could be days before it was
> loaded/compiled/ran, etc.

And for big systems still are. I was working on a mainframe COBOL
system as recently as 2000 that took a full weekend to build from
scratch...

> 2) Debuggers where weak and it was more efficient to avoid bugs than
> look for bugs.

It still is in any kind of complex system. If I have to look at
code to find which file a bug lives in then my design has failed.
And I can navigate through my design faster than I can my code...

> 3) Software management was emerging as a business and manage means
> control.

And for Waterfall this is crucial Waterfall always was about
project management more than actually building software. If a
project is big and spead over multiple teams, with different
contractual terms per team etc then waterfall is a manageable
way of maintaining control over the whole. Not the only way but
one that is well understood and has a proven track record.

> 4) Management flexed its new found power and said it will hold
> developers accountable.

And now lawyers are doing the same...

> So in todays world when is BDUF useful.

Where you need to communicate to a large dispersed group.
But one key thing to remember is that BDUF never(*) meant
design that was not repeated or refined, it just meant
having a first cut design available up front.

(*)OK Maybe a very long time ago a few folks tried it, but for
the last 25 years at least the experts have been saying iterate
your designs and requirements.

> Suppose you will have only one meeting with the customer.

That still leaves the problem of fixing the holes you find on
analysis, so onemeeting will never be enough. But if you only
have limited access it muight be valid.

The other situation, which only occurs rarely, is where the
requirements are very well known and not likely to change. An
example would be a protocol convertor (Could be for a new
hardware platform. Or in a new language. Or between two protocols
that have not been converted before) The protocols are well
established, the requirements clear and unambiguous and unlikely
to change. In this case designing up front is likely to pay off
and will probably consist of basically copying an existing
solution.

HTH

Alan G.
Author of the Learn to Program website
http://www.freenetpages.co.uk/hp/alan.gauld

From:	MethodMan
Subject:	Re: When does BDUF and Waterfall methods work?
Date:	18 Jan 2005 08:32:59 -0800

Perhaps I'm misunderstanding your meaning, but what's wrong in
enforcing
the business-rules at the data layer? That's standard with most large
data based systems. The applications using the data may change, but the

business 'rules' remain the same.

The thing is that business rules do change. A rule of thumb is to put
only very rudimentary business rules on the data layer which are also
unlikely to change. Let the database do what it does best which is to
manage, move, and present data. Let other applications do what they do
best which are business processes.

That isn't to say that we shouldn't use stored procs (because we
should), just that they should not be where complicated business logic
resides.

Also, it just hit me when you said enforcing business rules in data.
If you mean constraints that is also fine. These types of rules are
very rudimental and easy to support.

Now let me give you an example just so there is no confusion. If you
wanted to get the sum of all sales across a period of time, that would
be something to put in a stored proc. You pass in a date range, and
magic happens. If you wanted to then compare sales with expenses
across the same date range, my application (let's call it
SimpleAndUselessApp V1.0) would be designed using the following
architecture:

2 stored procs. Both accept date ranges. One returns sales totals and
one returns expense totals. On the application side we would take that
raw data and do whatever business calculations were needed. Perhaps we
are getting too much data and want to get weekly summaries, then I
would change the stored procs (or add two new ones) to return weekly
averages. That is about as complex as stored procs should get.

In my possesion right now is a query (korn shell script that contains
SQL) that uses no stored procs. Just an ad-hoc query. It is 7 pages
long when printed. The goal of this query is to get a history sum,
forecast sum, causal impact value, and forecast causal impact value.
You don't need to understand what those are, just that they are values
that this query then evaluates and returns a ratio (for supply chain
stuff).

The proper way to do this would be to create 4 very small stored procs
that return those values based upon a few defined values (like date
range and item number). You would then support 4 small stored procs
with about 6 lines each and a simple piece of code on the appication
side, probably 10 lines max plus error handling. Not only would you
decrease the size of the code, but you would make it easier to
understand the business rules behind it. Now we are working towards
understanding the business as opposed to trying to understand complex
code.
You should try to avoid adding complexity to you system, in my view.

From:	MethodMan
Subject:	Re: When does BDUF and Waterfall methods work?
Date:	17 Jan 2005 11:55:49 -0800

I agree that we should not arbitrarily discredit the efforts of our
predecessors. Many of the latest "and greatest" methodologies are
based upon past success and failures.

I don't agree with the notion that we should forget old practices
altogether, but on the other hand I think we should grow as a
profession. The jury is still out on some of the latest methodologies,
but even if you don't adopt them at worst you have been forced to think
critically about your current practices. At best you can adjust your
practices for the better and make/save your company money, plus do a
better job developing software. If people like Kent Beck are
discredited (which I highly doubt will happen) at least they have
prompted our industry to mature.

From:	MethodMan
Subject:	Re: When does BDUF and Waterfall methods work?
Date:	17 Jan 2005 11:50:11 -0800

I am doing a thesis on selecting a methodology; although many of the
ideas in traditional development are outdated many still hold true.

A scenario I can think of where one might want to use the old
methodology (waterfall) is in the case of updating legacy systems. In
this case the business rules still hold true, have already been
designed, and there is really no use to re-invent the wheel or develop
incrementally.

***HOWEVER*** I think that we have learned from experience the value of
unit testing and refactoring. So even though you might not adopt the
PROCESSES of more agile methodologies, I would consider adopting some
of the PRACTICES like unit testing, refactoring, etc.

Aside from legacy systems and very large enterprise systems I can't
think of a reason to use the old methodologies. Even in those cases I
would at lease adopt some agile practices in order to increase quality
(or decrease the chances of introducing a defect).

From:	Phlip
Subject:	Re: When does BDUF and Waterfall methods work?
Date:	Tue, 18 Jan 2005 04:46:57 GMT

First, a note about netiquette. Google has a splendid interface, but its
"Reply" system is inspiring people to leave out the quoted-to text. This
makes posts hard to read.

Fix it by hitting Preview, then Edit again. For whatever reason, Google only
_then_ pushes in the reply text.

/Then/ (>sigh<), don't top post, and trim the quotes back!!

MethodMan wrote:

> A scenario I can think of where one might want to use the old
> methodology (waterfall) is in the case of updating legacy systems. In
> this case the business rules still hold true, have already been
> designed, and there is really no use to re-invent the wheel or develop
> incrementally.

The worst aspect of Waterfall is making decisions about requirements in
large batches, long before implementing them, collecting feedback on their
implementation, and learning. Waterfall burdens a project with the

> ***HOWEVER*** I think that we have learned from experience the value of
> unit testing and refactoring. So even though you might not adopt the
> PROCESSES of more agile methodologies, I would consider adopting some
> of the PRACTICES like unit testing, refactoring, etc.

Those form software's inner cycle. The outer cycle is delivering software of
high business value to users, and ensuring they can use it. Replacing a
legacy system is not excused from this feedback. You should deliver the new
system's highest value features as early as possible, get them online, use
them to boost user productivity, and learn from them before committing to
decisions about the next batch of features.

> Aside from legacy systems and very large enterprise systems I can't
> think of a reason to use the old methodologies. Even in those cases I
> would at lease adopt some agile practices in order to increase quality
> (or decrease the chances of introducing a defect).

It's the very large enterprise systems where "big requirements up front" was
causing the most problems. Read /Agile & Iterative Development: A Manager's
Guide/ for a summary of study after study regarding this effect.

--
Phlip
http://industrialxp.org/community/bin/view/Main/TestFirstUserInterfaces

From:	Phlip
Subject:	Re: When does BDUF and Waterfall methods work?
Date:	Tue, 18 Jan 2005 04:49:43 GMT

Phlip wrote:

> The worst aspect of Waterfall is making decisions about requirements in
> large batches, long before implementing them, collecting feedback on their
> implementation, and learning. Waterfall burdens a project with

decisions made with the least information possible.

> --
> Phlip
> http://industrialxp.org/community/bin/view/Main/TestFirstUserInterfaces

From:	MethodMan
Subject:	Re: When does BDUF and Waterfall methods work?
Date:	18 Jan 2005 04:35:13 -0800

<<
It's the very large enterprise systems where "big requirements up
front" was
causing the most problems. Read /Agile & Iterative Development: A
Manager's
Guide/ for a summary of study after study regarding this effect.
>>

I have a book called Agile Software Development In The Large that
addresses some of your issues. I agree that the large systems are most
challenging, but I am not sure that XP addresses how to balance
development in large teams or organizations. They would benefit the
most if we could make agile work well in that environment. I work at
an organization with similar challenges because of its size.

I think a factor to also consider is culture. Where I work there is a
"mainframe" culture where batch scripts and large stored proceedures (I
mean pages and pages) are the norm. Although there is of course a
place for batch scripts (bulk data movement) the company I work for
puts alot of the business rules in the data layer. This creates a
problem in support, extensability, and simply tracking down the source
of a problem. This "old-school" mentality doesn't realize the value of
newer technologies.

Not sure how that relates to methodology, except that if large
organizations won't adopt newer technologies I doubt they will adopt
newer technologies. And I am willing to bet that these are the
companies who will sink rather than swim. In fairness my company is
coming around, but there is still a cultural "war". I value the
mainframe, but I see it as a workhorse not business engine.

-MethodMan

From:	Phlip
Subject:	Re: When does BDUF and Waterfall methods work?
Date:	Tue, 18 Jan 2005 14:59:15 GMT

MethodMan wrote:

> <<
> It's the very large enterprise systems where "big requirements up
> front" was
> causing the most problems. Read /Agile & Iterative Development: A
> Manager's
> Guide/ for a summary of study after study regarding this effect.
> >>
>
> I have a book called Agile Software Development In The Large that
> addresses some of your issues. I agree that the large systems are most
> challenging, but I am not sure that XP addresses how to balance
> development in large teams or organizations. They would benefit the
> most if we could make agile work well in that environment. I work at
> an organization with similar challenges because of its size.

A> read the latest XP book by Kent Beck.

B> I'm not sure, but I suspect there are iterative processes other
than XP. The goal of the game is frequent deliveries of high
business value.

> I think a factor to also consider is culture. Where I work there is a
> "mainframe" culture where batch scripts and large stored proceedures (I
> mean pages and pages) are the norm. Although there is of course a
> place for batch scripts (bulk data movement) the company I work for
> puts alot of the business rules in the data layer. This creates a
> problem in support, extensability, and simply tracking down the source
> of a problem. This "old-school" mentality doesn't realize the value of
> newer technologies.
>
> Not sure how that relates to methodology, except that if large
> organizations won't adopt newer technologies I doubt they will adopt
> newer technologies. And I am willing to bet that these are the
> companies who will sink rather than swim. In fairness my company is
> coming around, but there is still a cultural "war". I value the
> mainframe, but I see it as a workhorse not business engine.

It sounds like the folks with the keys to the mainframe are holding the
business process hostage, to maintain their domain. "If you want to change
the business rules, you must come thru us."

One wonders what their turnaround on change requests is. I would suspect
those big scripts are very, very hard to change.

Here's a little parable: I know a shop with scores of programmers, and with
a small "Information Technology" department. The latter use an online
request system, so you can write a ticket for anything you need, such as
"give my workstation a network alias", or "the bulb in my lamp went out".
The IT department triages those requests and finishes them one by one. And
the developers generally use code-and-fix.

So guess which department has a turnaround time orders of magnitude faster
than the other?

--
Phlip
http://industrialxp.org/community/bin/view/Main/TestFirstUserInterfaces

From:	Andrew Hardy
Subject:	Re: When does BDUF and Waterfall methods work?
Date:	Tue, 18 Jan 2005 15:35:37 +0000

MethodMan wrote:

> place for batch scripts (bulk data movement) the company I work for
> puts alot of the business rules in the data layer. This creates a
> problem in support, extensability, and simply tracking down the source
> of a problem. This "old-school" mentality doesn't realize the value of
> newer technologies.

Perhaps I'm misunderstanding your meaning, but what's wrong in enforcing
the business-rules at the data layer? That's standard with most large
data based systems. The applications using the data may change, but the
business 'rules' remain the same.

From:	Gilligan
Subject:	Re: When does BDUF and Waterfall methods work?
Date:	12 Jan 2005 11:50:19 -0800

Wow. In 2000 the Mainframe Cobol system took a full weekend to build. I
have heard of other large systems, it seems like the airline
reservation system takes several days to build as well.

I started this discussion because I am afraid that we will "run" to the
latest craze (XP or Scrum or whatever) and forget the lessons learned!
I feel that way to often we think of people and their actions in the
past as being unsophisticated. That is not the case. The Greek
Philosophers are an example. The Roman Senate and government is an
example. The reasons for this BDUF were addressing problems and
concerns. Those problems and concerns will resurface from time to time.
Let's not forget the lessons of the past just because we want to be
zealous about our company adopting a new way of development.

Here is a snippet from a book about the past:

"I know a textile company that wove fine English woolens. Its 200
employees worked in a machine-filled factory set in what might be
called an exurban industrial park. The chief executive was definitely
performance-oriented, starting with his own: he arrived early, left
late, and made all of the important decisions in between. The factory
was subdivided into specialized areas of production, each with its
workers. Each boss, in turn, had a group of foremen to watch the
workers. Accountants and salespeople were on the mezzanine above the
shop floor and reported to their respective department heads.
Everything was strictly hierarchical and pyramidal.

As I described this company to an international telecommunications
convention... I could see people in the audience becoming more and more
puzzled. What was the point? They seemed to be wondering. It was just
an ordinary business. There didn't seem to be anything distinctive
about it.

Except that this textile company existed in 1633. And the moral of the
story: our advances in technology have far outstripped our advances in
mentality."

My point is this, the people in 1633 where not unsophisticated. They
were addressing real problems and the results of their solutions are
very similar to the solutions we come up with today.
Thanks for you comments.

Geoff