A question on Stack Overflow recently gave me an opportunity to give an overview of the approach I am adopting in my personal system development project.  My answer, including a summary of the original question is reproduced here. 

"So my question is, has someone already done this work and found (or even half found) a structured way to extrapolate an appropriate metaphor or heuristic for a set of given programming problems?"

I may have the start of an answer to your query. This consists of multiple levels of abstraction; descriptions of multiple domains; and the application of "modelling" techniques in a particular way - really rather different from what is normally done in modelling. The overall approach is, I believe, what you are seeking - it gives metaphors that are operated on, and then translated into the actual result. It is based on a number of published approaches, and relies heavily on some of these approaches and methods. 


What follows is subject to these caveats: 

1. This is very much "work in progress". 
2. I have been on the receiving end of derogatory comments that I am an "architecture astronaut", with the consequent implication that I am out of touch with the nitty-gritty of real system development. This is one of the reasons this is work in progress - my current project is designed to validate this concept, and demonstrate that it has real utility. In order to do this I need to develop a system of sufficient scale and complexity, using my approach, that would normally be out of reach of an individual developer. I am only part way through such proof of concept development. 
3. While the concepts I describe are derived from consideration of complex problem domains (sonar contact tracking system, and chip design systems), my examples relate to a simpler domain - basically an information system, constrained by a rules system. The rules base includes not only rules about the domain, but also rules about the applicability of other rules. I suspect, from your question, that I am coming to it from the opposite direction to you. You ask for a way to extrapolate an appropriate metaphor or heuristic for a set of given programming problems - which to me implies starting with the programming problems. My impetus has been from the analysis side - trying to find and formulate ways of describing a proposed system so that it can be implemented as a real system. 
4. When I use the words "model" and "modelling" I mean modelling as described below. This description is somewhat different from common usage of these terms. 

The three main constituents needed for this approach are: 


Multiple domains 
My definition of a domain is wider than that normally adopted: A domain is a separate real, hypothetical, or abstract world inhabited by a distinct set of objects and phenomena that behave according to rules and policies characteristic of the domain. In problem analysis.the domain is a useful unit of consideration when developing complex systems. Under this definition, there are multiple domains for consideration within a system, Often when developers refer to a domain, they mean the problem (or application) domain (referred to hereafter as P). However for this approach, any aspect of the system, or system development is a potential subject for modelling. This includes the system architecture (A); system production artefacts (code, make scripts, database schemas, etc) (C); DBA functions; etc. To approach P via metaphor requires development of several such domains - relating to the metaphor and transformations from the metaphor to a model of the real world, or to the code realisation of the real world in the developed system. When multiple such models are developed, they are all developed to the same degree of scope and precision. 


Multiple levels of abstraction 
To describing a problem and system, one models not only P, but also models appropriate higher levels of abstraction. Thus the metaphor chosen to describe P is modeled (M). In a similar way, the formalism of A (F) is modeled, and if deemed necessary the transformation process between P and C using A (R). So one abstracts the problem domain; abstracts the abstraction and so on. The application of multiple models is similar to colour separations - they lie on top of each other, and the system has to meet all the descriptions (the "complete picture") of all the layers. Again this differs from common ways of modelling which tend to meet such multiple requirements by elaborating the original model to take in the different constraints. This has particular implications when all of the architecture domains are effectively applied to all the elements of all the problem domains. 


Modelling 
What I mean by modelling differs from more usual approaches to modelling in the following respects:

• The subject matter of the model is highly likely to be different from domain to domain. This contrasts with modelling where initial models are elaborations of other models. 
• Indeed, one test of the validity of a model is that it represents an extreme version of the DRY principles - if something is defined in more than one place, it indicates a deficiency in the model. This extends to all domains under consideration, so the way the system is built is defined only in one place. 
• A domain, once modeled, is likely to be fairly static. The higher the level of abstraction, the less likely it is to change. 
• The scope of a model is likely to be much narrower, and much deeper than those conventionally developed. There are likely to be fewer objects within a particular domain than in conventional modelling; but these models must be complete and there are tests which give some indication of the completeness of the model (obviously, one can never prove that a system description is complete). 
• P is described in terms defined by M. The model might be expressed as diagrams, OO representations, mathematical formulae, or whatever is appropriate for the domain M. 

The following example, derived from my proof of concept project, may give some flesh to my descriptions above. I list some of my domains, together with candidate contents of the domain models. 

• P [Vehicles, units, movements, exhaustion, speed...] 
• P1 [Rules, applicability, states, priorities...] (a subsidiary problem domain) 
• M [Object types, attributes, relationships, domains...] 
• A [Events, tables, columns, database domains, classes...] 
• F [Persistence mechanism, processing,.....] 
• C [Database schemas, source code, SQL scripts, build scripts,....] 
• R [Formalisms, transforms, mappings...] 

The sources for this answer are many and varied, but do rely heavily on works by: 
Michael Jackson on structured programming; system analysis; and problem domain descriptions. 
The Shlaer-Mellor method of system development by transformation rather than elaboration. The Kennedy-Carter method consultancy.

System generated and managed user input forms

Most of my system work so far has been purely architectural - establishing the way the system is to be constructed, and the overall structure of the system.  This work has led to a style of programming that separates out interfaces and their implementation. My system runs until it needs data from the user; at which point it runs an input form, and retrieve data from it. It was obviously going to be trivially easy to go from the architecture to implemention of a running system. This was a statement of the blindingly obvious that proved not to be so.

My final implementation eventually proved simple, but the route to the solution was long and tortuous. I eventually investigated the Delphi concept of component ownership; the nature of Windows processing loop; the behaviour of modal forms; how to implement callback functions; and how to implement an interface based system controlling Delphi forms.

My initial approach was a naive, direct implementation of the architecture, The code illustrates that naive approach. The only problem is that it doesn't work! As explained by one of the answers I obtained from StackOverflow:

it doesn't allow the user to provide any input before it retrieves the content of TForm6.Edit1.Text. You basically say:
Create the form
Show it to the user
Immediately read whatever was set at designtime in the form's Edit1.Text

In addition, this basic formulation is not exactly what I wanted in terms of the architecture. The secondary form is called from the main form; I want it to be called from a unit rather than a form. It does not separate out the interfaces from the implementation in the manner I want, and if the example were to be scaled up, there would be a memory leak on each invocation of a new form.

So I started investigating and experimenting. My first exploration was the above question to StackOverflow. The preliminary response to this led me to investigating the use of modal forms rather than non-modal. I came to the conclusion that this was not an option - this has the side effect that a modal form blocks all keystrokes to the application, which blocks the application from creating new tasks. I did come up with a workaround to this particular problem. The use of modal forms also did not seem to meet my inferred requirements from the architecture I have derived.

My intended system
An grossly simplified overview of my overall system structure is:

A number of tasks can be run by the application. Each task applies a particular tool to an object. In order to obtain any needed user input the task may run one or more user interfaces (UI). These UIs are of varying complexity from a simple selection form to a multiple page wizard type interface taking a very substantial degree of information and user interaction. Each task needs the full information from its UI before it can proceed or complete. Each task may be sufficiently complex that the user may need to start another task before completing the first one, in order to get relevant information or provide updates to existing objects. The application does have its own UI, but this is minimal, providing a task/object selection only. All the work within the application is delegated to the individual tasks.

Example  
Using my example modules, one might wish to parse an XML version of a document concerning the Russian trip schedule. When selecting the document it is evident that the trip schedule needs editing. In order to correctly edit the trip schedule, the user needs to view the Moscow hotel booking.

I am currently programming one of the tasks, to prove that the overall structure is feasible, and to act as a template for subsequent development and code generation.

I am trying to produce the overall system in an extremely DRY (Don't Repeat Yourself) manner (perhaps I should call it ARID (Architectural Responsibilities In Design/Development)). As such I expect much of my eventual system to be described in domain specific languages, which I am also developing, and be produced by generators and templates which again form part of my overall development.

The UIs will be created, destroyed and managed by the task. In this respect, the manager is acting as a controller of the UI. The application needs to be able to run other tasks concurrently.

The above considerations have lead me adopt an programming approach based on interfaces, implemented by objects in Delphi - in particular, the UI as a Delphi interface, implemented by a TForm.

The point is still, though, that you have to allow the user a chance to provide input before you can retrieve it.

Other Approaches
As I continued to investigate I encountered other approaches. In no particular order they were:

1. Use Show and have the form pass back information (via some mechanism like PostMessage or an event handler when the user clicks a button to let you know there's data to retrieve.
2. Build reference counting into an enclosing TInterfacedObject object that includes the form.
3. Build reference counting into a TForm descendant.
4. Build each task as a separate process - which I understand is costly in terms of system resources and performance, on a Windows system.
5. Have the implementation object be a wrapper to TForm and implement the event handlers for the TForm in the wrapper. Incorporate a delay loop function in the event handlers for all the relevant events.
6. Use callback functions in the unit controlling the form.
7. Use delegate as a keyword in the interface declaration.

Problems and obstacles

1. Callbacks - the parameters of a callback function have to be derived from TObject rather than integers, booleans, interfaces, or other strucutured types.
2. Callbacks - most of the resources I could find explaining callbacks used examples that had the callback function called from the code that initalized the callback. It took me a while to realise I had to split these two parts - set up the callback, run the form, and then call the callback function.
3. Callbacks and Interface wrappers - I kept tripping over the the cyclic reference problem of Unit A needing to call Unit B which needed to call Unit A. This repeated in various ways as I experimented with callbacks.
4. Interface wrappers - I hit an infinite regress: - if I seperated the form from its interface, then I needed a callback from the form; and if that was seperated out......
5. Unit controller - trying to switch control of the form from another form to a unit shows that the controlling object needs to be derived from TComponent. This ensures that the TForm descendant is owned by something that will hook into the form management routines correctly.
6. Windows processing loop - In order to use Delphi's implementation of the window's processing loop, all forms need to be ultimately owned by TApplication. This requires use of TApplication.CreateForm - either directly in Application; or indirectly via ownership by a TComponent, which is itself owned by Application (using TComponent.Create(Appliction)).
7. TForm creation - the procedures used for communication with the form have to be  published rather than public - if they are only public the form will not display - for reasons I have not fully investigated.

My solution
My solution defines

1. Two interfaces: IResult - defined to carry the required data from the form to UserInput; and     IUserInput - obtain user input from a form in an IResult. 
2. Two creator functions - one for each Interface
3. Implementation of the interfaces - one object for each interface, and one for the actual user input form.

Example code for this solution is attached. The example has not been extended to ensure that clean up is done correctly, and the form event that makes the information available is the closing of the form. In a real situation it would probably be a user triggered event.

The UserInput interface is a little odd since the sole property of the interface, UserInput, is a write only property - the intention being that property is filled by the user input; and implies that the appropriate setter needs to have been called before use is made of the interface. I am still not completely comfortable with this, though it does seem to be a solution to the issues I faced.

Limitations and Constraints of Translational Development

The previous notes I have drafted have outlined on the differences between translational and elaborational approaches to system development. There are several costs incurred in adopting a translational approach, and these are summarised here.

The whole approach of translational development is intended to deal with large scale, complex projects. It can therefore be seen as overkill for smaller projects, or for small team projects. I trust that my development of the project I am using as a test bed will show that it can be successfully, and efficiently used for a single man project.

The approach is somewhat prescriptive, and requires discipline in the development process. There are certain criteria which must be met in the development artifacts, or the method will not be successful.

There is a substantial problem of "where do I start". If the method prescribes modelling the application, and modelling the architecture and then applying the latter to the former in order to provide the final system, how do you start this process off? My test bed project will illustrate how to break into this (apparently incestuous) cycle of modelling and system generation. However, unless one starts off with a pre-populated architecture, there is undoubtedly a large amount of technical work to be done up front, which appears not to be productive, in that there is no tangible system as a result of it. The approach is productive, but initially does not appear so.

Tied in with the difficulty in starting the project, there is no doubt that the initial iteration, if an iterative approach is adopted, will be much larger than subsequent iterations, and does not tie in well with the lessons learned from extreme programming.

In order to develop a system iteratively, using the translational approach, one has to develop a vertical slice of both the application and the architecture. This may mean that analysts, and software architects may well have to be developing in multiple disparate domains which can lead to "jack of all trades" syndrome - where everything is done, but not necessarily very well. This will impact on the subsequent development, and may result in rework that would not otherwise be necessary.

This note concludes my overview of what I have described as an alternative approach to system development, from that normally adopted. These notes will go on to describe this alternative approach in detail, and illustrate it by reference to the software development I am using as a test bed, and exemplar for the method.

Translation and Extreme Programming

Extreme programming is an approach that brings together a variety of techniques with the aim of producing software more productively, with higher quality, and with speedy delivery of functionality to the user. The techniques it employs include the use of:
1. User stories - the idea that the programmers will select and develop one of the stories that the user uses to describe required functionality. These stories lead to rapid iteration since the objective of a story is to be met within one iteration (typically a fortnight).
2. Unit testing is paramount - to the extent that ideally, all tests will be written before any software is constructed.
3. Programming in pairs - to generate and utilise the effects of creative feedback on the software development process.
4. Constant and immediate access to the user.
Read More...

Requirements Specifications and Designs

Elaboration vs Translation

I described in my note comparing "mainstream" software development methods with "alternative" methods, I suggested that mainstream methods develop by elaboration and refinement, while alternative ones develop by translation. The comparison between the two was summarised by Sally Shlaer and Stephen J Mellor, in their paper "Recursive Design of an Application-Independant Architecture", and I cannot descibe the difference any better than by quoting from that paper:

Current View
Although practitioners within our industry hold diverse views, we believe that the following five widely held assumptions constitute the core perspectives on object-oriented methods, patterns, and architectures.

1. Analysis treats only the application—the subject matter of interest to the system’s end user, such as air traffic control, a telephone switch, an automated warehousing application, and the like.
2. The analysis must be couched in terms of the conceptual entities of the design. This assumption appears in much of the work on domain-specific software architectures, as well as in many object-oriented analysis and design methods. Hence a developer employing an object-oriented analysis method is necessarily constrained or biased toward an object-oriented design.
3. Despite the lack of a universally accepted definition for architecture, most people would agree that an architecture provides a view of the entire system. Because this view emphasizes the major elements of the system, many details are necessarily omitted.
4. Patterns are generally thought of as small units, consisting of a few objects with wellworked- out interactions.
5. Programmers expect to select, adapt, and modify patterns according to taste and experience. Patterns are seen as useful but advisory in nature: Use of patterns in general or in particular is not required.

An Alternative View
Our Recursive Design method is founded on a fundamentally different approach to this topic that makes five very different assumptions:
1. An analysis can be performed on any domain, not just the subject matter of interest to the system’s end user.
2. Use of an OOA method does not imply anything about the fundamental design of a system.
3. Because we treat the architecture domain just like any other domain, it can be modeled in complete detail using an object-oriented analysis method.
4. The OOA models of the architecture provide a comprehensive set of large-scale, interlocking patterns.
5. Because all the code other than purchased packages is automatically generated, use of the patterns is absolutely required. Read More...

Method Wars - NOT!

In the late eighties and early nineties when there was an explosion of different software development methods, there was a tendency for advocates of a particular method to develop arguments in favour of their particular method which went something like:
1. My method is undoubtedly the best (for all problems - an issue that I will come back to).
2. Because my method is the best, all others must be rubbish.
3. Method X (i.e. not mine) is rubbish because........(fill in your personal reasons here).
4. Just to reinforce my argument here is a detailed comparison between my (superior) method and method X.

This led to spurious method wars, with the advocates of a particular method slagging off the competition.

I am not interested in resurrecting any of the above method wars, but to describe my approach in building a system, I need to compare tendencies, rationales and approaches of the two streams. They are not absolutist - both strands of method development have learnt from each other - and neither strand is the definitive source of any one technique. My comparisons are of tendencies.

I am generally going to use Catalysis as the archtype of mainstream methods - because it is probably the most comprehensively documented and consistent method from the mainstream. Equally, the alternative stream will be described by reference to the Shlaer-Mellor method which is the most complete example of the alternative strand. Read More...