We are delighted to be the cover feature of the current issue of MIT Technology Review. It was fun for me to spend time with Scott Rosenberg, and to show him what we are doing here at Intentional Software.
The article discusses our ambitious goals. We are working on a very tough and very technical problem and we understand the skepticism. We have early users, they are very supportive of our goals and their feedback is vital during this critical stage of development.
Because the content is so technical, readers might have some additional questions on the details. We have addressed some of these topics such as Degrees of Freedom which addresses the concept of the “Law of Leaky Abstractions”, or what is the role of the Generator, or on Hungarian Notation.
Thank you again to Scott for taking the time to delve into Intentional Software, and congratulations for the publication of your new book, “Dreaming in Code”!
And, lastly, thanks to all of you who are following us on this journey. We will continue to keep you informed of our progress.
By the way, I’m posting this from Star City in Russia where my space mission training is continuing for the April launch.
Charles Simonyi
ps. We are always looking for talented programmers. Please send us your resume if you want to join a great group and have an impact on the most exciting area in software.
I read that article of Technology Review Article online.
Your “intentional tree” looks very much like lisp code.
for example:
Return
(
Assign
(
a,
Div
(
b,
Plus
(
c,
1
)
)
)
)
if you change it to:
(Return
(Assign
a
(Div
b
(Plus
c
1
)
)
)
)
you will get a lisp, if you make these ‘Return’, ‘Assign’ as operator,
then your “intentional tree” can run itself.
I read Technology Review’s article Anything You Can Do, I Can Do Meta with keen interest. I must say that I feel vindicated, as I have been espousing this approach for decades.
I started my career in the early 80’s with a company that wrote and customized accounting applications. We used an advanced tool that ran on Wang minicomputers. While not a true DSL, it was geared towards small business accounting, and was heavily driven by meta-data. Our company’s productivity was high, as the programmers had limited degrees of freedom—the tool always guided them down a path which (almost) always led to success.
With the demise of Wang Labs, we ported to a popular multi-platform 4GL. Unfortunately, this 4GL was more general purpose, offering many more degrees of freedom for the programmer. First to go was consistency, then quality, then productivity. We soon found ourselves longing for the “old” days.
In 1989 we ran into another VAR at a conference who had also used the same tool on Wang. We shared our mutual frustrations, and learned that they had started work on a new tool similar with a similar philosophy. The tool sat on top of the 4GL we were currently using, and was designed to bring back the productivity we enjoyed by driving us back towards meta-data and code generation.
For the most part, the tool worked. The only problem was that it fell behind the times. Since the tool’s author had limited resources, they could not keep up with the constant advances in technology. However, since the tool sat on top of the 4GL, programmers soon learned to bypass the tool and program in the 4GL directly to add new features. From that point forward, we lost consistency, then quality, then productivity.
In 2000, I began working for the VAR that produced the tool. My eventual role was not to work on the tool, but to port the company’s flagship application away from it because the 4GL it sat on was no longer trendy. Since our flagship application was sold to large companies with “discerning” IT departments, we felt our only two choices were .NET and J2EE.
We chose a trendy technology, and experienced more problems than ever. Our programmers had even more degrees of freedom, which resulted in the least consistency, quality, and productivity ever. It should come as no surprise that our first project was not the success we hoped for. To be fair, it worked. But it was not a model for moving forward.
We analyzed our failure and decided to go back to our roots. Towards the end of last year we made another push towards meta-data and code generation. We spent three months modeling—what meta-data was needed, what code should be generated. Then we brought up an application in two weeks. OK, so the application is small and not completely finished, as we didn’t think of everything the first time around. But, from now on, what we do is augment the meta-data and improve the code generator. Finally, we feel that we can enjoy consistency, quality, and productivity again.
Granted, we were only able to bring up the application so quickly because we mined the old meta-data. But that shows another power of meta-data: if expressed well, it is completely portable. And if not expressed well, it can be transformed. It just has to fully express the intent of the software.
Hmmm…intentional software.
So, I understand your mission. But I am probably in the minority. Intentional software is going to be a tough sell. In my 25 years in the software industry, whenever I brought in a new programmer, they often disliked our approach. They wanted degrees of freedom. They wanted to be appreciated for their craftsmanship. They did not want the computer writing code for them.
However, programmers are the wrong audience to be selling to. The right audience is the product managers and domain experts. At my current company, there is a divide. Our domain experts are not programmers. Our programmers are not domain experts. It is simply too much for one person to be both an expert programmer with complete command over OOP, SOA, etc. and have in-depth knowledge of our industry. Requirements often get lost in translation between the domain expert and programmer.
What I would love to do is put more power in the hands of our domain experts. I want to take our meta-data approach to the next level. I want the domain experts to be able to completely customize the application using a DSL specific to our industry. Specifically, the sum total of what our application can do should be fully expressed by the DSL. When the DSL falls short, we extend it, not bypass it. Our domain experts define the DSL, our programmers implement it.
So, I will definitely keep a watchful eye on your company, and wish you success. Our industry needs it.
PS: enjoy the final frontier.
Don’t you need an ideas bootstrap loader for your pre-intended software that will create your first prototype generator?
How are you going to test the final generated solution, if testing involves building a parallel system at the meta level when you just went to all that trouble to build a meta engine to build the solution in the first place. Do you need to re-express the business scenarios that fed the solution generation in a way that exercises rather then creates. Can this be automated to make your system self test the generated application, or is this another way for a system too complex for mere humans to understand to declare itself to ‘work as intended’. I know the intentions of the mission better than you know yourself … shades of HAL.
No matter what intention capture/modelling language you invent it will always require translation from those who think they know what they want by those who know how to make the generation process work, disjoint and mutually exclusive skills ! “Lost in Translation” Geek vs businessman.
What ever happened to Linc, now that product was ahead of its time (at the time) but why didn’t the industry embrace 4GL’s. I think the previous poster had it right … 4GL’s are not sexy enough in an industry driven by nerds (for want of another name) at all levels of IT management out of the direct control of the business. What ever happened to three schema architecture, I always thought thay had the potential to model the external to internal “intention” translation process via a framework of conceptualisations.
Good luck!
The Bench-making machine with knobs is very interesting.
How does it solve the software updates problem? What would happen, if they need to change some features six month from the installation? Can we put the bench back in the machine to refine the bench, or do we need to start over and pay for full new bench? Many online applications are being updated every other month.
If one needs to build a computer table or wooden cabinet, can he use that bench-making machine? Or does he need to build a new machine for each kind of products? I am not joking. You would agree, if you read the following.
We already invented such machines for building online-GUI-applications. Appreciate your feedback, what you think about our online GUI application making machine. Please click on the URL and review our application machine.
Best Regards,
Raju
Sorry, I could not include links yesterday. With the help from Typepad tech-support, please let me try again: http://www.cbsdf.com/ps_blog/app-mechine.htm
Greatly appreciate your feedback, what you think about our online GUI application making machine. Please review brief overview to our ‘universal automobile machine example’: http://cbsdf.com/technologies/software-irony.htm
Each ‘Component Factory’ in the left side of the Figure#1 acts as a knob, to refine each part (i.e. a loosely coupled component/AC) in the application (shown right side).
Please review the following WebPages, which show that such supply-chain of ‘Component Factories’ for the plug-n-play parts based process builds perfect ‘application machine’ with simple to operate knobs to refine each part. Please review summary at the end to understand why it cost only a fraction to refine the application (to assure the application longer life): http://cbsdf.com/ps_blog/Minimum-couplings.htm
Please review the following for my personal observations, on the shortcoming of the traditional programming languages and software engineering:
http://www.cbsdf.com/ps_blog/State-of-Software-research.htm
Best Regards,
Raju
The concept seems promising, but it’s hard to sell the idea.
Writing a good program is only part of the issue. A good programmer needs a complete understanding of the problem to be solved. Writing in IP may obscure the purpose.
Java and C# is easy enough to write a good program in a large scale. Ada95 (some taste of IP) may be more natural but it did not gain popularity.
My suggestion is to invest your money on a real world problem. For example, every time we get a new computer, we need to install software, configure desktop, and move data from old system into the new system. Can’t we redesign the OS or create a new desktop software to solve this? How about when the hard drive crashes or infested with virus?
count0,
Excellent example of a simple code generator targeting a lisp environment
———-
Andrew,
Our audience is managers, domain experts, and programmers. The process certainly will not succeed without programmer involvement; they continue to play a very important role.
———-
Peter,
It is true that translation by the programmer will always be necessary. (See my response to Kong, below.)
Perhaps you’ll agree that given the potential improvement in developer productivity, it is certainly plausible that two parallel systems (implementation and test) could be generated from the domain code to ensure quality of the code generation. As Charles wrote in March 2005 “The information contents of programs” (http://blog.intentionalsoftware.com/intentional_software/2005/03/index.html)
it is perfectly plausible to turn a 1,000,000 line problem into a 20,000 line problem. Would another 10,000 lines of test generating code be that expensive?
———-
Raju,
Thank you for your continued interest in our technology. Our OOPSLA 2006 Paper (http://www.intentsoft.com/technology/IS_OOPSLA_2006_paper.pdf) provides a somewhat more technical description of our process. Please consider that my bench story (http://blog.intentionalsoftware.com/intentional_software/2006/05/index.html) is simply a metaphor for an intentional software development process. It is intended to show the similarity between our approach and Christopher Alexander’s theories about architecture. It also hints at the difference between our approach and past solutions such as off-the-shelf software, component software (COM/CORBA), or “in-house”/custom solutions. Please do not consider it a literal description of our technology.
However, this (from the bench story):
“These questions went on for a while, and soon, the bench maker began arranging his machines in the back of the truck into a new sequence after each question; later, after each question was answered, he turned knobs or adjusted levers on individual machines.”
does not indicate a “holistic machine”.
I will also say that “loosely coupled components” are an implementation choice related to the output of the generator in a given deployment scenario.
———-
Kong,
Actually, one advantage of Intentional Software is that the domain expert and the programmer communicate about the problem in a rich and expressive way via the domain code. It is extensible and persistent. Therefore, the knowledge conveyed is not lost via encoding by the programmer or through attrition amongst either the programmers or the domain experts.
Thanks for all of your comments!
–Shane
Shane:
Thank for your feedback. I have read the OOPSLA 2006 paper. I can see your intentions, but like to see example to understand how you will accomplish that goal.
We both are working on same problem with different philosophies and approaches. You wanted to create a domain specific media to express the intent.
I have taken a different approach. I greatly appreciate your feedback on our approach: We see that the software applications are not that much different from any physical-product, where intent can be expressed in two dimensions: (1). loosely coupled ‘parts’ and (2). ‘The interactions between the parts’.
Loosely coupled parts (Part level requirements):
I want to build them by employing a ‘Factory-indirection’ to refine each part to satisfy the intent. Here, I decided to take low level programming approach to provide excellent control and high degree of flexibility to the programmer. This must be done by the programmer, since the domain expert cannot deal at the level of database access, iterating over data sets and algorithms etc..
Of course, the programmer can employee ‘reusable components’ to make this task simpler and feel like high-level, as explained in:
http://www.cbsdf.com/technologies/misc-docs/CF-Goog-Charts.htm
The domain expert expresses the needed features and the programmer implements the features for each part. We can implement any CF in about 300 line of code.
‘The interactions between the parts’ (Service level requirements):
This part, I wanted to make very high level, so that even the domain expert can compose the parts with little or almost no help from the programmer. For this one to work, we need to implement a ‘Linker’.
Please review the following and appreciate your feed back on the ‘Linker’ or ‘Magical-knob in the garage’: http://www.cbsdf.com/ps_blog/Lego-programming.htm
Please notice, how we are capturing the intent in two different phases: The Low level Component factory and the Linker.
By using reusable classes, it is possible to limit the low level code size of the CF to around 300 lines. Furthermore, it is possible to design the CF, such that, the part offers simple high-level service-interfaces. These simple high-level service interfaces allow even the domain-expert to connect service-providers and service-consumers in the system (with almost no help from the programmers by employing the ‘Linker’).
Of course, if in the automobile-garage example, if the 12 year old replaces the Honda-civic engine with Porsche-engine and turns the ‘magical-knob, it displays error messages showing incompatible interfaces. Then he needs a expert mechanic to redesign the interfaces of the other parts in the Honda-civic.
Please notice, how simple it is to replace parts, when we employ a ‘Linker’. If you wish to see a working example, please install Adobe’s SVG Viewer to see our working examples for building online-GUI applications.
Best Regards,
Raju
Well I did visit this blog few times and I did found a lot of great information’s, keep up on good work
Usually I’m just reading the articles here, but Andrew mentioned something that I can’t stand to react: [programmers who are being hired] “wanted degrees of freedom. They wanted to be appreciated for their craftsmanship.”. Yes. Play piano, take photos and publish them or just join the open source community…
I’ve met a guy today from the field at my favorite telco, who wanted me to review his code. I said OK, that’s part of my job. The code was stateless web service that is working with some configuration data and a little database… After looking at a method called “Cron” (!) that was spawned a separate thread with an infinite loop, that did several things like config reload, saving data from memory I was kind of disappointed. I expected that, we’ll talk about best practices, how to organize the code, name things instead of “Cron” and so on, but we ended up on the usual stuff, like where is the spec [WHAT] and architecture document [HOW]. So simple: WHAT and HOW.
Turned out we can’t answer these simple questions. I told the guy that I can’t decide whether is right or wrong without knowing what he wants to do. It’s just waste of time. The encrypted version of garbage is just garbage, or in other words What You Want Is What You Get. Don’t know what you want? Don’t expect too much
I don’t want to blame the guy himself – he’s nice and hard working -, it’s just not his fault. It’s the fault of the whole IT world that cannot give the right tool for his company to express his job.
So, my favorite telco company is working in craftmanship mode without any good craftmans. Which means, that they don’t have control upon their mission critical system that provides real time customer facing service generating their revenue. (Ok. That’s not quite true. They have some craftsman who know what to do and keep the systems alive. But the craftmans does not scale as their business.)
Want another story? Everyone who is working in IT services business can tell a scary one.
Adam
Well, I have been following non-meta path for a while. The COLA languages look interesting. Is there a meta-message-passing system, or is there just message-passing? How many times must we accept data and treat is as an uninterpreted blob? I find things tend to break down when you try to implement communication. Thus Croquet and it’s implementation on top of a COLA system seem very promising. What Croquet needs however is some form of subjective (as opposed to objective) computer science. They already have multiple 3D views. What I am thinking is ontological views (probably with meta–or perhaps something far more generalized than meta). Thus transforming design into implementation and back is a form of subjective computer science. Perhaps we can go up to the meta level to escape the subjectiveness of computer science, or perhaps we just will always need translators/generators for each processes view of the world. The COLA languages look promising for explaining the nuts and bolts.