What is Engineering?

Engineering is the application of technology to solve human problems. More specifically, engineering is a professional activity that uses imagination, judgement, and intelligence in the application of science, technology, mathematics, and practical experience to design, produce, and operate useful objects or processes that meet the needs and desires of humanity. Professional practitioners of engineering are called engineers. As a noun, "engineering" is also the collective body of available technological tools, activities, knowledge, and processes.

The application of theoretical findings to build products that rely on this theory for their realization.
The use of a process for the elaboration of needs, and production plans and for the building of such products.
The application of quality criteria and appropriate tests to ensure the quality of the products built.
The application of professional ethics to make sure that the development of such a system is safe and profitable for the people influenced by this system.
The application of economics principles that will make sure that the solution, including its development , use, and maintenance will be economically viable.

Examples of engineering areas include:

Building Engineering
Road Engineering
Aircraft Engineering
Aerospace Engineering
Electrical Engineering
Computer Engineering
Software Engineering

What is Software Engineering?

Software engineering is the profession that creates and maintains software applications by applying theories, technologies and practices from computer science, project management, engineering, application domains, and other fields.

Software is the set of directions that enables computer hardware to perform useful work. In the last decades of the twentieth century, cost reductions in computer hardware led to software becoming a ubiquitous component of the devices used by industrialized societies.

Software engineering, like traditional engineering disciplines, deals with issues of cost and reliability. Some software applications contain millions of lines of code that are expected to perform properly in the face of changing conditions.

Origins

The term software engineering was used occasionally in the late 1950s and early 1960s. The term software engineering was popularized by the 1968 NATO Software Engineering Conference held in Garmisch, Germany and has been in widespread use since.

Meanings

The term software engineering is commonly used with a variety of distinct meanings. Many of these more or less widely accepted meanings are inclomplete. Here is a list of such meanings in increasing order of completeness:

As the usual contemporary term for the broad range of activities that was formerly called programming and systems analysis and design;
As the broad term for all aspects of the practice of computer programming, as opposed to the theory of computer programming, which is called computer science;
As the term embodying the advocacy of a specific approach to computer programming, one that urges that it be treated as an engineering profession rather than an art or a craft, and advocates the codification of recommended practices in the form of software engineering methodologies.
Software engineering is "(1) the application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software, that is, the application of engineering to software," and "(2) the study of approaches as in (1)." -- IEEE Standard 610.12

The Need for Software Engineering

Software is basically code that is stored in and executed by an eletronic apartus. It can range from being extremely short and simple, for example the software that controls a watch, a calculator, a microwave oven, etc, to being enormous in size and complexity like banking systems, and government systems such as income tax management systems, or weather prediction systems.

Software Enginnering vs programming languages

The early evolution of the software field is strongly related to the development of programming languages.

40s - Binary programming: All computers (with very few exceptions) are using the binary language for computation. In the early days of computer programming, all programs were written in this language. Because of the completely alien nature of the binary language, there is a limit to the size of programs that can possibly be undestood or managed by humans. That meant that programs had to be kept rather small, and that programs growing above a certain size actually become incomprehensible and unmanageable.

50s - Assembly programming: The development of the assembly language brought some mnemonics that could be understood by humans, and later on some constructs like subroutines that permitted to develop reusable code for the first time. That permitted to develop larger programs before the comprehensibility barrier would be reached.

60s - Structured programming: The 1960s saw the development of structured programming languages, such as ALGOL, COBOL, Fortran, PL/1, PASCAL, and eventually C, along with the development of theories to develop compilers to translate these languages into binary executables, as well as, later, parser generators enabling the easier generation of new languages and their compilers.

During those very promising days of computer science, there was the idea was that these high-level languages would allow to solve all computing problems. Very large software systems (airline reservation systems, air traffic control systems, database systems) were developed. A major crisis was unfolding because of the power of the new languages developed, and the inability for people to sufficiently analyze very complex problems before attempting to solve them. Most of the more ambitious projects failed, which then made some people realize that software had ought to be developed as other complex artifacts, that is, by being Engineered rather than simply constructed. This created what is called the "software crisis", and many believe that we are now still struggling with this crisis, 40 years after it initially happened.

The Software Crisis

Symptoms

Software developers attempted to apply the same software development strategies used on the smaller software systems for large software system. This led to the following symptoms of software development projects:

many of the systems were never delivered;
if they were delivered, they were frequently late deliveries;
lack of user acceptance;
ineffcient software;
budget overruns;
lack of reliability;
lack of maintainability, etc.

Root causes

This was all due to the fact that software development was not treated as an engineering (i.e. systematical) activity and not as an ad-hoc activity. Too much emphasis was put on programming with lack of emphasis on problem analysis and abstract thinking about a solution, i.e. design activities. The main root causes of the above symptoms were as follows :

Software developers are capable of producing specific software functions properly most of the time, hence they have the feeling that they can properly build a program to solve any problem, as a program is a set of such funtions.
Many problems lie in the interactions with the client. There is a widespread inability to gather the right requirements for a project, and lack of comprehension of the fact that in most cases the client is not entirely aware of his own needs, even as the project is being built.
The needs are growing much faster than solutions. As new problems are being solved, other projects of higher magnitude are tried. There is potentially no limit to the complexity of software projects.
Most CASE tools are inappropriate. Having proper tools increases the productivity, which is lacking in some development phases. Moreover, many case tools vendors claim universal applicability of their tools. In practice, different projects require different tools. Particularly, using "overkill" tools on a smaller size project often results in a drop in productivity. Effort is currently made to have flexible tools, which often result in increased complexity of use.
Software solutions have to be found for widely different application areas. Understanding of the application area (i.e. the problem domain) by the development team is often a neglected effort.
The application area is sometimes not properly understood even by the client.
The greatest of all diffculties lies in the ever-present changes made to the system definition as it is being developed.
On top of that, all previously developed programs have to be maintained. Developing maintainable systems actually increase their complexity and the effort required to develop them. Most systems eventually have to be changed after their delivery, even if it was not realized at the time of their development.

Software Engineering: Challenges

Managing software project complexity

Software engineering is mostly about managing the complexity of software projects: complexity of the problems to be solved, complexity of the solutions, complexity related to human collaboration, etc. There are various factors making software projects complex:

Complexity of the problem to be solved: In many cases, it requires months to investigate on the problem itself. Some projects are eventually found out to have hundreds of requirements, most of them being cross-related, and having different importance. When the requirements are elicitated from people, there are additional communication problems, like users ineffectively communicating their needs, lack of consensus on real needs among different stakeholders, misunderstandings between stakeholders and developers, etc. Gathering and properly stating the problem description is a crucial aspect of software engineering, but yet is too often neglected in practice.

Size of the code: Adding up the total number of lines of code, some programs can have millions, even tens of millions of lines of code. Many people argue that the number of lines of code is not a good indicator of complexity, arguing that two different programs with the same number of lines of code can be of widely different complexity. However, nobody will deny that any 100 lines program is much less complex that any other with 100,000 lines of code. Code is written in different files that need to be assembled together to form an executable. As the development advances, some files are updated, creating new versions. As new file versions are created, one has to make sure that previous versions are still avaliable if something goes wrong and there is a need to revert to an earlier version. The problem is of course aggravated when a multitude of programmers are creating new files or new versions of existing files concurrently.

Size of the team: As the number of people involved in the project gets bigger, commmunication and coordination problems arize. These problems have to be tackled, or are very likely to lead to project failure, no matter the capacity of each individual in the project, or the level of difficulty of the project. Managing a project involving a team of 5-10 developers is somewhat easy with some experience. But one also have to take into consideration the client side, whose contributors also have to be managed, for example in order to organize meetings, and to assign roles to different client contributors in the project. As the number of people involved reach several tens, even hundreds or more, correctly managing human resources becomes paramount for the productivity in the project.

Managing change

Soft nature of software: One particularity of software is its soft nature, i.e. it can be changed as it is developed, or even after it is operational, unlike many other engineering artifacts. For example, after a skyscraper or a bridge is assembled, there is are strong limitations to the changes that can be applied to it. One cannot simply add 10 more floors to an existing building after it has been built, because of the physical limitations of the structure in place. Software is apparenlty infinitely mutable. Because software is composed of code that apparently has no stuctural limitations, one can imagine that existing software can be easily changed, even that there is no limit to the magnitude of changes that can be applied to software. This is wrong. Once software starts to be build, it actually starts to have an architecture that plays the same role, and brings the same kind of architectural constraints, as the architecture of a building. In fact, change happens at every stage of the lifecycles, from the ever changing requirements to the maintenance requiring unexpected adaptation and additions. For that reason, one very important consideration is to expect change and manage it properly. One example is to design for change, i.e. developing system components that can be easily changed or adapted thereafter.

Ripple effect of changes: When a change is applied on a praticular artifact, e.g. the code contained in a single file, there is often a ripple effect of changes, as changing a unit (i.e. a function or class) potentially has an effect on other neighbouring units, and so on. Moreover, changing a code unit is likely to have ripple effet of changes to other artifacts, such as the design, or the test cases related to this unit. The changes that are likely to generate more magnitude in that ripple effect are changes to the upstream artifacts, i.e. the artifacts on which all the others are based, namely the requirements. If requirements are changed late in the development, this is likely to affect not only other requirements, but also the design, code, and testing procedure. It is important to contain changes as much as possible and find ways to reduce the magnitude of the ripple effect at all levels. Developing the system into self-contained modules is one way, for example, modular design will tend to contain changes inside of modules. Minimizing the amount of accumulated and unsound information is also another way. Enormous documents that go in great amount of detailed description are hard to change. Documentation should include information that is minimally likely to change, and be written in a way that is easy to change.

Critics about software engineering

Critics argue that many of the foundations of software engineering are inherently flawed. The following paragraphs list many criticisms and responses. Note that many of these criticisms apply to other human activities including business and education, as well as other engineering areas.

Managing expectations

Criticism: One key to successful software engineering projects is managing the customer's expectations to something that can be built and delivered. So, the field resembles marketing or sociology more than traditional engineering with its responsibilities to society at large and the perils of legal liability when they fail to protect the public interest.

Response: Every profession manages expectations, including all branches of engineering. Moreover, responsibility to society means meeting the expectations of the general public, which is often a stakeholder.

Poor requirements

Criticism: The requirements for most SE projects are incomplete or inconsistent. Some clients have little experience writing requirements. Other clients do not know what they want, and say "I'll know it when I see it". Even experienced clients who know exactly what they want may not precisely articulate their requirements. Clients often expect much more than they write in the requirements. And, requirement documents can describe applications that have no computable or practical solutions.

Response: One response is to avoid all projects with poor requirements, which would avoid most projects. Another response is using agile development and rapid prototyping to clarify project goals and deliver value (the most important requirements) quickly. Embedded systems are special in that they fit inside products that are designed by other engineers, who can sometimes define the software requirements completely.

Rising complexity

Criticism: Critics argue that the complexity of requirements and user expectations have only increased. The probability of failure increases with the size, scope and complexity of the project. Technologies and practices have consistently improved over the years, but the gap between what is expected and what is delivered has not improved.

Response: Rising complexity actually shows the success of practitioners, because demand naturally follows supply. When clients demand more, it shows their belief that their demands will be supplied, based in their relative satisfaction with previously completed projects.

Ongoing change

Criticism: Practitioners continually develop new technologies and practices and use them whenever possible. Critics argue that ongoing change proves that older technologies and practices were failures.

Response: Many view ongoing change as proof that software engineering successfully learns and grows. Traditional engineers also continually develop new technologies and practices.

Ongoing failure

Criticism: Critics argue that incomplete or poorly designed systems are still too common. The early disasters in the field did not prevent subsequent disasters.

Response: No field that strives to do bigger and better projects has ever avoided all failures. Traditional engineers also have ongoing failures: automobiles kill 40,000 people every year in the U.S.; Three Mile Island, Chernobyl, and Bhopal Disaster harmed thousands; Space Shuttles Challenger and Columbia blew up. Although large, reliable software systems can be and have been constructed, software projects that fail during construction or in service are still too common. Ongoing failure is a problem for both traditional engineers and software engineers. Some claim that software engineering is already as predictable and reliable as many fields of engineering, such as space or biological engineering, which have proven to be relatively reliable.

Failure to pinpoint causes

Criticism: Critics argue that unlike traditional engineers (who analyze failures, find precise causes, and set up guidelines to avoid them in the future), software engineers routinely fail to pinpoint causes of failure or delay precisely enough to avoid repeats in the future.

Response: Debugging is the activity of pinpointing the cause of failures in applications. Process improvement includes the activity of pinpointing the cause of process problems. Software engineers routinely pinpoint causes and then use the results to create better languages, databases, processes, and applications.

Nothing new

Criticism: Critics argue that software engineers created nothing on their own, but merely use what computer scientists already know.

Response: Software engineers developed optimizing compilers, make, cvs, extreme programming, scripting, and bug databases on their own, out of necessity. Regardless of who creates or improves technologies and practices, software engineers are bright enough to adopt the best ones.

Anyone can do software engineering

Criticism: Many bright people from other fields (engineers, scientists, business people) write spreadsheet templates or simulations, and eventually switch to writing large applications, and believe that they do software engineering. So, software engineering is not a special skill.

Response: Software engineering is a skill that is refined through long learning and practice. Software engineers are the ones who get the necessary education and experience, and keep up with evolving technologies, practices, and applications. This is true of every skill.

We do not know what software engineering is

Criticism: Software engineering does not yield to the standard ways of categorization, under traditional definitions of engineering (calculus and science). This claim is often made by critics who want to impose their own definitions on everyone else.

Response: We know a lot about what software engineering is. Software engineering is grounded in software engineering technologies and practices, and applications; and in the community of software engineering practitioners. Of course, software engineers continue to disagree about many details.

No "software science"

Criticism: Civil, electrical, mechanical, and chemical engineering build on solid results from physics and chemistry. These results enable assembling complex systems in a principled and systematic way. No corresponding results are available for software: With software, we don't know how to systematically decompose complex systems into parts. Because there is no unified and agreed system for instruction on Software Engineering, it is difficult to evaluate the qualifications of those in the practice.

Response: Software engineering builds on solid results from computer science and information science. These results enable the building of very sophisticated software systems.

No underlying mathematics

Criticism: Many traditional engineering disciplines are based on logic and the mathematical disciplines of arithmetic, algebra, linear algebra and differential calculus, as well as the first principles of science such as Newton's laws, the laws of thermodynamics, Maxwell's equations, etc. Most of the engineering rules-of-thumb that make up traditional engineering derive from the first principles of their respective physical and mathematical foundations. Software engineering has no such first principles that all practitioners agree to, and this makes it harder to justify it as an engineering discipline as opposed to a trade or a craft.

Response: Software engineers uses logic, proof theory, generating functions and wide variety of discrete mathematics when creating programs, as well as when testing programs. Most tools that were created to automate software production, e.g. optimizing compilers, versioning systems, etc, rely on elaborated mathematical models.

No theorems about people and project management issues

Criticism: No theorems explain why one software engineer is more productive than another. No theorems explain why some software projects succeed and others fail. Without such knowledge, engineering is impossible.

Response: Software engineering, like other engineering disciplines, is a complex social activity. No theorems explain why one mechanical engineer is more productive than another. No theorems explain why some civil projects go over budget and fail, for example why the construction of the new metro line to Laval went way over budget, or why 2 space shuttles blew up.

Meaning of success

Criticism: According to a study by the Standish Group in 2000, 28 percent of software projects were complete successes (meaning they were executed on time and on budget), and 23% failed outright.

Response: Many engineering projects fail to live up to expectations: many bridges and buildings run over budget or schedule. Consider that 40% of all space shuttles have blown up and the rest have been out of service for years. Almost all custom housing projects run over budget and schedule. Success rates for software projects are meaningless without context.

Software creation is inherently creative

Criticism: Software is executable knowledge, which is discovered in a creative process, where trial and error, learning, and the ability to challenge one's assumptions are important. The true "construction" phase of software development is already automated by compilers and linkers. The difficult aspects relate to gathering requirements and designing systems. These are more like a craft than a science or engineering task. So, the potential benefit of software engineering is limited to making it easier for developers to try out ideas, to discover errors earlier, and to give them information about the state of a software system.

Response: Many software engineers use agile software development to embody the creative nature of software engineering and to foster learning. The area of requirements engineering looks at how to elicit and capture what a system should do and how this knowledge can be used in software engineering activities.

No consensus

Criticism: In traditional engineering there is a clear consensus how things should be built, which standards should be followed and which risks must be taken care of: If an engineer does not follow these practices and something fails he gets sued. There is no such consensus in software engineering: Everyone promotes their own methods, claiming huge benefits in productivity, usually not backed up by any scientific, unbiased evidence.

Response: Software engineering is a young discipline. As consensus emerges, software engineering will be thought of as a mature discipline.

Summary

Goals of Software Engineering:

Improve the productivity of the programming/development process
Improve the comprehension of the developed software systems
Improve the quality of the software product at all levels
- Reliability
- Effciency (Speed, resource usage)
- User-friendliness (user acceptance)
- Maintainability (comprehensive and flexible design and documentation)
To produce software which is economic and useful and safe for people.

References

NATO Software Engineering Conference. Garmisch, Germany, 7-11 Oct. 1968, Brussels, Scientific Affairs Division, NATO (1969) 231pp.
Wikipedia:Software engineering
Wikipedia:Engineering
Wikipedia:Criticism of software engineering

Introduction to Software Engineering

Contents