Overview of Software Engineering
CS 330
Spring 2007
Key Ingredients in successful organizations
People
Process
Technology
A better view
Process and Technology supporting people
People
Processes
Technology
Pyramids are stable. Wedges are not!
What is software?
Computer programs and associated documentation
Software products may be developed for a particular customer or may be developed for a general market Software products may be
Generic/COTS - developed to be sold to a range of different customers Custom- developed for a customer according to their specification
Engineering
Engineering is
The application of scientific principles and methods to the construction of useful structures & machines
Examples
Mechanical engineering Computer engineering Civil engineering Chemical engineering Electrical engineering Nuclear engineering Aeronautical engineering
Software Engineering
The term is 35 years old: NATO Conferences
Garmisch, Germany, October 7-11, 1968 Rome, Italy, October 27-31, 1969
The reality is it is finally beginning to arrive
Computer science one the scientific basis
Years of studies/experience/statistics provide basis too
Many aspects have been made systematic
Methods/methodologies/techniques Languages Tools Processes
Why Engineer Software ?
The problem is complexity Many sources, but size is a key:
Mozilla contains 3 Million lines of code UNIX contains 4 million lines of code Windows 2000 contains 108 lines of code
Second is role and combinatorics of state Third is uncertainty of inputs and their timing Fourth is the continuing changing environment and demands.
Software engineering is about managing all the sources of complexity to produce effective software.
Software Engineering in a Nutshell
Development of software systems whose size/complexity warrants team(s) of engineers
multi-person construction of multi-version software [Parnas 1987]
Scope
study of software process, development/management principles, techniques, tools and notations
Goal
production of quality software, delivered on time, within budget, satisfying customers requirements and users needs
What does a software engineer do?
Software engineers should
adopt a systematic and organised approach to all aspects of software development. use appropriate tools and techniques depending on
the problem to be solved, the development constraints and the resources available
Understand and communicate processes for improved software development within their organization Be effective team members and/or leaders. Can be very technical or more managerial depending on organizational need.
What is the difference between software engineering and computer science?
Computer Science
theory fundamentals
Software Engineering
the practicalities of developing delivering useful software
is concerned with
Computer science theories are currently insufficient to act as a complete underpinning for software engineering, BUT it is a foundation for practical aspects of software engineering
What is the difference between software engineering and system engineering?
Software engineering is part of System engineering System engineering is concerned with all aspects of computer-based systems development including
hardware, software and process engineering
System engineers are involved in system specification, architectural design, integration and deployment
Difficulties?
SE is a unique brand of engineering
Software is malleable Software construction is human-intensive Software is intangible and generally invisible Software problems are unprecedentedly complex Software directly depends upon the hardware
It is at the top of the system engineering food chain
Software solutions require unusual rigor Software state means behaviors can depend on history. Software has discontinuous operational nature
Software Engineering Software Programming
Software programming
Single developer Toy applications Short lifespan Single or few stakeholders
Architect = Developer = Manager = Tester = Customer = User
One-of-a-kind systems Built from scratch Minimal maintenance
Software Engineering Software Programming
Software engineering
Teams of developers with multiple roles Complex systems Indefinite lifespan Numerous stakeholders
Architect Developer Manager Tester Customer User
System families Reuse to amortize costs Maintenance accounts for 60%-80% of overall development costs
Economic and Management Aspects of SE
Software Engineering is about improved ROI (can be Capital and/or Social ROI) Software production = development + maintenance Maintenance costs 60%-80% of all (successful) development costs
20% corrective (12%-16% total costs) 30% adaptive (18%-24% total costs) 50% perfective (30-40% total costs)
Quicker development is not always preferable
higher up-front costs may defray downstream costs poorly designed/implemented software is a critical cost factor in system cost and delays
Relative Costs of Fixing Software Faults
200
30 10 1
Requirements
2
Specification
3
Planning
4
Design Implementation Integration Maintenance
Mythical Man-Month by Fred Brooks
Published in 1975, republished in 1995
Experience managing development of OS/360 in 1964-65
Central argument
Large projects suffer management problems different in kind than small ones, due to division in labor Critical need is the preservation of the conceptual integrity of the product itself
Central conclusions
Conceptual integrity achieved through chief architect Implementation achieved through well-managed effort software developers are not interchangeable work units.
Brooks Law
Adding personnel to a late project makes it later
Software Engineering: From Principles to Tools
TOOLS
METHODOLOGIES
METHODS AND TECHNIQUES
PRINCIPLES
Software Qualities
Qualities are goals in the practice of software engineering, and directly relate to many of the guiding principles.
External vs. Internal qualities Product vs. Process qualities
Software Qualities
Critical Quality Attributes
Correctness Maintainability Dependability Usability Reliability
Other Attributes
Completeness Compatibility Portability Internationalization Understandability Scalability Robustness Testability Reusability Customizability Efficiency
External vs. Internal Qualities
External qualities are visible to the user
reliability, usability, efficiency (maybe), robustness, scalability
Internal qualities are the concern of developers
they help developers achieve external qualities verifiability, maintainability, extensibility, evolvability, adaptability, portability, testability, reusability
Product vs. Process Qualities
Product qualities concern the developed artifacts
maintainability, performance, understandability,
Process qualities deal with the development activity
products are developed through process maintainability, productivity, predictability
Some Software Qualities
Correctness
ideal quality established w.r.t. the requirements/specification absolute
Reliability
statistical property probability that software will operate as expected over a given period of time/inputs relative
Some Software Qualities (cont.)
Robustness
reasonable behavior in unforeseen circumstances subjective a specified requirement is an issue of correctness; an unspecified requirement is an issue of robustness
Usability
ability of end-users to easily use software extremely subjective
Some Software Qualities (cont.)
Understandability
ability of developers to easily understand produced artifacts internal product quality subjective
Verifiability
ease of establishing desired properties performed by formal analysis or testing internal quality
Some Software Qualities (cont.)
Performance
equated with efficiency assessable by measurement, analysis, and simulation
Evolvability
ability to add or modify functionality addresses adaptive and perfective maintenance problem: evolution of implementation is too easy evolution should start at requirements or design
Some Software Qualities (cont.)
Reusability
ability to construct new software from existing pieces must be planned for occurs at all levels: from people to process, from requirements to code
Interoperability
ability of software (sub)systems to cooperate with others easily integratable into larger systems common techniques include APIs, distributed programming interfaces (CORBA, DCOM), plug-in protocols, etc.
Some Software Qualities (cont.)
Scalability
ability of a software system to grow in size while maintaining its properties and qualities assumes maintainability and evolvability goal of component-based development
Process Principles
Prescribes all major activities Uses resources, within a set of constraints, to produce intermediate and final products May be composed of sub-processes Each activity has entry and exit criteria Activities are organized in a sequence Has a set of guiding principles to explain goals Constraints may apply to activity, resource or product
Software Development Stages
Requirements Analysis & Specification
Conceptual/System/Architectural Design Detailed/Program Design Implementation/Coding Unit & Integration Testing System Testing/Validation System Delivery/Deployment Maintenance
Note there are many variations on the names. You are responsible for the main categories above (an on the next pages)..
Software Lifecycle Models
Waterfall Model V Model Phased Development Model
Incremental Model
Prototyping Model Spiral Model
Software Development Lifecycle Waterfall Model
Requirements
Plan/Schedule Replan/Reschedule
Design
Implementation Integration Validation Deployment
V Model
REQUIREMENTS ANALYSIS Validate requirements OPERATION & MAINTENANCE
ACCEPTANCE TESTING SYSTEM DESIGN SYSTEM TESTING
Verify design
PROGRAM DESIGN
UNIT & INTEGRATION TESTING
[Pfleeger 98]
CODING
Phased Development Model
DEVELOPERS
Development systems
Build Release 1
Build Release 2
Build Release 3
Time
USERS
Use Release 1
Use Release 2
Use Release 3
Production systems
[Pfleeger 98]
Software Development Lifecycle Incremental Model
Requirements Version 1: Design Complete General Design Implementation Integration Validation Deployment
Requirements Design Implementation Version 2: Integration Design/Implement first set Validation of planned new features. Deployment Note overlap with V1 schedule Version 3: Design/Implement second set of planned new features
Requirements Design Implementation Integration Validation Deployment
Prototyping Model
Listen to Customer
Build/Revise Mock-Up
Customer Test-drives Mock-up
[Pressman 97]
Prototyping Model
LIST OF REVISIONS LIST OF REVISIONS LIST OF REVISIONS
revise prototype
user/ customer review
PROTOTYPE REQUIREMENTS
PROTOTYPE DESIGN
PROTOTYPE SYSTEM
TEST
SYSTEM REQUIREMENTS (sometimes informal or incomplete)
DELIVERED SYSTEM
[Pfleeger 98]
Spiral development
Process is represented as a spiral rather than as a sequence of activities with backtracking. Each loop in the spiral represents a phase in the process. No fixed phases such as specification or design loops in the spiral are chosen depending on what is required. Risks are explicitly assessed and resolved throughout the process.
Spiral model of the software process
Spiral model sectors
Objective setting
Specific objectives for the phase are identified.
Risk assessment and reduction
Risks are assessed and activities put in place to reduce the key risks.
Development and validation
A development model for the system is chosen which can be any of the generic models.
Planning
The project is reviewed and the next phase of the spiral is planned.
Evolutionary development
Exploratory development
Objective is to work with customers and to evolve a final system from an initial outline specification. Should start with well-understood requirements and add new features as proposed by the customer.
Throw-away prototyping
Objective is to understand the system requirements. Should start with poorly understood requirements to clarify what is really needed.
Evolutionary development
Evolutionary development
Problems
Lack of process visibility; Systems are often poorly structured; Special skills (e.g. in languages for rapid prototyping) may be required.
Applicability
For small or medium-size interactive systems; For parts of large systems (e.g. the user interface); For short-lifetime systems.
Component-based software engineering
Based on systematic reuse where systems are integrated from existing components or COTS (Commercial-off-the-shelf) systems. Process stages
Component analysis; Requirements modification; System design with reuse; Development and integration.
This approach is becoming increasingly used as component standards have emerged.
Reuse-oriented development
Component-Based Development
Develop generally applicable components of a reasonable size and reuse them across systems Make sure they are adaptable to varying contexts Extend the idea beyond code to other development artifacts Question: what comes first?
Integration, then deployment Deployment, then integration
Different Flavors of Components
Third-party software pieces Plug-ins / add-ins Applets Frameworks Open Systems Distributed object infrastructures Compound documents Legacy systems
Process iteration
System requirements ALWAYS evolve in the course of a project so process iteration where earlier stages are reworked is always part of the process for large systems. Iteration can be applied to any of the generic process models. Two (related) approaches
Incremental delivery; Spiral development.
Incremental delivery
Rather than deliver the system as a single delivery, the development and delivery is broken down into increments with each increment delivering part of the required functionality. User requirements are prioritised and the highest priority requirements are included in early increments. Once the development of an increment is started, the requirements are frozen though requirements for later increments can continue to evolve.
Incremental development
Incremental development advantages
Customer value can be delivered with each increment so system functionality is available earlier. Early increments act as a prototype to help elicit requirements for later increments. Lower risk of overall project failure. The highest priority system services tend to receive the most testing.
Extreme programming
An approach to development based on the development and delivery of very small increments of functionality. Relies on constant code improvement, user involvement in the development team and pairwise programming. Covered in Chapter 17
Software Development Lifecycle Waterfall Model
Requirements
Plan/Schedule Replan/Reschedule
Design
Implementation Integration Validation Deployment
Software specification
The process of establishing what services are required and the constraints on the systems operation and development. Requirements engineering process
Feasibility study; Requirements elicitation and analysis; Requirements specification; Requirements validation.
Requirements
Problem Definition Requirements/Specification
determine exactly what the customer and user need (maybe want) Requirements develop a contract with the customer Specification say what the software product is to do
Difficulties
client is computer/software illiterate (no idea what is doable) client asks for wrong product (want vs need) client is computer/software literate (specifies solution not need) specifications are ambiguous, inconsistent, incomplete
Studies have shown that the percentage of defects originating during requirements engineering is estimated at more than 50 percent. The total percentage of project budget due to requirements defects is 25 to 40 percent.
The requirements engineering process
Software design and implementation
The process of converting the system specification into an executable system. Software design
Design a software structure that realises the specification;
Implementation
Translate this structure into an executable program;
The activities of design and implementation are closely related and may be inter-leaved.
Design process activities
Architectural design Abstract specification Interface design Component design Data structure design Algorithm design
The software design process
Structured methods
Systematic approaches to developing a software design. The design is usually documented as a set of graphical models. Possible models
Object model; Sequence model; State transition model; Structural model; Data-flow model.
Architecture vs. Design
[Perry & Wolf 1992]
Architecture is concerned with the selection of architectural elements, their interactions, and the constraints on those elements and their interactions necessary to provide a framework in which to satisfy the requirements and serve as a basis for the design. Design is concerned with the modularization and detailed interfaces of the design elements, their algorithms and procedures, and the data types needed to support the architecture and to satisfy the requirements.
Architecture/Design
Requirements/Specification Architecture/Design
architecture: decompose software into modules/objects/components with interfaces design: develop module/object/component specifications (algorithms, data types) and communication details maintain a record of design decisions and traceability specifies how the software product is to do its tasks
Difficulties
miscommunication between module designers design may be inconsistent, incomplete, ambiguous How to achieve a requirement may be unknown
Planning/Scheduling
Before undertaking cost of development, need to estimate the costs/sizes of various steps
Estimate Code size Estimate tools needed Estimate personnel
Often Done after Architecture and before rest of design, but revised again after full design. Develop schedule for aspects of project lifecycle If doing predictive/quantitative SE, build on past experience, considering how to improve process.
Implementation & Integration
Design Implementation
implement modules; verify that they meet their specifications combine modules according to the design specifies how the software design is realized
Difficulties
module interaction errors order of integration may influence quality and productivity
Programming and debugging
Translating a design into a program and removing errors from that program. Programming is a personal activity - there is no generic programming process. Programmers carry out some program testing to discover faults in the program and remove these faults in the debugging process.
The debugging process
Software validation
Verification and validation (V & V) is intended to show that a system conforms to its specification and meets the requirements of the system customer. Involves checking and review processes and system testing. System testing involves executing the system with test cases that are derived from the specification of the real data to be processed by the system.
Verification and Validation
Analysis
Static Science Formal verification Informal reviews and walkthroughs Dynamic Engineering White box vs. black box Structural vs. behavioral Issues of test adequacy
Testing
The testing process
Testing stages
Component or unit testing
Individual components are tested independently; Components may be functions or objects or coherent groupings of these entities.
System testing
Testing of the system as a whole. Testing of emergent properties is particularly important.
Testing with customer data to check that the system meets the customers needs.
Acceptance testing
Testing phases
Quality Assurance
Done as part of each step Reduce costs by catching errors early. Help determine ambiguities/inconsistencies Help ensure quality product. 200
30
Requirements
Specification
3
Planning
4
Design
10
Implementation Integration Maintenance
Deployment
Completed End-User Documentation
Separate from Developer documentation
Installation Process(es) Customer test procedures Support Processes (help desk, etc) Trouble Tracking Repair/rework to address bugs Regression testing (as bugs are fixed)
Maintenance & Evolution
Operation Change
maintain software during/after user operation determine whether the product still functions correctly
Difficulties
Rigid or fragile designs lack of documentation personnel turnover
Software evolution
Software is inherently flexible and can change. As requirements change through changing business circumstances, the software that supports the business must also evolve and change. Although there has been a demarcation between development and evolution (maintenance) this is increasingly irrelevant as fewer and fewer systems are completely new.
System evolution
Why I include CASE Tools
Computer Aides Software Engineering tools support good SE processes (e.g. UML) Some tools absolute requirement for scaling e.g. build and configuration management. Integrated CASE (ICASE) tools embody good processes and improve productivity (E.g. Rational tool set) Some tools (e.g. debuggers, Purify) do almost impossible for humans.
But.. Tools change
No SE tools from my first 3 jobs exist (except Fortran/C languages) I use regularly use 3 SE tools from my next set of jobs. Other tools I learned have been replaced with similar but expanded concepts.. Understanding today;s tools gives a basis for learning future ones.
ICASE Design Tools
Rational Rose and Rational Unified Development. From UML drawing to code and back. Generates stubs and eventually testing code. Supports multiple languages
Car
public class Car { public Driver theDriver; /** * @roseuid 3EAFF17E035B */ public Car() { Associations are } }
Driver
public class Driver { /** * @roseuid 3EAFF53F02FD */ public Driver() {
implemented as reference attributes. No explicit role name defined so, Rose adds automatically a role name to the code: theDriver
} }
Templates for the default constructors are provided. (Similar for methods/members when given in the class diagram.)
Configuration Management
CM is a discipline whose goal is to control changes to large software through the functions of
Component identification Change tracking Version selection and baselining Managing simultaneous updates (team work) Build processes with automated regression testing Software manufacture
CM in Action
1.0 1.1
1.2
1.3 1.4 1.5
2.0 2.1
4.0
2.2
3.0 3.1
Build Tools
Necessary for large projects. Keep track of what depends upon on what, and what needs recompiled or regenerated when things change. Important even for small 1-person projects as soon as you have multiple files. Can do much more than just compile, can generate document (if using code-based docs), generate manufactured code (e.g. SOAP interfaces), even send emails or suggest alternatives.
E.g. in our IUE project, edit some files compile was one in seconds, edit another and a rebuild taking days would be needed. If more than 30 files impacted, our make process recommend a new branch to avoid conflicts!
Debugging Tools
How do you see what the code is really doing (not what it seems it should do)? How to you see what happened to code during compiler optimization? How do you find/track down the cause of Segfault/GFP in code youve never seen before? How can you test various possibilities without generating special code or recompiling. How do you track down a memory leak?
Tools, workbenches, environments
CASE techn olo g y T ls oo Wor kb en ch es Envir ments on
Ed ito rs
Co mp ilers
File comp ar ators
Integ rated en vir ments on
Process -cen tr ed en vir ments on
An alys is and des ig n
Pro gramming
T estin g
Mu lti-metho d workb en ch es
Sin gle-meth od workb en ch es
General-pu rp os e workb en ch es
Lang uage-sp ecific workb en ch es
The Rational Unified Process
A modern process model derived from the work on the UML and associated process. Normally described from 3 perspectives
A dynamic perspective that shows phases over time; A static perspective that shows process activities; A practive perspective that suggests good practice.
RUP phase model
Phase it erat ion
Incept ion
Elaborat ion
Const ruct ion
Transit ion
RUP phases
Inception
Establish the business case for the system.
Elaboration
Develop an understanding of the problem domain and the system architecture.
Construction
System design, programming and testing.
Transition
Deploy the system in its operating environment.
RUP good practice
Develop software iteratively Manage requirements Use component-based architectures Visually model software Verify software quality Control changes to software
Static workflows
Work flow Business modelli ng Requirements Analysis and design Implementation Description The business processes are modelled using business use cases. Actors who interact with the system are identified and use cases are developed to model the system requirements. A design model is created and documented using architectural models, component models, object models and sequence models. The components in the system are implemented and structured into implementation sub-systems. Automatic code generation from design models helps accelerate this process. Testing is an iterative process that is carried out in conjunction with implementation. System testing follows the completion of the implementation. A product release is created, distributed to users and installed in their workplace. This supporting workflow managed changes to the system (see Chapter 29). This supporting workflow manages the system development (see Chapter 5). This workflow is concerned with making appropriate software tools available to the software development team.
Test
Deployment Configuration and change management Project management Environment
Computer-aided software engineering
Computer-aided software engineering (CASE) is software to support software development and evolution processes. Activity automation
Graphical editors for system model development; Data dictionary to manage design entities; Graphical UI builder for user interface construction; Debuggers to support program fault finding; Automated translators to generate new versions of a program.
Case technology
Case technology has led to significant improvements in the software process. However, these are not the order of magnitude improvements that were once predicted
Software engineering requires creative thought - this is not readily automated; Software engineering is a team activity and, for large projects, much time is spent in team interactions. CASE technology does not really support these.
CASE classification
Classification helps us understand the different types of CASE tools and their support for process activities. Functional perspective
Tools are classified according to their specific function.
Process perspective
Tools are classified according to process activities that are supported.
Integration perspective
Tools are classified according to their organisation into integrated units.
Functional tool classification
Tool type Planning tools Editing tools Change management tools Configuration management tools Prototyping tools Method-support tools Language-processing tools Program analysis tools T esting tools Debugging tools Documentation tools Re-engineering tools Examples PERT tools, estimation tools, spreadsheets T ext editors, diagram editors, word processors Requirements traceability tools, change control systems Version management systems, system building tools Very high-level languages, user interface generators Design editors, data dictionaries, code generators Compilers, interpreters Cross reference generators, static analysers, dynamic analysers T est data generators, file comparators Interactive debugging systems Page layout programs, image editors Cross-reference systems, program re-structuring systems
Activity-based tool classification
Re-en g ineering tools Testin g too ls Debu gg ing too ls Prog ram analy sis to ols Lang uage-p ro ces sing too ls Meth od s up po r t to ols Proto ty ping too ls Co nfiguration management to ols Ch an ge man ag emen t too ls Do cu men tatio n too ls Ed iting too ls Planning to ols
Sp ecif icatio n
Design
Implemen tatio n
V erification and V alidatio n
CASE integration
Tools
Support individual process tasks such as design consistency checking, text editing, etc.
Workbenches
Support a process phase such as specification or design, Normally include a number of integrated tools.
Environments
Support all or a substantial part of an entire software process. Normally include several integrated workbenches.
Boults view of SE
SE must balance risks in software development process:
Risks of error in
requirements specification, design, implementation, and integration
Risks of exceeding available resources Risks of being late on delivery or missing the market
Dont let push for formality dominate your process. Dont let push for expedience destroy your process.
Software Process Qualities
Process is reliable if it consistently leads to highquality products Process is robust if it can accommodate unanticipated changes in tools and environments Process performance is productivity Process is evolvable if it can accommodate new management and organizational techniques Process is reusable if it can be applied across projects and organizations
Assessing Software Qualities
Qualities must be measurable/quantifiable Measurement requires that qualities be precisely defined Improvement requires accurate and consistent measurements For most SD groups, qualities are informally defined and are difficult to assess
Software Engineering Axioms
Adding developers to a project will likely result in further delays and accumulated costs The longer a fault exists in software
the more costly it is to detect and correct the less likely it is to be properly corrected
Up to 70% of all faults detected in large-scale software projects are introduced in requirements and design
detecting the causes of those faults early may reduce their resulting costs by a factor of 200 or more
Basic tension of software engineering
better, cheaper, faster pick any two! functionality, scalability, performance pick any two!
Want/Need Managements buy in to formal SE process. If you dont document your process, you dont have one!
Boehms Spiral Model
DETERMINE GOALS, ALTERNATIVES, CONSTRAINTS EVALUATE ALTERNATIVES AND RISKS
Risk analysis
4
Risk analysis
Risk analysis Risk analysis Budget
4
1 1
Budget
Budget
Budget
Prototype
1
Proto type 2
Proto type 3
Proto type 4 Detailed design
start
Requirements, life-cycle plan
Concept of operation
Code Unit test Implementation plan Acceptance test System test
PLAN
DEVELOP AND TEST
Key points
Software processes are the activities involved in producing and evolving a software system. Software process models are abstract representations of these processes. General activities are specification, design and implementation, validation and evolution. Generic process models describe the organisation of software processes. Examples include the waterfall model, evolutionary development and componentbased software engineering. Iterative process models describe the software process as a cycle of activities.
Key points
Requirements engineering is the process of developing a software specification. Design and implementation processes transform the specification to an executable program. Validation involves checking that the system meets to its specification and user needs. Evolution is concerned with modifying the system after it is in use. The Rational Unified Process is a generic process model that separates activities from phases. CASE technology supports software process activities.
