Thought of the day: Questions to ask your clients and developers?

There are times when it is a great idea to ask your clients and your developers a few questions about the project you are about to join.

Naturally, you will not be able to ask all the people the questions but you never know.

However, the answers should give you a rough idea of the maturity of the project.

The following table is an overview of the detail & questions you may need from each of the stakeholders. You may determine that additional/different people are needed to fulfil this and some people may have multiple roles but it should be a starting point.

Who (Client)	Information sought
Project Sponsor	Provide an overview of the Objectives and benefits expected What are the Business’ general expectations of the project? How are they being assessed? What are the Business’ technical expectations of the project? How are they being assessed? What are the Business’ security expectations of the project? How are they being assessed? Is there a feature project roadmap and is it shared with the Team?
Product Owner	Provide an overview of the prioritised delivery roadmap What are the Business’ general expectations of the project? What are the Business’ technical expectations of the project? What are the Business’ security expectations of the project? Is there a feature project roadmap and is it shared with the Team? What security requirements has the business placed on the system? What non-functional and performance requirements has the business placed on the system? What is the current and future growth of the system expected to be? Are requirements prioritised using a MoSCoW scheme? As PO are you involved in the Sprint planning phases of the project? Are you involved in the development of the project feature backlog? Are you involved in the development of the sprint planning process? Are technical blockers reported to you? Is there sufficient project information being provided by the team to assist with the creation of further requirements? Provide a walkthrough of the feature backlog and priority Provide insight into the current issue/risks/constraints to the project
Project Manager	Provide an overview of the current SDLC, measures and controls What are the processes & tools for requirements control? Does the project have a Release timeline? What is the process the team is following to control their development? If it is agile can you describe the process? Who is involved in the development of the project feature backlog and whats the process? Who is involved in the development of the sprint planning process and whats the process? As PM how would you describe your role in this SDLC? Does the project have a measurable sprint velocity? As PM are you involved in the Sprint planning phases of the project? How are testing outcomes & blockers/impediments reported and tracked? Are technical blockers reported to you?
Data/Business Analyst Business SME	What is the Business’ general expectations of the project? Provide an overview of the top priority Use Cases for the solution What are the processes the system needs to perform? What is the current quantity of data the system must process? What are the data sources, transformations and outputs the system must perform? (schemas, definitions, data flows etc) What security requirements has the business placed on the system? What performance requirements has the business placed on the system? What is the current and future growth of the system expected to be? How are the requirements communicated to the team? How is system functionality measured against the requirements?
Technical Architect	What is the overall System architecture context? What are the main functional components of the system? What are the processes & tools for requirements control? Are you involved in the development of the project feature backlog? Are you involved in the development of the sprint planning process? What security requirements were placed on the system? What performance requirements were placed on the system? What is the current and future growth of the system expected to be? What architecture requirements were placed on the system? What other non-functional requirements were placed on the system? Is there a Disasters recovery plan for the system? If so how is it implemented architecturally? Is it possible to get access to all existing design notes for the system? How are design choices and patterns communicated with the team for development? Is there a deployment strategy or process for the system? (Infrastructure as code?) Is there a testing/assurance strategy or process for the system? When there are issues with the development are you informed and consulted? When there are issues reported by assurance are you informed and consulted? Prior to the system going operational are you consulted by the Assurance, Operations or development team on how systems should be configured, or any potential issues? When in operation do you have visibility of operational monitoring systems? Once the system is deployed how do you know that it has been deployed according to your specification? Does the system have a ‘Technical Debt log’ or a list of known issues? Provide an overview of the solution architecture Provide details of constraints, dependencies and issues encountered and impacting delivery
Vendor Tech Lead	How are requirements communicated to the team? What is the process the team is following to control their development? If it is agile can you describe the process? Are you involved in the development of the sprint planning process? Is the team aware of the release schedule for the system? Is the team recording development information in some way? If so using which tool? What security requirements were placed on the system? What performance requirements were placed on the system? What architecture requirements were placed on the system? What is the process of reporting progress in development back to project management? What are the development toolsets in use across the team? When source code is developed what coding standards are being followed? When source code is being developed which source code management (SCM) tool is in use? Who has Source code access? When a build of the system is performed what are the build artefacts and how are they stored? What is your SCM versioning & branching & merging strategy? When defects are found, how are they reported, tracked and reported? What are the capabilities of development staff & their relative strengths? Is the team following a BDD/TDD development approach? What level of code review is performed during development? What is the release process for new functionality? Do you produce release notes for each new release? How many environments are used for development? What is the process for procuring cloud development stacks? What are the data sets used during development? How is the development environment secured? Is there a build process for the system? If so what is it? Is the system deployable using automation tools? If so what are they? What level of testing (manual & automated) exists for the system to prove functional & non-functional requirements? When the system is in UAT what information/documentation is requested by Assurance to determine the available functionality? When the system is in LIVE what information/documentation is requested by Operations regarding its operation and deployment? Provide an overview of the solution and the code Provide details of constraints, dependencies and issues encountered and impacting delivery
Security & Compliance	Provide an overview of the prioritised Security & Compliance requirements for the solution When the system requirements are being determined was security consulted? When the system is being developed what involvement does Security have? When the system deployment is being planned is Security consulted? Is Security informed of code related security issues? Is security informed of how the system is tested for compliance with security requirements? When operational how is the security of the system managed? If there is a security issue with the system how are you informed? What is your process for reporting a security breach? Provide an overview of constraints & dependencies for delivery
Assurance Lead	Provide an overview of the testing strategy and framework Provide an overview of the test coverage on the solution and the measures in place What involvement does assurance have in the code-Review process? What level of functional testing does assurance perform on the system? What level of non-functional testing does assurance perform on the system? What proportion of the testing is automated vs manual? Does the system have automated test reports and how are they reported to assurance What is the test plan for penetration testing? How does assurance get involved with the defect management process? Does assurance have any input on the architecture of the system? Once deployed what involvement does Assurance have with the system?
Operations	Provide an overview of the path to live/transition to live process. When the system is being designed what visibility do you have or need of its functional requirements? When the system is being designed what visibility do you have or need of any security requirements? When infrastructure is needed by the Development, Test and support team what is the process they must follow? How are you informed by development of the build artefacts needed for deployment? How do you enable or control the provisioning of cloud resources by the development team? What information do you need regarding the security of the proposed system? What information do you need about the testing process? How are deployment details of the Deployment & Configuration process communicated to you? What is the process for post-deployment system testing? How is performance and systems failure reported? What are the Monitoring processes and how & what is reported? If the system has a bug how is it reported and who does the repair work? What are the system’s supported hours? What is the systems supported availability? What is the DR plan for the system? When in operation does development have access to the system to perform fault analysis?

 Have fun!

How to design a RESTful API architecture from a human-language spec

This is a precise of a separate post: https://www.oreilly.com/learning/how-to-design-a-restful-api-architecture-from-a-human-language-spec?imm_mid=0fa416&cmp=em-prog-na-na-newsltr_20180106


By RESTful API we mean an API that follows Representational State Transfer (REST) architectural style. REST is a very popular approach to building APIs because it emphasizes simplicity, extensibility, reliability, and performance.

Let’s briefly look at each of those concepts to see how using a RESTful architectural approach affects the way your API behaves for its users1 :

• Simplicity: The way to interact with API resources is well-defined and strict: you have a clear and simple path to follow concerning what you can and cannot do. Interaction is stateless: requests have all data needed to move resources to the next state.
• Extensibility: Support for representing the same resources with different formats or even versions, ability to add new resources without changing others, etc.
• Reliability: Clear separation of actions with and without side-effects, the possibility of retrying requests with idempotent actions, etc.
• Performance: Caching made possible via well-defined semantics, scalability made possible because server resources can be isolated in each request due to stateless interaction, etc.

The process for designing a RESTful HTTP API architecture has two steps:

1. Identify nouns and verbs from the system spec
2. Extract URLs and the methods they respond to from those nouns and verbs

With those simple things, we'll have the basis of our API. It will be enhanced with other considerations in the next two articles.

Start with the basics: REST key concepts

Let's review a few key concepts of RESTful APIs as we start reasoning about our own application. First of all, we need to think about the basic elements used in REST:

• Resources - You can think of those as the "nouns" of your system. For example, in a food-delivery service API, the nouns would be restaurant, menu, menu item, restaurant owner, etc. Those nouns are what hold the State, the "S" in REST.
• Representations - Representations are the way API clients see the resources. A RESTful API never hands resources directly to a user. Interactions happen only via representations of the real resource. For example, you can store all the menu items for a restaurant in a database table, but the representations of these items in your API might be a list of names and prices, perhaps filtered to reflect restrictions specified by the user. To do this, we use media types such as JSON and XML. That's why REST is "RE"presentational.
• Actions - Because API clients do not have direct access to resources, they need actions to alter the state of a resource. Those actions are the verbs of our system. They are what "transfer" the state, the "T" of REST.

One of the main authors of HTTP is also the creator of REST. No wonder HTTP has parts that fit very well to these REST basic elements:

• URLs - Uniform resource locators, as the name states, are what people and programs use to locate resources from servers across the Internet. They locate REST resources.
• Media Types - These specify how the data in requests and responses look like, i.e., the REST representations. Currently, the most popular media types are JSON, HTML, and XML.
• Methods - HTTP built-in methods (such as GET, POST, PUT, DELETE) are what we use to implement REST actions. Each HTTP method has characteristics that should be taken into account when implementing the actions. Methods are called over URLs and have data that is represented with media types.

From Spec to RESTful API

We've seen that REST is made of resources, representations, and actions. We've also seen that resources are like nouns and actions are like verbs. So, identifying the nouns and verbs available at a system spec is a good way to start our spec to API process. By reading the spec, we can get the following list of nouns and verbs related to the system use cases. For example an API for a bike rental service, night have these verbs & nouns:

Nouns:

• City
• User
• Station
• Bike

Verbs:

• View (user views the list of stations)
• Rent (user rents a bike)
• Show (app shows the quantity of available bikes at a station)
• Release (station releases a bike)
• Ride (user rides a bike)
• Return (user returns a bike)
• View (user views the history of rents)
• Change (user changes destination station)
• Cancel (user cancels the active rent)

It's easy to convert nouns into HTTP resources with URLs. For example, the noun “station” can become a resource located by the URL path /stations/ in our API. But there are some gotchas in the conversion of verbs to HTTP methods.

A word on HTTP "verbs"

One limitation we have while mapping a spec to a RESTful API is the restricted set of verbs HTTP provides. We can only use the documented HTTP methods: GET, POST, PUT, PATCH, DELETE, etc. As you can see, we don't have a "RENT" verb, even though at first glance our application requires it.

Instead of custom verbs, we must define custom nouns for our RESTful API. Anything can be a noun. Therefore anything can be a resource that's mapped by a URL. Even if it doesn't look like a noun at first sight. "RENT" fits more into a verb, but you can noun-ify it to make it a resource. Think about the "Rent" noun, not the "RENT" verb. Then you'll see that you can "add a Rent". And HTTP fits here because it has the POST verb for adding things. See? Just by changing the way we think about noun and verbs we can adapt our application to HTTP. 

The correct way to rent something via HTTP is to POST a Rent.

Thus, to properly design our RESTful API we need to know really well the meaning of each HTTP method. What are the semantics behind each method? How should we use them? Let's see:

• GET - Should be used to retrieve data from the server. Means "give me a representation of this resource identified by this URL". It's worth noting that we must never use GET to change resources. For this, we have the following other methods.
• POST - Should be used to provide data to the server. Means "process the resource representation as a new subordinate of this URL". In practice, POST is mostly used for creating resources. The client specifies the resource representation in the request body and makes a POST to the URL that holds the collection of resources.
• PUT - Should be used to update existing server resources by replacing their old state with a new one. Means "use this representation to replace the resource identified by this URL". 
• DELETE - Should be used to remove server resources. Means "delete the resource identified by this URL".

By following the semantics of each method, we can design a truly RESTful API.

Conclusions

Before starting any work on an API DO NOT start coding, spend time writing the API down and discussing and agreeing it.

Doing so will save days of lost effort.

For a full analysis of how to convert a particular API see the original article as it details the transition of the Bike Rental API into its methods.

Word of the Day - JBoss

JBoss is a division of Red Hat that provides support for the JBoss open source application server program and related middleware services marketed under the JBoss Enterprise Middleware brand. JBoss is an open source alternative to commercial offerings from IBM WebSphere and SAP NetWeaver.

Red Hat JBoss products are available through a subscription model that includes technical support and long-term maintenance. JBoss products include:

• JBoss EAP - an enterprise application platform for building, deploying and hosting Java applications and services. 
• JBoss Web Server - a Web server built on Apache and Tomcat.
• JBoss Data Grid - an in-memory distributed database.
• JBoss Developer Studio - an Eclipse-based integrated development environment (IDE).
• JBoss Operations Network - a systems management suite for JBoss application environments.
• JBoss Fuse - a modular integration platform with an enterprise service bus (ESB) to support hybrid and public cloud deployments. 
• JBoss A-MQ - a lightweight messaging platform.
• JBoss Data Virtualization - a data virtualization service that allows multiple data sources to be viewed and worked with as if they are a single entity.
• JBoss BRMS - a platform for business rules management, business resource optimization and complex event processing (CEP).
• JBoss BPM Suite - a platform for business process management (BPM).

Java Random

I helped a colleague recently  determine whether Java Random() was worth using in a situation where the client had to be certain that the numbers were truly random.

Our concern was that while a small sample of random numbers (especially while seeded with the current time ) are fit for most purposes they are not appropriate where due dillegence needs to be performed.

The reason being most random number generators are only pseudo random.

So we did some research and asked some of our experts about the suitability of the Java Random function.  Things to note are defined below:

• Most of the documents on the subject are quite old and therefore there may be improvements, but I couldn’t find anything which said it had materially changed.
• It is worth noting that we are looking at Pseudo Random Number Generators and not truly random numbers. For the use cases discussed I don’t believe that would be an issue, but that would not be the case in some gambling and cryptography applications
• Where true random numbers are required, a random number service and/or specific random number hardware is used. I don’t believe this is required in this application [https://api.random.org/json-rpc/1/]
• People have been fairly scathing of Java Random, identifying that the current implementation only uses 17 bits of entropy from the initial seed (i.e. 1 in 131072 starting points) and demonstrates repeating patterns after low numbers of calls to generate numbers (of the order of ~50000). Statistical tests have been used to identify that it doesn’t produce very good random numbers [http://www.alife.co.uk/nonrandom/]
• SecureRandom is a drop in replacement for Random and does produce much better random numbers that pass Statistical tests, but it is 60 times slower than Random [https://dzone.com/articles/java-programmer%E2%80%99s-guide-random]. In the way that the Batch report is using random numbers, it appears that of the order of 500 random numbers would be required. A quick test shows that 500 SecureRandom numbers only takes ~15 Milliseconds to calculate. Which isn’t going to significantly affect the time taken for a result.
• There are libraries that support alternative ways of creating random numbers. These produce good quality pseudo random numbers in less time than the standard Java implementations, but would require additional libraries and dependencies to be managed. In this case I do not believe it would be worth the additional effort. [http://maths.uncommons.org/] [https://www.bouncycastle.org/java.html]

Based on the number of random numbers being generated (maximum of around 500 at a time), Random may well be sufficient.  But given the minimal cost of using SecureRandom it may be worth converting to that instead, to remove even a small possibility of concern.

The AWS Well-Architected Framework

A repost from InfoQ but with a good PDF for later reading.

---

Amazon has published the AWS Well-Architected Framework (PDF), a guide for architecting solutions for AWS, with design principles that apply to systems running on AWS or other clouds.


Amazon has based the AWS Well-Architected Framework on four pillars and a number of design principles as outlined in short bellow.

Security. 

According to Amazon, security in the cloud regards 4 areas - Data Protection, Privilege Management, Infrastructure Protection, Detective Controls – and they recommend the following design principles to strengthen the security of a system:

• Apply security at all levels 
• Trace everything 
• Automate responses to security events 
• Secure the system at the application, data and OS level 
• Automate security best practices 

Reliability. 

This pillar represents a system’s ability to “recover from infrastructure or service disruptions, dynamically acquire computing resources to meet demand, and mitigate disruptions such as misconfigurations or transient network issues.” The areas covered by reliability are Foundations, Change Management and Failure Management, and the paper recommends the following design principles:

• Test recovery procedures 
• Automatically recover from failure 
• Use horizontal scalability to increase availability 
• Automatically add/remove resources as needed 

Efficiency. 

This is about efficient use of CPU, storage, and database resources. It also covers the space-time trade-off, i.e. consuming more memory and disk space to solve a problem quicker or using less resources but solving it in more time. The design principles recommended are:

• Use advanced technologies 
• Deploy the system globally for lower latency 
• Use services rather than servers 
• Try various configurations to find out what performs better 

Cost Optimization. 

This is evidently about optimizing costs, eliminating unneeded or suboptimal resources. Cost optimization should consider matching supply with demand, using cost effective resources, keeping an eye on expenses, and lowering the costs over time. This can be done by:

• Transparently attribute expenditure 
• Use managed services 
• Buy computing resources in the cloud rather than hardware 
• Use the cloud for its pay-as-you-go policy 
• Do not invest in data centers 

The framework includes a list of questions to be used when assessing a proposed architecture, such as “How are you encrypting and protecting your data at rest?” or “How are you planning your network topology on AWS?”. The authors also provide their recommendations for addressing each of the problems mentioned in these questions, some of them applying only to AWS, others being valid for any cloud computing architecture.


This article has extracted the main points from the 56-pages whitepaper on architecting solutions for the cloud. For a detailed explanation of all the best practice.

Version Numbering - Redux

The issue

I have been looking at version numbering for a project where the developers had stuck at 0.0.1-SNAPSHOT for 12 months and were starting to encounter issues with getting the correct JARS for their projects from their binary repository.

The solution was to use the features that are present in Maven and Jenkins to assist them in their processes.

I covered the basics of version numbering in an other post (May 2012), but that just gives the version strategy for Apache projects. Since then I have come across Semantic Versioning which is a great a almost definitive source on how you should work with versions.
However this does not cater for the way Maven actually treats version numbers, it is just compatible with it..

A good version number has a number of properties:

• Natural order: it should be possible to determine at a glance between two versions which one is newer
• Maven support: Maven should be able to deal with the format of the version number to enforce the natural order
• Machine incrementable: so you don't have to specify it explicitly every time

What does Maven do?

For reference, Maven version numbers are comprised as follows: <MajorVersion>.<MinorVersion>.<IncrementalVersion>-<BuildNumber | Qualifier>. Where MajorVersion, MinorVersion, IncrementalVersion and BuildNumber are all numeric and Qualifier is a string. If your version number does not match this format, then the entire version number is treated as being the Qualifier. [See]
If all the version numbers are equal, the qualifier is compared alphabetically. "RC1" and "SNAPSHOT" and sorted no differently to "a" and "b". As a result, "SNAPSHOT" is considered newer because it is greater alphabetically. See this page as a reference.
The issue on may projects is how to manage the versions and how to do so with out breaking the Maven format which will cause Maven to treat your versions as just the Qualifier which is a string (not good). Note that a.b.c-RC1-SNAPSHOT would be considered older than a.b.c-RC1, because of text comparisons.

What to use as an Incremental version number?

I think it is reasonably straight forwards for a project to determine the major and minor version as they often come directly from the business drivers for the project.
The Incremental version can give some issues as the business may not be interested and it is the developers that need it for tracking purposes.
Therefore the incremental number has be be meaningful to them.
So usefull numbers could be the database schema version, the sprint number the feature set that is being implemented (although this can be hard if you have several teals working in parallel).

The simplest way is to start at 1 and for the team leads to determine when to increase the number.

Whether to use as a Qualifier or a BuildNumber?

There appear to be several schools of thought on this and Maven simply fits with them all.
The Apache method is to not have either and using the SNAPSHOT qualifier allows you to follow this pattern.
However, SNAPSHOT does not allow your developers to know what to use the version for.
JBoss has qualifiers (alpha[n], beta[n], release candidate 'CR[n]' and Final) with optional numbers. [See].
The OSGI specification adds a further complication as does the eclipse numbering which are
MajorVersion, MinorVersion, IncrementalVersion.TIMESTAMP[-Mn]
MajorVersion, MinorVersion, IncrementalVersion.CR[n]
MajorVersion, MinorVersion, IncrementalVersion.Final
There are only two qualifiers.  The first one is for the milestone releases, and the qualifier starts with a numeric timestamp.  The project can use a timestamp as shown below as it will sort according to the compareTo method of the String class just like any other qualifier. ie YYYYMMDD
Optionally, if for some reason there is a need to make two releases in the same day, you can add a sequence number to the end of the timestamp. The next part of the qualifier is the milestone number, where M stands for milestone, and n is the milestone number.
After all the milestone releases that have added the various functional pieces are complete, and the project and any sub-projects that are integrated are at least at a candidate release stage, then a CR release will follow.  Just like in the traditional model, there may be multiple CR releases depending on the feedback from the community.

It is the above approach minus the milestone number I would advise (assuming the 'IncrementalVersion' is controlled by the development team.

Setting up Maven

Making sure your version numbers are incremented can be a pain in the arse but there is  a plugin for Maven that helps you mange this.

The Maven 'Release' Plugin

The Release plugin [See] is helpful but not essential. It is used to help a developer release a project with Maven, saving a lot of repetitive, manual work. Its best usage is to allow the developer to update their version number with out effort and correctly.

It is added to the maven project as follows:

 <project>  
     ...  
     <build>  
         <plugins>  
             ...  
             <plugin>  
                 <groupId>org.apache.maven.plugins</groupId>  
                 <artifactId>maven-release-plugin</artifactId>  
                 <version>2.5.2</version>  
             </plugin>  
             ...  
         </plugins>  
         ...  
     </build>  
     ...  
 </project>  

easy!

This allows the developer to issue a command as follows:

 mvn -B release:update-versions  

... and the version will be updated to the next increment. They then only need to commit it as part of their code.
It is always the last part of the version number that is incremented and it even works if you have a text qualifier such as CRn (see above).

Maven Versions Plugin

This plugin is much more useful when it comes to controlling your release via your CI server. [See]
(In these examples I'm going to quote Jenkins but this process should work for others.)
(I am also not sure if Maven 3.1+ doesn't include this plugin.)

Unlike the previous plugin that increments the version, this will allow a specific version number to be set in the POM, like this:

 mvn versions:set -DnewVersion=0.1.1-RC1  

Where '0.1.1-RC1' is an example of a version number.

The process

Now to tie this together.
The process we want is:

1. Jenkins checks out the latest revision from SCM (Subversion, Mercurial, Git, ...)
2. Release Plugin transforms the POMs with the new version number
3. Maven compiles the sources and runs the tests
4. Release Plugin commits the new POMs into SCM
5. Maven publishes the binaries into the Artifact Repository

Prerequisites: Jenkins with a JDK and Maven configured, and both the Git and the Workspace Cleanup, and Parameterized Trigger Plugin  plugins installed.
We're going to start by creating a new Maven job and making sure we have a fresh workspace for every build:

After assigning you SCM, the next step is to set the version upon checkout.
A good version number is both unique and chronological. We're going to use the Jenkins BUILD_NUMBER (the current build number, such as "153") as it fulfills both these criteria wonderfully.
We could use BUILD_ID which is such as "2005-08-22_23-59-59" (YYYY-MM-DD_hh-mm-ss).
or even the Git commit number using GIT_REVISION.
This is configured as follows:

and in the build step:

And that's it! Every time this job is run, a new release is produced, the artifacts will be deployed and the source code will be tagged. The version of the release will be the BUILD_NUMBER (or GIT_REVISION) of the Jenkins project. Nice and simple.

Worth a read: UX is not UI

I love UX and spend hours thinking about how to make a great UX (no really).
So when I read this article I just had to share.

UX has become a neologism. When something has “good UX” it is an implied meaning of having the core components of UX (research, maybe a persona, IA, interaction, interface, etc etc…). It’s not really necessary or desirable to tack the word design onto the end anymore. It’s a distraction and leads people down a parallel but misguided path… the path to thinking that UX = User Interface Design.

See: http://www.helloerik.com/ux-is-not-ui  and the graphic from the article here.

The lost world of java.nio

The O'Reilly book on the subject

I was ask today to help some colleagues with a comms issue on a Telephony application.

When they ask a whole set of questions and problems started to come to mind. What happens with the Garbage Collect, how fast is the code, does it mess packets.

So I had a little look around for some information on Java NIO, it seems to have been kicking butt since Java 1.4 and we have all ignored it.

Indeed for Java 7 it has a new friend … java.NIO.2.

I promised to do some research so here are my results:

• This article seems to contain the best information: http://www.javaworld.com/article/2078654/java-se/java-se-five-ways-to-maximize-java-nio-and-nio-2.html, its concise and clear.
• To get to grips with Socket IO this article http://www.javaworld.com/article/2853780/java-app-dev/socket-programming-for-scalable-systems.html is also very nice (and by coincidence on the same site). 
• If you really wish to dealve deep into the details then this old article from Sept 2001, gives all the nitty-gritty http://www.javaworld.com/article/2075575/core-java/core-java-master-merlin-s-new-i-o-classes.html
• · And then there is the ubiquitous entry from O’Reilly on high performance servers (http://archive.oreilly.com/pub/a/onjava/2004/09/01/nio.html) 
• In the docs for Files.copy (http://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html) was an example of downloading an html page to a file.
  It’s very clean and also means I no longer need to depend on an external jar to perform many file operations.
  

Good bye commons-io. You’ve served me well, but I have a new ‘friend’!

In fact if you look at the Channels package (http://docs.oracle.com/javase/7/docs/api/java/nio/channels/Channels.html) it will allow you to convert to your familiar Stream classes while still sending data via the nio classes to your socket (AsynchronousSocketChannel).

And if we really-really want to know how to do this stuff:


NIO.2 Cookbook - http://www.javaworld.com/article/2882984/core-java/nio2-cookbook-part-1.html

I may never use Java.IO again!

A good book : Software Measurement and Estimation: A Practical Approach

Estimating the size of a software development is a black art.
This book attempts to put some science onto the topic.

It attempts to give you methods for determining how big your project is compared to others, what is your code quality like and how efficient are your developers.

It contains many gems such as the relative inefficiencies of coding in different languages. For example VB has a "gear ration" of 42 while java & C# have a GR of 59 which means you must write 40% (59/24=1.41) more code to perform the same task in C# or Java. Which shouldn't come as a surprise to any one.

It even suggests that once you get a couple of projects into a development you might be able to develop a ratio between your specification size (user stories) and the number of NLOC (none-commented lines of code). A wooly estimate but perhaps a usefull one if pushed for an estimate, by a manager.

It goes on to argue that while LOC can be measured easily it is Functional Point Analysis (FPA) which will perhaps yield the best estimation results.
To me this is a better approach as it ties in with the Agile use of User Stories which can be estimated into their relative difficulties. However in Scrum the developers tend to use planning poker which is a method I am not personally in favor of as I prefer a more empirical approach and the use of historical data (see this post on using historical data).

The next stage this book suggests is then converting your Functional points to LOC (which is useful if parts are using different tech) and comparing it to your historical results ie your productivity.

The point here is DON'T GUESS!

Work it out!


"When you can measure what you are speaking about, and can express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind." - Lord Kelvin


"Science/It works bitches." - xkcd

Finally some tools to calculate LOC:

• http://metrics.sourceforge.net/
• And a big listhttp://www.monperrus.net/martin/java-metrics/

And some books:

Software Quality Metrics Overview

Implementing Automated Software Testing - Continuously Track Progress and Adjust Accordingly

A final thought on functional point estimation.

In my experience, the size and complexity of a project is a function of a quite limited number of parameters, that compares from a project to another, whatever the sector or domain area.

Each application usually has a bit of specificity in some way (workflow complexity, external connection challenges, dynamic user interfaces, customized advanced business calculations or multi-tenancy for SaaS applications), so you can determine the impact that element has in terms of development cost is relatively constant from a project to another.

When determining the complexity of the project you combine several complexity factors, you may also find ratios that help you deal with this complexity and you should be able to evaluate a project based on the following numbers:

• number of entities
• number of simple business rules
• number of advanced business rules (cross-entity for example)
• number of user interface elements (screens, web pages)
• number of reports
• number of external interfaces
• number of batch calculations or processes
• number of technology variations for components (rich-client, web page, mobile, database providers, cloud systems

These numbers must then be combined with an uncertainty level to create a range for use with three point estimation.

But more on that an other day.!

Estimation by stuffing things into boxes

I liked this article by Johannes Brodwall so much I thought I would steal it.
The original is here.

---

I’ve (sic:Johannes Brodwall) started using an approach for software project estimation that so far is proving to be fairly transparent, quick and reliable. I’ve observed that within a reasonable degree of variation, most teams seems to complete about one “user-relevant task” per developer per calendar week.

There are so many theoretical holes in my argument that there’s no point trying to cover them all. The funny thing is that it seems to work fairly well in practice. And to the degree that it’s dirty, it’s at least quick and dirty. The only thing I will address is that one of these “user relevant tasks” is smaller than a typical application feature.

Most importantly: Most teams never get it right on the first try. Or they spend too long gold-plating everything. Or both.

This article shows an example of estimating a fictive project: The Temporary Staffing System.

The high-level scope

Let’s say that our organization has come up with the following vision:

For a temporary employment agent who wants to match candidates
to client needs, the Temporary Staffing System is an
interactive web application, which lets them register and
match candidates and positions. Unlike competing systems
this lets us share selective information with our clients.

We come up with the following flow through the application:

1. A new company wants to hire a skilled worker for a temporary position
2. Administrative user adds the client details to the system
3. Administrative user adds client logins to the system
   (perhaps we also should let the clients log in with LinkedIn etc?)
4. Client logs into the application and completes new position
   description, including skill requirements
5. Temp agency adds a worker to the system
6. Temp agency proposes the worker to a position registered by a client
   (in the future, the worker may register themselves!)
7. Client gets notified of new proposals (via email)
8. Client views status of all open positions in the system
9. External to the system: Client interviews candidate, request further
   information and makes a decision whether to hire or not
10. Client accepts or rejects the worker in the system
11. As worker performs work, they register their time in the system
12. At the end of a billing period, the system generates billing information
    to accounting system
13. At the end of a salary period, the system generates salary information
    to the accounting system

Some of these steps may be one user story, some may be many.

The top of the backlog

We choose some of the most central parts of the scope to create the beginning of the backlog. In order to accommodate for the learning as we go along, the first draft of our backlog may look like this:

1. Experimental create open position
2. Experimental list positions
3. Simplified create open position
4. Simplified list positions
5. Complete create open positions
6. Complete list positions

An “experimental” version of a story is a functionality trivial version that touches all parts of the technology. In the case of these two stories, perhaps we have the application leave the logged in client as a hard coded variable. The story may only include writing some of the fields of the positions, maybe only title and description.

The Simplified version may add more complex properties, such as skills from a skill list or it may add filters to the list.

The complete version should be something we’re prepared to put in front of real users.

By revisiting a feature like this, we have the chance to get the feedback to create a good feature without gold-plating.

Continuing the backlog

We add enough of the other stories to the backlog to cover an interesting part of the scope:

• …
• Basic create client account
• Complete create client account
• Basic login admin user
• Basic login client user
• Complete login client user
• Basic add worker
• Complete add worker
• Basic propose worker for position
• Complete propose worker for position
• Complete confirm worker for position
• Basic enter timesheet (in this version temp agency enters on behalf of worker)
• Experimental billing report
• Basic billing report
• Basic salary report

This functionality should be enough to have a pilot release where some clients and workers can be supported by the new system. Or we may complete the backlog with complete versions of all functionality, worker login and perhaps a polished version of a feature or two.

Adding the non-functional tasks

There are some tasks that we want to plan some extra time for. I generally find that many of these tasks are tasks that customers understand quite well:

• …
• Attend training on CSS (the team is rusty in design skills)
• Basic layout and styling of web pages
• Complete layout and styling of web pages
• Polished layout and styling of web pages (they want it really nice)
• Locate slowest pages and make some performance improvements
• Deploy solution to target platform
• Deploy demo version to wider set of stakeholders
• Deploy pilot version
• Exploratory test of complete flow

Planning the project

In this example project, we have five team members plus a coach/project manager on half-time. Since our team will be working in pairs, we want to work on three functional areas per week. This way, we can avoid huge merge conflicts. The team agrees to plan for five stories per week, but only three the first week, because things generally go slower. Here is the top of the completed backlog:

• Week 1: Experimental create open position
• Week 1: Experimental list positions
• Week 1: Attend training on CSS
• Week 2: Simplified create open position
• Week 2: Simplified list positions
• Week 2: Basic create client account
• Week 2: Basic layout and styling of web pages
• Week 3: Basic login client user
• Week 3: Deploy solution to target platform
• Week 3: Basic add worker
• Week 3: Basic propose worker for position
• Week 3: Basic enter timesheet (temp agency enters on behalf of worker)
• Week 4: Experimental salary report
• Week 4: Complete layout and styling of web pages
• Week 4: Complete create open positions
• Week 4: Complete list positions
• Week 4: Deploy demo version to wider set of stakeholders
• …
• Week 6: Exploratory test of complete flow
• Week 7: Deploy pilot version

Presenting the plan

Working through the list gives us a complete timeframe of just over 6 weeks for full feature set for the pilot release. To cover realities of life, we probably want to plan for at least one week of slack or even more, depending on the strength of our commitment and the consequences of being wrong.

This gives a plan indicating 7 weeks times 5 people at 40 hours per week plus a 50% project manager at 20 hours per week or a total of 1540 hours.

I generally find that after a pilot release (or even before it), things change a lot. So I don’t invest much time into planning this.

Tracking the development

The true strength of a plan like this appears when you start running the project. Each week, the team will report on which stories they completed. This allows us to adjust the plan to actual progress.

On the flip side, the weekly planning comes down the team and the customers agreeing on the definition of a story. The vagueness of “basic add worker” is by design! But the team should agree on what they mean by “experimental”, “simplified”, “basic”, “complete” and “polished”.

Conclusions

In this article, I have showed a quick and accurate way of coming up with a project forecast, complete with time and cost estimates. It’s easy to see and react to deviations from the forecast.

A few critical critical observations support this methodology:

• I never believe a developer estimate other than “by the end of the day” or “by the end of the week”. (Don’t talk to me about hours!)
• Estimating in hours is a silly way to get to project costs. Any hour-based estimate is always prodded and padded before magically turning into cost. Instead, estimate number of features, feature per week and cost by week.
• Visiting a feature multiple times lowers total cost due to less gold-plating and investment of in poorly understood areas. It also improves the success of the final feature
• The ambition of a feature (that is, how many times we will visit it) is a more reliable indication of cost than developer gut feeling

I’ve left many questions on the table, for example: What about architecture? What is meant by a “simplified” user story? How to deal with deviations from the forecast? Feel free to grill me for details in the comments to the article.

“So what will it cost?” Using this simple method to lay out your project forecast week by week, you can give a better answer next time someone asks.

Published at DZone with permission of Johannes Brodwall, author and DZone MVB. (source)

Scalability

For the second time in two months I have been asked about "Big-O" functions when talking about software efficency.
You may have seen people quote a metric for the efficency of a piece of code as O(n) or O(n^2) etc. where 'n' is the amount of data you have got.
But can you recall waht it means ... have you seen it sicne your comuting course?
Well I hadn't and it is along time since then (don't ask)!

Due to experience I know that a nested set of loops is going to be O(n*m) but this stuff realy counts now that we are increasinly seeing huge data sets.

I have regularly seen processes rescently that sill take tens of minutes to run and that is despite multiple processors and fast ones.

So what is going on?

Well lets look at an over view of the basics of Big O notation (plagarised from the Perlmonks)

Common Orders of Growth
O(1) is the no-growth curve. An O(1) algorithm's performance is conceptually independent of the size of the data set on which it operates. Array element access is O(1), if you ignore implementation details like virtual memory and page faults. Ignoring the data set entirely and returning undef is also O(1), though this is rarely useful.
O(N) says that the algorithm's performance is directly proportional to the size of the data set being processed. Scanning an array or linked list takes O(N) time. Probing an array is still O(N), even if statistically you only have to scan half the array to find a value. Because computer scientists are only interested in the shape of the growth curve at this level, when you see O(2N) or O(10 + 5N), someone is blending implementation details into the conceptual ones.
Depending on the algorithm used, searching a hash is O(N) in the worst case. Insertion is also O(N) in the worst case, but considerably more efficient in the general case.
O(N+M) is just a way of saying that two data sets are involved, and that their combined size determines performance.
O(N²) says that the algorithm's performance is proportional to the square of the data set size. This happens when the algorithm processes each element of a set, and that processing requires another pass through the set. The infamous Bubble Sort is O(N²).
O(N•M) indicates that two data sets are involved, and the processing of each element of one involves processing the second set. If the two set sizes are roughly equivalent, some people get sloppy and say O(N²) instead. While technically incorrect, O(N²) still conveys useful information.
"I've got this list of regular expressions, and I need to apply all of them to this chunk of text" is potentially O(N•M), depending on the regexes.
O(N³) and beyond are what you would expect. Lots of inner loops.
O(2^N) means you have an algorithm with exponential time (or space, if someone says space) behavior. In the 2 case, time or space double for each new element in data set. There's also O(10^N), etc. In practice, you don't need to worry about scalability with exponential algorithms, since you can't scale very far unless you have a very big hardware budget.
O(log N) and O(N log N) might seem a bit scary, but they're really not. These generally mean that the algorithm deals with a data set that is iteratively partitioned, like a balanced binary tree. (Unbalanced binary trees are O(N²) to build, and O(N) to probe.) Generally, but not always, log N implies log₂N, which means, roughly, the number of times you can partition a set in half, then partition the halves, and so on, while still having non-empty sets. Think powers of 2, but worked backwards.

2¹⁰ = 1024
log₂1024 = 10

The key thing to note is that log₂N grows slowly. Doubling N has a relatively small effect. Logarithmic curves flatten out nicely. It takes O(log N) time to probe a balanced binary tree, but building the tree is more expensive. If you're going to be probing a data set a lot, it pays to take the hit on construction to get fast probe time.
Quite often, when an algorithm's growth rate is characterized by some mix of orders, the dominant order is shown, and the rest are dropped. O(N²) might really mean O(N² + N).

 
This is quite nice as you can see it gives you an idea of the efficency from just the SHAPE of the curve an not the absolute values.

Now those nested loops are understandable in context and can be expressed as a O(n^2) or O(n^3) curve.

But what if your program is using a set of code from the Java collections package for example?
You don't want to go digging through it to just get an idea of the efficency.
Fortunatly there is a cheet sheet that you can wander over to and pull out the efficencies of those off the shelf fuctions you are using.

See: http://bigocheatsheet.com/.

For example you have a HashSet with 'n' elements that is compared to 'm' data itesm s what is the efficency?

The worst case search for a hashset is O(n) and if you are having to search it 'm' times you will ahave an efficency of m * O(n) or ... O(m*n).

Easy!



So how do you improve your effiency?

Apart from having a good look to see if you can improve your algoarithm it is now possible to get things done quicker if they are done in parallel.
If this is done by multiple machines it is known as "Scaling Out" or if it is done by increasing the power of your machine then it is known as "Scaling up" (also known respectivly as Scaling Horizontally or Scalling Vertically).

But if you add twice the threads doers it get twice as fast?

If you read the article on WIKIPEDIA (http://en.wikipedia.org/wiki/Scalability), you will reach the part on Amdahl's law.

This gives a nice equasion:

Where α (Alpha) is the percentage of the process that can be done in parralel,
While P is the number of processors.

The equation has the following properties:

• As  α tends to ZERO ... so nothing can be done in prallel ... the function tends to 1.
• As α tends to ONE ... so it can all be done in prallel ... the function tends to P. 

In the first case no number of extra processors will help you ... while in the other it is scalable and adding more processors will increase efficency.

The worked example on Wikipedia gives an case where 30% of the code ca be done in parallel and compares 4 vs 8  processors increasing efficency by 20% only!

The conclusion to all this is unless software devlopers improve their code efficency we are all just going to have to throw more 'tin' at the problem and only get a minor improvement for out money.

Class package naming and a more pragmatic approach to MVC.

During a trawl of posts on a software development postings on LinkedIn (or some where) I stumbled across a blog post that set me thinking.

A typical collaboration of the MVC components

We have all be come very familiar with the concept of Model-View-Controller (MVC), that I think as developers we don't give it any more thought and when it doesn't really fit what we are doing we blithely continue.
So after some thought I thought I would annotate some of my current thoughts.
Microsoft tinkered with the pattern when they talked about MVVM and MVP but they are basicly just variants on MVC and not clearly adopted outside their software.

So the alternates with my favourite last:

Business Process and Data with MVC (Model-View-Controller)

In her blog Lea Hayes talks about using MVC but adopts the concept of splitting the model part into a business process and data [See blog].

Business Process and Data with MVC (Model-View-Controller)

This is a nice idea as it shows how to consider a controller that actually does a task instead of just going directly to a view.

Unfortunately this approach does not absolutly resolve all the issues as it relegates the controller to the role of "postman" or router. It also does not address what happens when the service has changed the data. 

It also has no good monika to remember it by ... MS-VC ... nope!

Lea Hayes included a link on her blog to a second concept...

'VESPA': A better MVC

This post on a further varient from Bennett McElwee's blog to me offers more promise as sooner than redefining the paradigm as MVVM and MVP do, it adds definition to the elements of MVC.  I agree largely with VESPA but I have only one refefinition (see later).

The Original VESPA pattern

What VESPA does is to refactor sooner than to redefine the MVC pattern.

The reasoning is that as a designer you can communicate the MVC pattern to your developers but you may need to add some definition to how they start coding.

Essentially you break the M & C into four parts and effectively create a M=(SE) & C=(AP) which gives us SE-V-AP ... or rearranged as VESPA so you can communicate it.

But what are the parts:

The Model becomes "Store" and "Entity"

In MVC data is still stored and used on a view but there may not be a direct 1-1 mapping between stored data and the entities shown on the views. Which is where the MVVM comes in.

In VESPA your Store classes deal with persisting your data and contain any business logic on how to combine elements.

The Entity classes are those that are presented on the view but are not stored, they are however responsible for collating Store objects into a usable form.

An example of StoreB objects could be a set of JAXB objects talking to a web service, while the Entities  are a set of POJOs that are used to place a degree of separation between the application and the data sources.

The controller becomes "Action" and "Presenter"

In MVC the controller is still in charge but there are two variants.

There is the Presenter which is the simple "mail-man" controller I spoke of earlier. It just finds the data entities and passes it to the view.

The other type is the Action, which is called "Actor" in the original VESPA but for me this conflicts with the Actor term from UML [See]. It is also similar to the "Service" Lea Hayes speaks about, but I think it is more of a controller concept.

My input to VESPA

My input to VESPA

I agree with the original concept of VESPA 95% and I am adopting it into my projects.

My differences from the original are:

1. Its Action not "Actor";
2. The Action and Presenter only deal with the Entities.

Class packages and VESPA

Finally being a bit anal about package naming I would break down the classes into the following packages so you can find them.

Using a base package of mcnought.myproject you get:

• Actors are in mcnought.myproject.controller.actor;
• Presenters are in mcnought.myproject.controller.presenter;
• Stores are in mcnought.myproject.model.store;
• Entities are in mcnought.myproject.model.entities;
• View are in mcnought.myproject.view.

I hope this helps you.