Showing posts with label Maven. Show all posts

Monday, 11 July 2016

Git - Submodules

It often happens that while working on one project, you need to use another project from within it. Perhaps it’s a library that a third party developed or that you’re developing separately and using in multiple parent projects. A common issue arises in these scenarios: you want to be able to treat the two projects as separate yet still be able to use one from within the other.

Git addresses this issue using submodules. Submodules allow you to keep a Git repository as a subdirectory of another Git repository. This lets you clone another repository into your project and keep your commits separate.

One good example of where this is appropriate is the JSON java libary located at https://github.com/stleary/JSON-java which is just the Java source files and not even arranged in a project structure.

To cope with this I used git-sub-modules and built a maven project around it.

Adding the sub-module

To add the sub-module I first created my maven project as normal.
Then I created the folder /src/main/java/org/json to contain my link to the other project.
Adding the submodule is then as easy as opening a command prompt on that folder and issuing this command:

git submodule add https://github.com/stleary/JSON-java.git

You will now have a .gitmodules file in your project.
Add it to your source control to record what you have linked to.
As easy as that!

What are your sub-modules

Later you may not recall where your sub-modules are so from the project route issue this command:

git submodule status

This will tell you the folders with sub-modules.

Getting maven to play along!

It is a good idea to ensure that you are using the correct code and that you have any changes in place when building.
When you update the project your git client may ask & check sub-modules ... it may not.
So adding the following to your POM.xml ensures that it is added.

  <plugin>
      <groupId>org.codehaus.mojo</groupId>
      <artifactId>exec-maven-plugin</artifactId>
         <version>1.2.1</version>
         <executions>
             <execution>
                 <phase>initialize</phase>
                  <id>git-submodules-init-update</id>
                     <goals><goal>exec</goal></goals>
              </execution>
          </executions>
          <configuration>
              <executable>git</executable>
              <arguments>
                  <argument>submodule</argument>
                  <argument>update</argument>
                  <argument>--init</argument>
                  <argument>--recursive</argument>
              </arguments>
          </configuration>
   </plugin>

This causes Maven to execute an executable command.

In this case the sub-module init command which runs maven and initializes the content based on the project's .gitmodules file.

And that is all it takes!

Friday, 27 May 2016

“Could not Calculate Build Plan” A problem with Maven and Eclipse?

Overview

When switching projects there is a chance that you may encounter an issue with your Maven pom.xml file in Eclipse.

This, if you are lucky, manifests as “Failed to authenticate with Proxy” and if you are not as a simple red error at the top of the window containing the pom.xml, which says “CoreException: Could not calculate build plan: Plugin org.apache.maven.plugins:maven-compiler-plugin:3.1…” or something similar.

The cause of this problem is mixed.

However the root-cause is the same … the IPL proxy.

Somewhere in all the project dependencies is one that is an https and that will trigger the proxy & its infamous man in the middle attack.

Fortunately all the failure scenarios have the same cause and are relatively easy to fix (once you know how).

How does this appear?

As an error on the POM.xml screen

The first way this may manifests is often when you update your project immediately after opening a new Eclipse project.

You may get this error message:

There is a message telling you about “multiple problems”; ignore this.

The dialogue behind shows the real reason, but you may not see it for what it is.

Your POM.XML window also is decorated with an error:

Clicking it simply reveals more confusion and woe.

It will tell you that org.apache.maven.plugins:maven-compiler-plugin has failed to resolve but when you check your Maven repository it is there.

In fact when you run Maven from the command line there does not seem to be a problem.

To make things worse there is a second possible error:

As an error within the POM.xml

Sometimes the error appears in the Project Explorer and as a simple red cross next to lines of the POM.xml when viewing the source.

For example:

What the @&%$ is going on?

What is happening?

So something is broken in Eclipse when it uses Maven, but Maven is ok; therefore the issue is in Eclipse …. WRONG!

Actually the issue is with Maven because Eclipse is actually attempting to do a “force update” of the project dependencies, plus its own verification of other dependencies.

The issue is that the majority of the repositories are accessed via http and minority are via HTTPS.
You may be operating within a corporate network with a proxy that needs to inspect all traffic. Therefore it performs a man-in-the-middle attack on all HTTPS traffic and requires you to pass your login credentials to it so that you can access the HTTPS sites. So the client (you) still sees HTTPs and the server still sees HTTPs, vut our listener can inspect everything.

This arrangement can upset JAVA when it notices the certification is one it does not trust.

What are the causes?

The four causes identified so far are:

Your credentials, stored in the Maven settings, are wrong and need changing.
You have changed your Java version and no longer are using a version that has the IPL certificate added to it.
You are no longer using the specific maven settings that existed for a project;
You no longer have valid user credentials settings in your profile to access the correct mirror.
Speak to your SysOps guys and have your creds checked.

How do I fix it?

Do not …

Firstly do not initially attempt to resolve it by altering the Eclipse settings as this may make things worse.

The problem is with probably with Maven and not Eclipse.

This doesn’t mean that your copy of Eclipse is fully working it simply isn’t responsible for this issue.

So…

Check your project from the command-line

It is possible that your project will build fine from the command line.

mvn clean package -DskipTests

If this works then move on to “Check which Maven installation Eclipse is using…”.

If not then as long as the error is not just a code build error but is clearly a “can’t resolve dependency” issue, then the problem may be with your repository & there may simply be a broken dependency.

So check it using:

mvn -U clean package -DskipTests

This will force an update of the repository and download new files.

If the project is broken still this doesn’t help move on to “Clear your Maven Repository”.

Clear your Maven Repository

It is possible that your repository is in a “broken” state and will not update correctly because there is a library that is waiting to be updated. Therefore your local repository is somehow mucked up for release jars as opposed to snapshots (-U and --update-snapshots only update snapshots), you can purge your local repo of all of its update flags. Open a command prompt in your repository folder and then delete all the flags using:

forfiles /m *.lastUpdated /s /c "cmd /c del @file"

Then try the “mvn -U clean package –DskipTests” on your project.

If this doesn’t work try the following:

mvn dependency:purge-local-repository

You should see all the dependencies maven needs as standards downloading.

Now rebuild using “mvn -U clean package –DskipTests”.

Failing that there is one nuclear option which is to totally delete your repository and rebuild.

So when you have found out where your local repository is … delete it; then rebuild.

There should be no issues at this point as maven should load all its files via http.

So proceed to “Check which Maven installation Eclipse is using…”.

Check which Maven installation Eclipse is using…

One issue could be if you seeing, especially if your copy of Eclipse is new, is you could be using the embedded copy of Maven that the M2E plugin ships with.

If this is the case got to WindowàPreferencesàMavenàInstallations and add your Maven installation as your active installation.

If you change it check your project … Close eclipse, delete your maven local repository & re-open Eclipse & force an update of the project.
If not fixed then check “Are you using the correct Maven settings files?”.

Are you using the correct Maven settings files?

On the screen WindowàPreferencesàMavenàUser Settings;

They should be Global Settings pointing to the folder where Maven is installed & User Settings should point to an .m2 folder in your user directory.
i.e.

If they are not the correct files (which can happen if this is a new Eclipse install) update the paths.

Then … Close eclipse, delete your maven local repository & re-open Eclipse & force an update of the project.

If not fixed then check “Is Eclipse using the same Java version?”.

Is Eclipse using the same Java version?

It is possible that your maven install & your eclipse install are using different Eclipse settings.

First check maven by running:

mvn –version

Which will give you something like this:

Now check in Eclipse.

The JVM in use can be found by looking in HelpàAboutàInstallation Details, Configuration Tab which gives you:

If they don’t match; ensure your JAVA_HOME is set correctly and your PATH is picking up the correct Java install.

If this doesn’t fix the issue then we have to “Diagnose the Eclipse Issue.”.

Diagnose the Eclipse Issue.

At this point you should have a project which works on the command-line but not in eclipse.

You should find that the project will build in Eclipse if you click on the pom.xml file and use Run AsàMaven Install. This is because the project is being built using the same JVM as Eclipse and on the Command-line.

However there is an error & the key issue here is:

The requested resource is on a https site and access to a HTTPs site is blocked by the firewall.

It is possible to configure Maven to authenticate its self with the proxy (see “A not authenticated with Proxy error”). However, the correct methodology is described in the next section.

A secondary, problem here is not with the project dependencies but with “Plugin Dependencies”. These are being read by the Eclipse IDE.

At this point it is best to check whether the correct repositories are being loaded into Eclipse.

This can be found by viewing the Maven repositories loaded by Eclipse in its “Maven View” panel.

To get this view got to WindowàShow ViewàOther.. then select “Maven Repositories”.

If you only see “Central (https://repo.maven.apache.org/maven2)” then it is likely you do not have your profile set correctly.

If this is the case proceed to “Configure your profiles”.

Configure your profiles

In the example above the system is configured to look at the internal mirror.

There should be settings for your project and they should be established as your active profile.

This example uses generic settings with the example of using a Sonatype Nexus server as the Mirror.

In your <home>/.m2/settings.xml file ensure that you have a profile defined as follows:

<id>Nexus</id>

<id>Nexus-Maven-Public-Assets</id>

<name>Maven Public Assets</name>

<url>http://example.com:8888/nexus/content/groups/public/</url>

</repository>

</repositories>

<id>Nexus-Plugins-Maven-Public-Assets</id>

<name>Maven Public Assets</name>

<url>http://example.com:8888/nexus/content/groups/public/</url>

</pluginRepository>

</pluginRepositories>

</profile>

…

<activeProfile>Nexus</activeProfile>

</activeProfiles>

In this example two repositories are added, both pointing at a mirror of the Public code on the fictional example.com Nexus mirror. It is the second repository that is the important one as it relates to the Plugins that are failing to load in Eclipse.

This should fix all your Eclipse issues but if not proceed to “Do you have a mirror set up in your settings?”.

Do you have a mirror set up in your settings?

A mirror is different to a repository. With Repositories you specify from which locations you want to download certain artefacts, such as dependencies and maven-plugins. Repositories can be declared inside a project, which means that if you have your own custom repositories, those sharing your project easily get the right settings out of the box. However, you may want to use an alternative mirror for a particular repository without changing the project files.

Why use a mirror

Some reasons to use a mirror are:

1. There is a synchronized mirror on the internet that is geographically closer and faster

2. You want to replace a particular repository with your own internal repository which you have greater control over.

3. You want to run a repository manager to provide a local cache to a mirror and need to use its URL instead

Within IPL we do have a mirror so an argument could be made for using the mirror configuration.

One side of this argument is all traffic via Maven should go via the mirror so it can be recorded and used to analyse the content of the project.

The problem with this approach is it is very easy to set them up to capture everything sooner than specific traffic.

The current methodology (also described here) is to mirror everything using ‘*’, sooner than a specific mirror for a public site such as ‘central’. This would mean that your mirror has to hold everything.

So the counter proposal is to set up specific profiles or add the mirror directly to the project in which was described in the previous section.

Configure the mirror

The method of defining mirror settings is documented here: http://maven.apache.org/guides/mini/guide-mirror-settings.html

The usage guide states:

“Specifies a repository mirror site to use instead of a given repository. The repository that this mirror serves has an ID that matches the mirrorOf element of this mirror. IDs are used for inheritance and direct lookup purposes, and must be unique across the set of mirrors.”

It is advisable that the mirror settings are located in the same settings file as the localRepository settings. This is because files will have been copied to from the mirror to the local repository and when the mirror changes it are these files that will alter.

<settings>
...
<mirrors>
<mirror>
<id>UK</id>
<name>UK Central</name>
<url>http://uk.maven.org/maven2</url>
<mirrorOf>central</mirrorOf>
</mirror>
</mirrors>
...
</settings>

Where the mirrorOf node is important in selecting what is drawn from the mirror.
In the above case all files are fetched.

Getting a “not authenticated with Proxy” error

You shouldn’t be stuck with this error if you have followed the previous steps.

However if you are the two causes of an error like this:

1. Incorrect proxy credentials;

2. An issue with JAVA JVM.

Do you have the correct proxy settings?

Within your maven settings you should have a proxy settings section that looks something like this:

<settings>
.
.
<proxies>
<proxy>
<id>example-proxy</id>
<active>true</active>
<protocol>http</protocol>
<host>proxy.example.com</host>
<port>8080</port>
<username>proxyuser</username>
<password>somepassword</password>
<nonProxyHosts>www.google.com|*.example.com</nonProxyHosts>
</proxy>
</proxies>
.
.
</settings>

Check the settings:

Is the password right!?

The username must … must … start with MYDOMAIN\\ and not just be your account name or only have a single back-slash.
It is vital to check that you don’t have more than one proxy profile with active=true;
It is vital that the protocol=http and not https as it must match the protocol of the proxy not what is being served;

If you change anything save, and close Eclipse, delete your maven local repository & re-open Eclipse & force an update of the project.

Got proxy settings established but now it’s a ‘certification error’?

You just got an error that looks like this:

Whoa, what the heck is that?

Why would java not be able to connect to my internal repository?

To make sense of the error and what it's actually complaining about, it's necessary to understand a little bit about how SSL works. The server in question — and any server which connecting to might result in the error message above — is protected by HTTPS, which is HTTP with SSL added on. SSL was designed to protect against a lot of different security problems, one of the most complex of which is the man-in-the-middle attack. For various reasons, internet traffic is particularly susceptible to active attacks whereby a malicious party pretends to be the server you're trying to talk to and intercepts your communications. To guard against this, SSL mandates (at the risk of slightly oversimplifying) that the server present a certificate that identifies it as the true bearer of the hostname you're trying to connect to; maven.2xoffice.com in this case. Of course, in and of itself, this provision is next to useless — after all, any attacker who can intercept your communications and masquerade as the target server can just as easily forge a certificate claiming to be the correct server.

As it turns out, this was a well-studied problem in cryptography circles in the mid 90's when SSL was designed — the solution is a Public Key Infrastructure (PKI). In this system, a handful of trusted parties are authorized to digitally sign certificates — in effect, "vouching for" the legitimacy of the bearer of the certificate. Such trusted parties are called certificate authorities

With this brief background, the error message "unable to find valid certification path to requested target" begins to makes some sense — what Java (by way of Maven) is trying to tell you is that the server presented a certificate, and the certificate did identify itself as the rightful bearer of the hostname maven.2xoffice.com. Furthermore, the certificate was properly digitally signed — unfortunately, not by a legitimate certificate authority.

So, what makes a certificate authority a "legitimate" one? Every SSL-capable client has its own answer; browsers, for example, have a list of trusted certificate authorities. They're identified by their own certificates (more specifically, by the secure hash of their certificate's contents). In Chrome, for instance, you can see a list of trusted certificate authorities' certificates by going into Settings->Manage Certificates and seeing a list of several trusted "root" certificates.

Any certificate signed by one of these roots will be trusted; untrusted ones will result in a warning message.

Java, on the other hand, doesn't have a "Settings" tab; instead, it has a setup folder. Specifically, $JRE_HOME/lib/security. Here, there's a file named cacerts that lists all of the trusted root certificate authorities. You can view this list using the keytool that comes with the JDK.[1]

This is where the version of Java in use with Eclipse really starts to matter.

As long as your copy of Java has the IPL certificate & Eclipse is pointing to the right JDK, it will work[2].

Installing the IPL Certificate

If you don’t have a copy of your company certificate, ask your sysOps guys & save it as certnew.cer;
Go to your java_home\jre\lib\security folder;
(Windows) Open admin command line there using 'cmd' and CTRL+SHIFT+ENTER
Run keytool to import certificate:

..\..\bin\keytool -import -trustcacerts -keystore cacerts -storepass changeit -noprompt -alias corpCert -file path\to\certnew.cer

Now this particular java JVM will have the Certificate.

This process needs to be repeated for all your JVMs.

Uninstalling the IPL Certificate

Should there be an issue where the certificate is wrong or needs to be removed then the process is as follows:

Go to your java_home\jre\lib\security folder;
(Windows) Open admin command line there using 'cmd' and CTRL+SHIFT+ENTER
Run keytool to delete certificate:

..\..\bin\keytool -delete -keystore cacerts -storepass changeit -noprompt -alias corpCert

Testing the Corporate Certificate

Should want to test whether the certificate is working then this class can be used to test it:

/**

* A Certificate test which checks for HTTP/HTTPS access using the current JVM.

package com.example;

import java.io.DataInputStream;

import java.net.URL;

import java.net.URLConnection;

public class CheckCert {

private static final String PROXY_HOST = "example.com";

private static final String PROXY_PORT = "8888";

/**

* @param args

public static void main(String[] args) {

System.out.println("JAVA VERSION:\t"+System.getProperty("java.version"));

System.out.println("JAVA HOME:\t"+System.getProperty("java.home"));

System.out.println("JAVA RUNTIME:\t"

+System.getProperty("java.runtime.version"));

// doConnect("http://example.com/");

System.out.println("USE HTTP");

doConnect("http://repo.maven.apache.org/maven2");

System.out.println("USE HTTPS");

doConnect("https://repo.maven.apache.org/maven2");

System.out.println("USE HTTPS - WITH PROXY SETTINGS");

setProxySettings();

doConnect("https://repo.maven.apache.org/maven2");

System.out.println("DONE.");

}

public static void setProxySettings() {

System.clearProperty("http.proxyHost");

System.setProperty("http.proxyHost", PROXY_HOST);

System.setProperty("http.proxyPort", PROXY_PORT);

System.setProperty("https.proxyHost", PROXY_HOST);

System.setProperty("https.proxyPort", PROXY_PORT);

System.setProperty("java.net.useSystemProxies", "true");

}

private static void doConnect(String urlText) {

System.out.println("-------------------------------------------");

System.out.println("Connecting to " + urlText);

try {

URL myURL = new URL(urlText);

URLConnection myURLConnection = myURL.openConnection();

myURLConnection.setUseCaches(false);

myURLConnection.connect();

System.out.println("Connected to " + myURLConnection);

DataInputStream di

= new DataInputStream(myURLConnection.getInputStream());

int max = 100;

byte[] b = new byte[max];

if (-1 != di.read(b, 0, max)) {

System.out.println("DATA:\n" + new String(b));

}

} catch (Throwable ex) {

ex.printStackTrace();

System.err.flush();

}

System.out.println("-------------------------------------------");

}

You can use this class to verify the JVM can access a HTTPS site as long as it is told to use the proxy.

Tuesday, 28 July 2015

Version Numbering - Redux

The issue

I have been looking at version numbering for a project where the developers had stuck at 0.0.1-SNAPSHOT for 12 months and were starting to encounter issues with getting the correct JARS for their projects from their binary repository.

The solution was to use the features that are present in Maven and Jenkins to assist them in their processes.

I covered the basics of version numbering in an other post (May 2012), but that just gives the version strategy for Apache projects. Since then I have come across Semantic Versioning which is a great a almost definitive source on how you should work with versions.
However this does not cater for the way Maven actually treats version numbers, it is just compatible with it..

A good version number has a number of properties:

Natural order: it should be possible to determine at a glance between two versions which one is newer
Maven support: Maven should be able to deal with the format of the version number to enforce the natural order
Machine incrementable: so you don't have to specify it explicitly every time

What does Maven do?

For reference, Maven version numbers are comprised as follows: <MajorVersion>.<MinorVersion>.<IncrementalVersion>-<BuildNumber | Qualifier>. Where MajorVersion, MinorVersion, IncrementalVersion and BuildNumber are all numeric and Qualifier is a string. If your version number does not match this format, then the entire version number is treated as being the Qualifier. [See]
If all the version numbers are equal, the qualifier is compared alphabetically. "RC1" and "SNAPSHOT" and sorted no differently to "a" and "b". As a result, "SNAPSHOT" is considered newer because it is greater alphabetically. See this page as a reference.
The issue on may projects is how to manage the versions and how to do so with out breaking the Maven format which will cause Maven to treat your versions as just the Qualifier which is a string (not good). Note that a.b.c-RC1-SNAPSHOT would be considered older than a.b.c-RC1, because of text comparisons.

What to use as an Incremental version number?

I think it is reasonably straight forwards for a project to determine the major and minor version as they often come directly from the business drivers for the project.
The Incremental version can give some issues as the business may not be interested and it is the developers that need it for tracking purposes.
Therefore the incremental number has be be meaningful to them.
So usefull numbers could be the database schema version, the sprint number the feature set that is being implemented (although this can be hard if you have several teals working in parallel).

The simplest way is to start at 1 and for the team leads to determine when to increase the number.

Whether to use as a Qualifier or a BuildNumber?

There appear to be several schools of thought on this and Maven simply fits with them all.
The Apache method is to not have either and using the SNAPSHOT qualifier allows you to follow this pattern.
However, SNAPSHOT does not allow your developers to know what to use the version for.
JBoss has qualifiers (alpha[n], beta[n], release candidate 'CR[n]' and Final) with optional numbers. [See].
The OSGI specification adds a further complication as does the eclipse numbering which are
MajorVersion, MinorVersion, IncrementalVersion.TIMESTAMP[-Mn]
MajorVersion, MinorVersion, IncrementalVersion.CR[n]
MajorVersion, MinorVersion, IncrementalVersion.Final
There are only two qualifiers. The first one is for the milestone releases, and the qualifier starts with a numeric timestamp. The project can use a timestamp as shown below as it will sort according to the compareTo method of the String class just like any other qualifier. ie YYYYMMDD
Optionally, if for some reason there is a need to make two releases in the same day, you can add a sequence number to the end of the timestamp. The next part of the qualifier is the milestone number, where M stands for milestone, and n is the milestone number.
After all the milestone releases that have added the various functional pieces are complete, and the project and any sub-projects that are integrated are at least at a candidate release stage, then a CR release will follow. Just like in the traditional model, there may be multiple CR releases depending on the feedback from the community.

It is the above approach minus the milestone number I would advise (assuming the 'IncrementalVersion' is controlled by the development team.

Setting up Maven

Making sure your version numbers are incremented can be a pain in the arse but there is a plugin for Maven that helps you mange this.

The Maven 'Release' Plugin

The Release plugin [See] is helpful but not essential. It is used to help a developer release a project with Maven, saving a lot of repetitive, manual work. Its best usage is to allow the developer to update their version number with out effort and correctly.

It is added to the maven project as follows:

 <project>  
     ...  
     <build>  
         <plugins>  
             ...  
             <plugin>  
                 <groupId>org.apache.maven.plugins</groupId>  
                 <artifactId>maven-release-plugin</artifactId>  
                 <version>2.5.2</version>  
             </plugin>  
             ...  
         </plugins>  
         ...  
     </build>  
     ...  
 </project>

easy!

This allows the developer to issue a command as follows:

 mvn -B release:update-versions

... and the version will be updated to the next increment. They then only need to commit it as part of their code.
It is always the last part of the version number that is incremented and it even works if you have a text qualifier such as CRn (see above).

Maven Versions Plugin

This plugin is much more useful when it comes to controlling your release via your CI server. [See]
(In these examples I'm going to quote Jenkins but this process should work for others.)
(I am also not sure if Maven 3.1+ doesn't include this plugin.)

Unlike the previous plugin that increments the version, this will allow a specific version number to be set in the POM, like this:

 mvn versions:set -DnewVersion=0.1.1-RC1

Where '0.1.1-RC1' is an example of a version number.

The process

Now to tie this together.
The process we want is:

Jenkins checks out the latest revision from SCM (Subversion, Mercurial, Git, ...)
Release Plugin transforms the POMs with the new version number
Maven compiles the sources and runs the tests
Release Plugin commits the new POMs into SCM
Maven publishes the binaries into the Artifact Repository

Prerequisites: Jenkins with a JDK and Maven configured, and both the Git and the Workspace Cleanup, and Parameterized Trigger Plugin plugins installed.
We're going to start by creating a new Maven job and making sure we have a fresh workspace for every build:

After assigning you SCM, the next step is to set the version upon checkout.
A good version number is both unique and chronological. We're going to use the Jenkins BUILD_NUMBER (the current build number, such as "153") as it fulfills both these criteria wonderfully.
We could use BUILD_ID which is such as "2005-08-22_23-59-59" (YYYY-MM-DD_hh-mm-ss).
or even the Git commit number using GIT_REVISION.
This is configured as follows:

and in the build step:

And that's it! Every time this job is run, a new release is produced, the artifacts will be deployed and the source code will be tagged. The version of the release will be the BUILD_NUMBER (or GIT_REVISION) of the Jenkins project. Nice and simple.