How To Use SSH to Connect to a Remote Server in Ubuntu VIA a BASTION

What Is SSH?

One essential tool to master as a system administrator is SSH.

SSH, or Secure Shell, is a protocol used to securely log onto remote systems. It is the most common way to access remote Linux and Unix-like servers, such as VPS instances.

In this guide, we will discuss how to use SSH to connect to a remote system.

It is often good practice to set up a Bastion server which is essentially a Gateway to other servers.

If you do this then you can configure your otheer servers to only allow SSH access from your Bastion ONLY.

This post is to help with that process...

 If stuck completly read this first ... https://www.digitalocean.com/community/tutorials/how-to-use-ssh-to-connect-to-a-remote-server-in-ubuntu

SSH connection via the Bastion

SSH into any instance should be done via the Bastion server.

SSH-ing from the Bastion server into a different server generally requires the Bastion public key to be in the authorized_keys in the target server.

Connecting to a new instance, where the Bastion public key is not in authorized_keys

Ensure that the security group for the new instance allows connection from the Bastion.

You can use SSH agent forwarding. This technique allows you to use a keypair to connect from the Bastion, without the private key needing to be on the Bastion.

You should know the keypair the new instance is created with. Open Pageant (on your PC) and import the keypair corresponding to your new instance.

Note: You know Pageant is running if you check the Notification Icons on your PC's task bar.

The other way you know it isn't running is if your Linux server gives the following error following issuing the ssh-add -L command:

In putty, connect to the Bastion using its private key, but in ”Connection → ssh → auth” check the box for Allow agent forwarding.

On the Bastion server, you can check that the key is available by calling:

  ssh-add –L

Now from the Bastion server, ssh into the target instance:

  ssh [PRIVATE IP OF TARGET INSTANCE]

Once you are here, add the Bastion public key to authorized_keys so that agent forwarding won't be required for future connections from the Bastion.

Or leave it as is to make things easy.

generatedata.com - Fake Your Data

Sometimes you need fake data for things like testing software, populating databases, creating realistic-looking mockups and so on. For text, we have lorem ipsum; but for everything else you have to do a bit of legwork. Human-data especially - like names, addresses, town names etc. - are particularly hard to fake because you need a semi-realistically looking data set.

Try the script online at generatedata.com

Word of the Day: zombie server (comatose server)

A zombie server is a physical server that is running but has no external communications or visibility and contributes no compute resources; essentially, it consumes electricity but serves no useful purpose. Zombie servers are also referred to as comatose servers.
Zombie servers are often created because user-requested applications end up getting no use or almost no use (typically defined as under six percent). Other causes include redundant or legacy applications and services that have been replaced. An estimated one in three servers in North America falls into the "undead" category.
According to a study conducted by the consulting firm Anthesis Group and Jonathan Koomey, a research fellow at Stanford University, there are approximately 3.6 million zombie servers in the United States; worldwide, the total could be as high as 10 million.
AOL's five-year project to purge its sites of zombie servers netted the organization $10 million and in just one year resulted in a 35 percent reduction in its carbon footprint. Based on calculations by TSO Logic, a company with 1000 servers could achieve a net savings of $300,000 simply by pulling the plug on its comatose servers.

[Originaly published on http://whatis.techtarget.com/]

Word of the day: unikernel

A unikernel is an executable image that can execute natively on a hypervisor without the need for a separate operating system. The image contains application code as well as all the operating system functions required by that application.
Unikernels are usually built using compilers that leverage library operating systems, which are collections of libraries that represent an operating system's core capabilities. This allows a unikernel developer to selectively include only those library components required to make an application work. Traditional operating system functions, such as network or file system handling, are selectively compiled in to the final executable on an as-needed basis.
Unikernels use a fraction of the resources required by full multipurpose operating systems, such as Linux or Microsoft Windows Server. Their diminutive size enables sub-second startup times and high deployment densities unmatched in traditional server virtualization. Additionally, the minimal footprint of the library OS functions and the absence of traditional operating system utilities greatly reduces the attack surface available for exploitation by malicious hackers.
Some unikernel build systems leverage type-safe languages like Haskell or Erlang, while others can bind to more common languages like C, C++, or Java. Not all applications are suitable for unikernels. Applications that require multiple processes in a single VM are not good candidates, but a huge number of traditional application images could become much smaller and faster when recompiled as unikernels.
Numerous unikernel build systems are available from multiple sources, with the open source community leading the way. Some of the more popular unikernel systems include MirageOS from the Xen Project incubator, Drawbridge from Microsoft, Haskell Lightweight Virtual Machine (HaLVM), LING (formerly Erlang on Xen), OSv, Project Guest VM Microkernel, IncludeOS, rump kernels which leverage NetBSD's library of OS functions, ClickOS created by NEC Laboratories Europe, and Clive from researchers at the Universidad Rey Juan Carlos of Madrid (Spain).

Java Random

I helped a colleague recently  determine whether Java Random() was worth using in a situation where the client had to be certain that the numbers were truly random.

Our concern was that while a small sample of random numbers (especially while seeded with the current time ) are fit for most purposes they are not appropriate where due dillegence needs to be performed.

The reason being most random number generators are only pseudo random.

So we did some research and asked some of our experts about the suitability of the Java Random function.  Things to note are defined below:

• Most of the documents on the subject are quite old and therefore there may be improvements, but I couldn’t find anything which said it had materially changed.
• It is worth noting that we are looking at Pseudo Random Number Generators and not truly random numbers. For the use cases discussed I don’t believe that would be an issue, but that would not be the case in some gambling and cryptography applications
• Where true random numbers are required, a random number service and/or specific random number hardware is used. I don’t believe this is required in this application [https://api.random.org/json-rpc/1/]
• People have been fairly scathing of Java Random, identifying that the current implementation only uses 17 bits of entropy from the initial seed (i.e. 1 in 131072 starting points) and demonstrates repeating patterns after low numbers of calls to generate numbers (of the order of ~50000). Statistical tests have been used to identify that it doesn’t produce very good random numbers [http://www.alife.co.uk/nonrandom/]
• SecureRandom is a drop in replacement for Random and does produce much better random numbers that pass Statistical tests, but it is 60 times slower than Random [https://dzone.com/articles/java-programmer%E2%80%99s-guide-random]. In the way that the Batch report is using random numbers, it appears that of the order of 500 random numbers would be required. A quick test shows that 500 SecureRandom numbers only takes ~15 Milliseconds to calculate. Which isn’t going to significantly affect the time taken for a result.
• There are libraries that support alternative ways of creating random numbers. These produce good quality pseudo random numbers in less time than the standard Java implementations, but would require additional libraries and dependencies to be managed. In this case I do not believe it would be worth the additional effort. [http://maths.uncommons.org/] [https://www.bouncycastle.org/java.html]

Based on the number of random numbers being generated (maximum of around 500 at a time), Random may well be sufficient.  But given the minimal cost of using SecureRandom it may be worth converting to that instead, to remove even a small possibility of concern.

The AWS Well-Architected Framework

A repost from InfoQ but with a good PDF for later reading.

---

Amazon has published the AWS Well-Architected Framework (PDF), a guide for architecting solutions for AWS, with design principles that apply to systems running on AWS or other clouds.


Amazon has based the AWS Well-Architected Framework on four pillars and a number of design principles as outlined in short bellow.

Security. 

According to Amazon, security in the cloud regards 4 areas - Data Protection, Privilege Management, Infrastructure Protection, Detective Controls – and they recommend the following design principles to strengthen the security of a system:

• Apply security at all levels 
• Trace everything 
• Automate responses to security events 
• Secure the system at the application, data and OS level 
• Automate security best practices 

Reliability. 

This pillar represents a system’s ability to “recover from infrastructure or service disruptions, dynamically acquire computing resources to meet demand, and mitigate disruptions such as misconfigurations or transient network issues.” The areas covered by reliability are Foundations, Change Management and Failure Management, and the paper recommends the following design principles:

• Test recovery procedures 
• Automatically recover from failure 
• Use horizontal scalability to increase availability 
• Automatically add/remove resources as needed 

Efficiency. 

This is about efficient use of CPU, storage, and database resources. It also covers the space-time trade-off, i.e. consuming more memory and disk space to solve a problem quicker or using less resources but solving it in more time. The design principles recommended are:

• Use advanced technologies 
• Deploy the system globally for lower latency 
• Use services rather than servers 
• Try various configurations to find out what performs better 

Cost Optimization. 

This is evidently about optimizing costs, eliminating unneeded or suboptimal resources. Cost optimization should consider matching supply with demand, using cost effective resources, keeping an eye on expenses, and lowering the costs over time. This can be done by:

• Transparently attribute expenditure 
• Use managed services 
• Buy computing resources in the cloud rather than hardware 
• Use the cloud for its pay-as-you-go policy 
• Do not invest in data centers 

The framework includes a list of questions to be used when assessing a proposed architecture, such as “How are you encrypting and protecting your data at rest?” or “How are you planning your network topology on AWS?”. The authors also provide their recommendations for addressing each of the problems mentioned in these questions, some of them applying only to AWS, others being valid for any cloud computing architecture.


This article has extracted the main points from the 56-pages whitepaper on architecting solutions for the cloud. For a detailed explanation of all the best practice.

Version Numbering - Redux

The issue

I have been looking at version numbering for a project where the developers had stuck at 0.0.1-SNAPSHOT for 12 months and were starting to encounter issues with getting the correct JARS for their projects from their binary repository.

The solution was to use the features that are present in Maven and Jenkins to assist them in their processes.

I covered the basics of version numbering in an other post (May 2012), but that just gives the version strategy for Apache projects. Since then I have come across Semantic Versioning which is a great a almost definitive source on how you should work with versions.
However this does not cater for the way Maven actually treats version numbers, it is just compatible with it..

A good version number has a number of properties:

• Natural order: it should be possible to determine at a glance between two versions which one is newer
• Maven support: Maven should be able to deal with the format of the version number to enforce the natural order
• Machine incrementable: so you don't have to specify it explicitly every time

What does Maven do?

For reference, Maven version numbers are comprised as follows: <MajorVersion>.<MinorVersion>.<IncrementalVersion>-<BuildNumber | Qualifier>. Where MajorVersion, MinorVersion, IncrementalVersion and BuildNumber are all numeric and Qualifier is a string. If your version number does not match this format, then the entire version number is treated as being the Qualifier. [See]
If all the version numbers are equal, the qualifier is compared alphabetically. "RC1" and "SNAPSHOT" and sorted no differently to "a" and "b". As a result, "SNAPSHOT" is considered newer because it is greater alphabetically. See this page as a reference.
The issue on may projects is how to manage the versions and how to do so with out breaking the Maven format which will cause Maven to treat your versions as just the Qualifier which is a string (not good). Note that a.b.c-RC1-SNAPSHOT would be considered older than a.b.c-RC1, because of text comparisons.

What to use as an Incremental version number?

I think it is reasonably straight forwards for a project to determine the major and minor version as they often come directly from the business drivers for the project.
The Incremental version can give some issues as the business may not be interested and it is the developers that need it for tracking purposes.
Therefore the incremental number has be be meaningful to them.
So usefull numbers could be the database schema version, the sprint number the feature set that is being implemented (although this can be hard if you have several teals working in parallel).

The simplest way is to start at 1 and for the team leads to determine when to increase the number.

Whether to use as a Qualifier or a BuildNumber?

There appear to be several schools of thought on this and Maven simply fits with them all.
The Apache method is to not have either and using the SNAPSHOT qualifier allows you to follow this pattern.
However, SNAPSHOT does not allow your developers to know what to use the version for.
JBoss has qualifiers (alpha[n], beta[n], release candidate 'CR[n]' and Final) with optional numbers. [See].
The OSGI specification adds a further complication as does the eclipse numbering which are
MajorVersion, MinorVersion, IncrementalVersion.TIMESTAMP[-Mn]
MajorVersion, MinorVersion, IncrementalVersion.CR[n]
MajorVersion, MinorVersion, IncrementalVersion.Final
There are only two qualifiers.  The first one is for the milestone releases, and the qualifier starts with a numeric timestamp.  The project can use a timestamp as shown below as it will sort according to the compareTo method of the String class just like any other qualifier. ie YYYYMMDD
Optionally, if for some reason there is a need to make two releases in the same day, you can add a sequence number to the end of the timestamp. The next part of the qualifier is the milestone number, where M stands for milestone, and n is the milestone number.
After all the milestone releases that have added the various functional pieces are complete, and the project and any sub-projects that are integrated are at least at a candidate release stage, then a CR release will follow.  Just like in the traditional model, there may be multiple CR releases depending on the feedback from the community.

It is the above approach minus the milestone number I would advise (assuming the 'IncrementalVersion' is controlled by the development team.

Setting up Maven

Making sure your version numbers are incremented can be a pain in the arse but there is  a plugin for Maven that helps you mange this.

The Maven 'Release' Plugin

The Release plugin [See] is helpful but not essential. It is used to help a developer release a project with Maven, saving a lot of repetitive, manual work. Its best usage is to allow the developer to update their version number with out effort and correctly.

It is added to the maven project as follows:

 <project>  
     ...  
     <build>  
         <plugins>  
             ...  
             <plugin>  
                 <groupId>org.apache.maven.plugins</groupId>  
                 <artifactId>maven-release-plugin</artifactId>  
                 <version>2.5.2</version>  
             </plugin>  
             ...  
         </plugins>  
         ...  
     </build>  
     ...  
 </project>  

easy!

This allows the developer to issue a command as follows:

 mvn -B release:update-versions  

... and the version will be updated to the next increment. They then only need to commit it as part of their code.
It is always the last part of the version number that is incremented and it even works if you have a text qualifier such as CRn (see above).

Maven Versions Plugin

This plugin is much more useful when it comes to controlling your release via your CI server. [See]
(In these examples I'm going to quote Jenkins but this process should work for others.)
(I am also not sure if Maven 3.1+ doesn't include this plugin.)

Unlike the previous plugin that increments the version, this will allow a specific version number to be set in the POM, like this:

 mvn versions:set -DnewVersion=0.1.1-RC1  

Where '0.1.1-RC1' is an example of a version number.

The process

Now to tie this together.
The process we want is:

1. Jenkins checks out the latest revision from SCM (Subversion, Mercurial, Git, ...)
2. Release Plugin transforms the POMs with the new version number
3. Maven compiles the sources and runs the tests
4. Release Plugin commits the new POMs into SCM
5. Maven publishes the binaries into the Artifact Repository

Prerequisites: Jenkins with a JDK and Maven configured, and both the Git and the Workspace Cleanup, and Parameterized Trigger Plugin  plugins installed.
We're going to start by creating a new Maven job and making sure we have a fresh workspace for every build:

After assigning you SCM, the next step is to set the version upon checkout.
A good version number is both unique and chronological. We're going to use the Jenkins BUILD_NUMBER (the current build number, such as "153") as it fulfills both these criteria wonderfully.
We could use BUILD_ID which is such as "2005-08-22_23-59-59" (YYYY-MM-DD_hh-mm-ss).
or even the Git commit number using GIT_REVISION.
This is configured as follows:

and in the build step:

And that's it! Every time this job is run, a new release is produced, the artifacts will be deployed and the source code will be tagged. The version of the release will be the BUILD_NUMBER (or GIT_REVISION) of the Jenkins project. Nice and simple.