For example, a sample of valid command line options are shown below in bold.

java -Xmx6g AClass
java -version
java -Xms4g -Xmx6g SomeClass
java -DSomeVal="foo" MyProgram
java -DSomeVal="foo" -cp MyProgram.jar -Xmx6g -XX:+UseCompressedOops -XX:+UseG1GC com.foo.Bar 

JVM also support many command-line options which allow users to specify:

• Minimum and Maximum Size of the Heap Memory
• Type of Garbage Collector
• Type of JIT Compiler (Client or Server), or,
• To display JVM version,
• etc.

Java Language Specification (JLS) breaks the command-line options into three categories based on their maturity level. The most mature options belong to a category called “Standard Options” and must be supported by all JVM’s. Less mature options are called “Non-Standard“. These are specific to JVM (e.g. HotSpot) and are subject to change between releases. There is also a third category of  options called “Developer Options“. Developer options are a shade of Non-Standard.

Quick Recap: JVM command-line options are used to specify configuration settings to control execution of the Virtual Machine. These options are broken into three categories as shown below:

1. Standard Options
2. Non-Standard Options
3. Developer Options (Experimental)

Let’s look at the categories in detail.

1. Standard Options

The Standard Options are regulated by the Java Virtual Machine Specification and must be supported by all implementations of Java Virtual Machines. For example, OpenJDK, HotSpot, etc. Standard Options are stable and do not change between releases. Standard Options begin with a - followed by the name of option, e.g. -version

Some standard options are shown below:

-version:  java -version

-cp:  java -cp <PATH>

-jar:  java -jar MyProgram.jar

-Dproperty=value:  java -DSomeVal="foo"

2. Non-Standard Options

Non-Standard Options always begin with -X.  The fact that they are not guaranteed to be supported in all implementations of the JVM or even between versions, did little to hurt their popularity. Non-Standard Options remain popular and are widely used. These options often specify an integer value with a suffix of k,m, or g to specify kilo, mega or giga. To get a list of all Non-Standard Options supported by your JVM, you can invoke the launcher with -X, e.g. `java -X`. Examples:

-Xms:  java -Xms2g 

-X:  java -X

3. Developer Options

Developers Options always begin with -XX.  They are also Non-Standard. They follow the following format for setting boolean Options: -XX: followed by either + or - to indicate true or false, followed by the name of option.

-XX:+UseCompressedOops (Indicates that the Option UseCompressedOops must be used)

From, Java Documentation:

• Boolean options are turned on with -XX:+<option> and turned off with -XX:-<option>.

• Numeric options are set with -XX:<option>=<number>. Numbers can include ‘m’ or ‘M’ for megabytes, ‘k’ or ‘K’ for kilobytes, and ‘g’ or ‘G’ for gigabytes (for example, 32k is the same as 32768).

• String options are set with -XX:<option>=<string>, are usually used to specify a file, a path, or a list of commands

Examples:

java -XX:+PrintCompilation

java -XX:+ParallelGCThreads=10

Click here for a complete list of Options from Oracle.

Fortunately, MBean names follow some standard conventions and names can determine how Clients display MBeans. MBean names look like this:

domain:key=property

Remember this convention. Read the last line carefully. Notice that there are two parts with a : separating them. The first part is called domain and the second part is called key-properties-list.

Here’s an example MBean name with both domain and properties: com.somecompany.app:type=ThreadPool

Domain Naming Conventions

The domain could be any arbitrary string, but it cannot contain a : since it is used as a separator. Slash (/) isn’t allowed as well. If the domain name is not provided, then the MBean shows up under “DefaultDomain“. As mentioned earlier, domain names should be predictable. According to Oracle Technet:

….if you know that there is going to be an MBean representing a certain object, then you should be able to know what its name will be.

And then add further:

The domain part of an Object Name should start with a Java package name. This prevents collisions between MBeans coming from different subsystems. There might be additional text after the package name. Examples:

com.sun.someapp:type=Whatsit,name=5
com.sun.appserv.Domain1:type=Whatever

Key=Property Naming Conventions

The property list is optional and is in the following format:

name=property

If you wish to specify multiple properties, separate each of them by comma(,). For example:

com.somecompany.app:type=ThreadPool,poolname=Parser,scope=internal

They Key=Property should be used to uniquely identify MBean Objects, such as, each Object in the same domain may have different properties. For example:

com.somecompany.app:type=ThreadPool,name=Parser

com.somecompany.app:type=ThreadPool,name=Generator

That’s MBean naming convention from  1000 feet. If you want to get more information, please read here.

1. Error 
2. Exception
3. RuntimeException

Error

These exceptional circumstances are like “act-of-god” events. For example, car manufacturers don’t design cars to prevent damage from tsunamis, asteroids or alien invasions. If an alien invasion happens, there is (usually) nothing anyone can do.

These are exceptional conditions that are external to the application, and that the application usually cannot anticipate or recover from. For example, suppose that an application successfully opens a file for input, but is unable to read the file because of a hardware or system malfunction.

Source: The Java Tutorials

Exception

These are most widely known as checked exceptions. Back to our car manufacturer analogy, they are like not wearing passenger seat belts. We don’t always know our passengers. This can de detected by seat-belt sensors and the guilty passenger could be warned or even buzzed with harmless 12 volts.

According to official documentation:

These are exceptional conditions that a well-written application should anticipate and recover from. For example, suppose an application prompts a user for an input file name,  [..] But sometimes the user supplies the name of a nonexistent file, and the constructor throws java.io.FileNotFoundException. A well-written program will catch this exception and notify the user of the mistake, possibly prompting for a corrected file name.

Source: The Java Tutorials

RuntimeException

Fitting in our analogy, this is like pumping diesel into a gasoline car. This can be prevented (by being smarter)  in the first place.

These are exceptional conditions that are internal to the application, and that the application usually cannot anticipate or recover from. These usually indicate programming bugs, such as logic errors or improper use of an API.

Source: The Java Tutorials

Try/Catch

In Java, exceptional conditions must be caught by try-catch block. However, Errors and  RuntimeExceptions (and their derivatives)  are not subject to this requirement, which means code which throws them, need not be wrapped in a try-catch block. Whereas code that throws checked Exceptions or their derivates, must be wrapped in try-catch block.

Back to the point…

Dividing a number by 0 will generate a run time exception, ArithmeticException, which according to our previous arguments, is not required to be caught. Java official take on the subject of catch RuntimeExceptions is:

The application can catch this exception, but it probably makes more sense to eliminate the bug that caused the exception to occur.

If you follow this official recommendation, you should not catch the ArithmeticException, which will be thrown in the denominator == 0.  Instead, before division, we can check if (denominator > 0). If not, we can halt further execution or even dare to throw a exception of our own IllegalArgumentException.

OK, so far it is looks like we can have a well designed program that checks inputs and can get away by not catching RuntimeExceptions as per Oracle’s recomendations.

So when is it OK for an application to catch RuntimeExceptions? Never?

A while back, I architected a high-performance traffic director with the goal of operating in the proximity of 10,000 transactions per seconds (TPS). The project had a very high availability criteria and one of the requirement was that it “must-never-exit”.

The director performs minimum amount of processing on each transaction before passing it further. Transactions came in two flavours, call them: A and B. We were only interested in transactions of type A. We had a transactions handler to process type A. Naturally, it “choked” run time exceptions when we passed in transactions of type B. The solution? Create a function and pass it every single transaction. If it returned true, we continued to further processing. Otherwise, we simply ignored the transaction, and continued onto the next one.

boolean checkFormat(Transaction t) {
//return true if the t is of type A. false otherwise.
}

This worked well, except…..

… the analysis showed that this function returned false only once a year. The reason, 99.99999999999999% transactions were of type A. Yet, we were subjecting every single transaction to be checked. This does not sound so bad, but due to the nature of transactions, the only way to differentiate was by doing expensive String comparison on various fields.

When this finding was brought to my knowledge, I immediately had the `checkFormat(…)` function removed and instead let the handler do it’s course and throw RuntimeException upon encountering transaction of type, B. When the exception gets thrown once a year, we catch it, log it and move onto the next transaction. The result: improvement in performance, and room to squeeze in additional calculations.

Summary

Although, we can get away by not catching RuntimeExceptions most of the time, there are definitely some circumstances which justify catching these types of exceptions.

Tasks are activities that perform some action or do calculations. For example a task could calculate prime numbers up to some upper limit. Good tasks do not depend on other tasks: they are independent. In this post, when I refer to tasks, I would mean tasks that are independent.

Tasks in Java can be represented by a very simple interface called Runnable that has only one method: run(). The singular function neither returns a value nor can throw checked exceptions.

public interface Runnable {
    void run();
}

Many new comers to Java presume Threads to be the primary abstraction for running tasks. This means that a task can be submitted to a thread which then runs the task. In fact, the Thread class has constructors which take a Runnable for execution:

Thread(Runnable target)
Thread(Runnable target, String name)
Thread(ThreadGroup group, Runnable target)
...

There are obvious benefits in segregating tasks and threads.

A Task, defined by implementing Runnables, is submitted to Thread for execution. The Thread doesn’t know anything about the task and the same thread could run several different tasks.

Enter Executor:

Executor was introduced in Java 1.5 as a clean abstraction for executing tasks. Mantle was passed to Executor from Thread. According to the Java API, an Executor:

“… executes submitted Runnable tasks.  This interface provides a way of decoupling task submission from the mechanics of how each task will be run, including details of thread use, scheduling, etc.” In essence, Executor is an interface, whose simplicity rivals that of Runnable:

public interface Executor {
    void execute(Runnable command);
}

The ‘very simple’ Executor interface forms basis for a very powerful asynchronous task execution framework. It is based on a Producer-Consumer pattern: Producers produce tasks and Consumer threads execute these tasks.

ExecutorService

There is little chance you will use ever use Executor directly. It is very powerful, yet feature starved interface with a lone method for executing tasks. Fortunately for us, it has a famous child called ExecutorService, which provides lifecycle support such as shutdown, task tracking and the ability to retrieve results.

Tracking Task Progress via Future

ExecutorService defines a method called `submit(Runnable task)` which returns a `Future` that can be used to track task’s progress and cancel it (if desired). Future is an interface. From its javadocs:

“A Future represents the result of an asynchronous computation. Methods are provided to check if the computation is complete, to wait for its completion, and to retrieve the result of the computation. The result can only be retrieved using method get when the computation has completed, blocking if necessary until it is ready. Cancellation is performed by the cancel method. Additional methods are provided to determine if the task completed normally or was cancelled. Once a computation has completed, the computation cannot be cancelled.”

RunnableFuture

Earlier on, I said that the interface Runnable doesn’t return a value. Runnable tasks can indicate completion by modifying a shared data structure. RunnableFuture implements both Future and Runnable interfaces. It can be submitted to any method which expects Runnable and the Future allows to access its result.

So far we have only discussed interfaces (Executor, ExecutorService and Future). Before we look into concrete classes, let us consider one very important concept.

Thread Pool

A  design pattern: http://en.wikipedia.org/wiki/Thread_pool_pattern. It has a task queue which holds incoming tasks and has a pool of thread which takes tasks from the queue and execute them.

A sample thread pool (green boxes) with waiting tasks (blue) and completed tasks (yellow)

Benefits of Thread Pools are thread re-use (creating new threads is a significant CPU overhead) and improved responsiveness (there may already be a waiting thread when a task arrives).

Now let us discuss concrete classes.

AbstractExecutorService

This is a skeletal implementation for ExecutorService, providing default implementations for some of it’s methods.

public abstract class AbstractExecutorService
implements ExecutorService

ThreadPoolExecutor

This is an ExecutorService that applies the Thread Pool pattern to execute tasks. From its javadocs:

“An ExecutorService that executes each submitted task using one of possibly several pooled threads, normally configured using Executors factory methods.” It provides several methods for setting pool and task queue sizes. For more information:

public class ThreadPoolExecutor extends AbstractExecutorService
implements Executor, ExecutorService

FutureTask

Provides an implementation of Future and RunnableFuture. From its javadoc:

“…provides a base implementation of Future, with methods to start and cancel a computation, query to see if the computation is complete, and retrieve the result of the computation.”

Since a FutureTask implements RunnableFuture, you can submit it directly to an ExecutorService.

Callable:

Callable‘s were introduced in Java 5 as the next version of Runnable. Just like Thread passed mantle to Executor for task execution, Runnable passed mantle to Callable for representing tasks.

Callable for used to represent tasks. Unlike Runnable’s, they can return a value and even throw Exceptions. They even support generics.

Summary:

Executor and ExecutorService form a very powerful framework for asynchronous task execution. Future is a wrapper that provides a way to track a task’s progress and could be used to cancel it. Callable represents a task and allows the task to return a value and throw exceptions.

So you might ask why do we still have Threads and Runnables if we have better choices available, in the form of Executor and Callable. As far as Callable Vs Runnable is concerned, the reason is purely backwards compatibility. Threads are not languishing in Java. ExecutorService simply provides a cleaner abstraction for executing tasks. They still rely on Threads to execute these tasks.

Java language designers got drunk once and attempted to support forced thread termination by releasing the following methods: `Thread.stop()`, `Thread.suspend()` and `Thread.resume()`. However, when they become sober, they quickly realized their mistake and deprecated them. Abrupt thread termination is not so straight forward. A running thread, often called by many writers as a light-weight process, has its own stack and is the master of its own destiny (well daemons are). It may own files and sockets. It may hold locks. Termination is not always easy: Unpredictable consequences may arise if the thread is in the middle of writing to a file and is killed before it can finish writing. Or what about the monitor locks held by the thread when it is shot in the head? For more information on why `Thread.stop()` was deprecated, follow this link: http://docs.oracle.com/javase/6/docs/technotes/guides/concurrency/threadPrimitiveDeprecation.html

Anyways, back to the point.

In Java, there is no way to quickly and reliably stop a thread.

To stop threads in Java, we rely on a co-operative mechanism called Interruption. The concept is very simple. To stop a thread, all we can do is deliver it a signal, aka interrupt it, requesting that the thread stops itself at the next available opportunity. That’s all. There is no telling what the receiver thread might do with the signal: it may not even bother to check the signal; or even worse ignore it.

Once you start a thread, nothing can (safely) stop it, except the thread itself. At most, the thread could be simply asked – or interrupted – to stop itself.

Hence in Java, stopping threads is a two step procedure:

• Sending stop signal to thread – aka interrupting it
• Designing threads to act on interruption

A thread in Java could be interrupted by calling `Thread.interrupt()` method. Threads can check for interruption by calling `Thread.isInterrupted()` method. A good thread must check for interruption at regular intervals. The following code fragment illustrates this:

public static void main(String[] args) throws Exception {

        /**
         * A Thread which is responsive to Interruption.
         */
        class ResponsiveToInterruption extends Thread {
            @Override public void run() {
                while (!Thread.currentThread().isInterrupted()) {
                    System.out.println("[Interruption Responsive Thread] Alive");
                }
                System.out.println("[Interruption Responsive Thread] bye**");

            }
        }

        /**
         * Thread that is oblivious to Interruption. It does not even check it's
         * interruption status and doesn't even know it was interrupted.
         */
        class ObliviousToInterruption extends Thread {
            @Override public void run() {
                while (true) {
                    System.out.println("[Interruption Oblivious Thread] Alive");
                }
                // The statement below will never be reached.
                //System.out.println("[Interruption Oblivious Thread] bye");
            }
        }

        Thread theGood = new ResponsiveToInterruption();
        Thread theUgly = new ObliviousToInterruption();

        theGood.start();
        theUgly.start();

        theGood.interrupt(); // The thread will stop itself
        theUgly.interrupt(); // Will do nothing
}

[Interruption Oblivious Thread] Alive
[Interruption Responsive Thread] Alive
[Interruption Responsive Thread] Alive
[Interruption Oblivious Thread] Alive
[Interruption Responsive Thread] bye**
[Interruption Oblivious Thread] Alive
[Interruption Oblivious Thread] Alive
[Interruption Oblivious Thread] Alive
[Interruption Oblivious Thread] Alive
....

A well designed thread checks its interrupt status at regular intervals and take action when interrupted, usually by cleaning and stopping itself.

Blocking Methods and Interruption:

A thread can check for interruption at regular intervals – e.g. as a loop condition – and take action when it is interrupted. Life would have been easy if it weren’t for those pesky blocking methods: these methods may “block” and take a long time to return, effectively delaying the calling thread’s ability to check for interruption in a timely manner. Methods like `Thread.sleep()`, `BlockingQueue.put()`, `ServerSocket.accept()` are some examples of blocking methods.

If the code is waiting on a blocked method, it may not check the interrupt status until the blocking method returns.

Blocking methods which support interruption usually throw an Exception when they detect interruption, transferring the control back to the caller. Blocking methods either throw InterruptedException or ClosedByInterruptionException to signal interruption to the caller. Let us consider an example. the code below calls `Thread.sleep()`. When it detects interruption, `Thread.sleep()` throws InterruptedException and the caller exits the loop. All blocking methods which throw InterruptedException also clear the interrupted status. You must either act on interruption when you catch this exception or at the very least, set the interrupted status again to allow the code higher up the stack to act on interruption.

   @Override
   public void run() {
        while(true) {
            try {
                Thread.sleep(Long.MAX_VALUE);
            } catch (InterruptedException exit) {
                break; //Break out of the loop; ending thread
            }
        }
    }

This may sound preposterous, but the code that does nothing on InterruptedException is “swallowing” the interruption, denying other code to take action on interruption.

Interruption Oblivious Blocking Methods:

In the first code example in this post, we have two threads, ResponsiveToInterruption and ObliviousToInterruption. The former checked for interruption regularly – as loop condition – whereas the later didn’t even bother to check. Blocking methods in Java library fall in the same two categories. The Good ones throw Exceptions when they detect interruption whereas the Ugly one’s don’t do anything. Blocking methods in the java.net.Socket don’t respond to Interruption. For example, the thread below cannot be stopped by interruption when it is waiting for clients. When a client is connected, accept() returns Socket allowing the caller to check for interruption:

        /**
         * Thread that checks for interruption, but calls a blocking method
         * that doesn't detect Interruptions.
         */
        class InterruptibleShesNot extends Thread {

            @Override
            public void run() {
                while(!Thread.currentThread().isInterrupted()) {
                    try {
                        ServerSocket server = new ServerSocket(8080);
                        Socket client = server.accept(); // This method will not
                                                         // return or 'unblock'
                                                         // until a client connects
                    } catch (IOException ignored) { }
                }

            }

        }

So how do you deal with blocking methods that do not respond to Interruption? You will have to think outside the box and find ways to cancel the operation by other means. For example, Socket operations throw SocketException when the underlying socket is closed (by `Socket.close()`). The code below takes advantage of this fact and closes the underlying socket, forcing all blocking methods such as ServerSocket.accept() to throw SocketException.

package kitchensink;

import java.net.*;
import java.io.*;

/**
 * Demonstrates non-standard thread cancellation.
 *
 * @author umermansoor
 */
public class SocketCancellation {

    /**
     * ServerSocket.accept() doesn't detect or respond to interruption. The
     * class below overrides the interrupt() method to support non-standard
     * cancellation by canceling the underlying ServerSocket forcing the
     * accept() method to throw Exception, on which we act by breaking the while
     * loop.
     *
     * @author umermansoor
     */
    static class CancelleableSocketThread extends Thread {

        private final ServerSocket server;

        public CancelleableSocketThread(int port) throws IOException {
            server = new ServerSocket(port);
        }

        @Override
        public void interrupt() {
            try {
                server.close();
            } catch (IOException ignored) {
            } finally {
                super.interrupt();
            }
        }

        @Override
        public void run() {
            while (true) {
                try {
                    Socket client = server.accept();
                } catch (Exception se) {
                    break;
                }
            }
        }
    }

    /**
     * Main entry point.
     * @param args
     * @throws Exception
     */
    public static void main(String[] args) throws Exception {
        CancelleableSocketThread cst = new CancelleableSocketThread(8080);
        cst.start();
        Thread.sleep(3000);
        cst.interrupt();
    }
}

Summary:

• Threads cannot be stopped externally; they can only be delivered a signal to stop
• It is up to the Thread to: i) check the interruption flag regularly, and ii) to act upon it
• Sometimes checking interruption is not possible if the thread is blocked on a blocking method, such as `BlockingQueue.put()`. Luckily, most blocking methods detect interruption and throw InterruptedException or ClosedByInterruptionException
• To support blocking methods that do not act on interruptions, non-standard cancellation mechanisms must be used, as illustrated in the last example

Extra:

The thread class also has a method called `interrupted()`. This is what is does: it clears the interrupted status and returns its previous value. Use this method only when you know what you are doing or when you want to clear the interrupt status.

In this post we will focus on the Hadoop Distributed File System (HDFS) and MapReduce. These two are core Hadoop modules and are widely used. Together (HDFS + MapReduce) form a very powerful framework for distributed batch processing – I want you to remember this statement by heart.

HDFS and MapReduce form a framework for distributed batch processing.

Almost anything you can accomplish with Hadoop’s modules  (HDFS + MapReduce) could be done with built-in Linux utilities like grep, awk, bash, etc. Hadoop excels in large scale, distributed processing of data, where data to be processed is distributed on hundreds of nodes. With the advent of Cloud Computing, this is quickly becoming the norm. Distributed servers running on multiple nodes producing decentralized logs make it difficult to analyze data in one central place. Consider Google - it runs on thousands of web servers in multiple data centers around the world. Each web server generates a log file which is stored on its local disk, ending up with thousands of log files stored on as many servers. Analytics program should be able to view these dispersed logs as a single logical unit. For example, the following hypothetical queries require processing every single log file to find the total for that server and then add up results from all servers to get the final aggregate sum:

• Number of unique users between 12:00-1:00am.
• Number of users in a day from Chile.

Hadoop’s true power lies in its ability to scale to hundreds or thousands of nodes to distribute large amounts of work across a set of machines to be performed in parallel.

HDFS

Hadoop modules are built upon a Distributed File System appropriately named Hadoop Distributed File System (HDFS). The most famous `distributed` file system in existence today is the Network File System or NFS. HDFS is different from NFS on many levels, especially with regards to scalability.

Note: The design of HDFS is based on Google File System (GFS) described in this paper.

HDFS is based on a master/slave architecture and the design requires that a single master node keeps track of all the files in the file system. This is called NameNode. A NameNode stores only the meta-data about the files present on the file system: it doesn’t store the actual data. The data is stored on DataNodes. Files are stored on the HDFS in blocks which are typically 64 MB in size.

NameNode versus DataNode: A NameNode manages the HDFS’s namespace and regulates access to files and directories. It distributes data blocks to DataNodes and stores this mapping information. DataNodes are responsible for storing data and serving read and write requests from clients.

Let us consider an example, suppose we are storing a 131 MB file on the HDFS. The file will be stored in three blocks on the HDFS  (64 + 64 + 3). NameNode will distribute the three blocks to DataNodes and keep track of the mapping. To read a file stored on the HDFS, the client must have HDFS installed. The client HDFS will obtain the file information from NameNode such as the number of file blocks and their location and then download these blocks directly from DataNodes.

Fore more information on the HDFS, I recommend the following link: http://hadoop.apache.org/docs/hdfs/current/hdfs_design.html

MapReduce

MapReduce (MR) is a framework or a library for writing applications to process large amounts of distributed data in parallel. Like HDFS, it’s architecture is also based on master/slave model. The master is a special node which coordinates activity between several worker nodes.

Here’s how it works: The master receives the input data which is to be processed. The input data is split into smaller chunks and all these chunks are distributed to and processed in parallel on multiple worker nodes. This is called the Map Phase. The workers send their results back to the master node which aggregates these results to produce the sum total. This is called the Reduce phase.

Note: I’ve oversimplified the inner workings of MapReduce. The Map phase output is written to the local disk of workers which is partitioned in as many regions as there are Reduce workers available. This locations is then passed to the master which passes it onto Reduce workers. I recommend this paper on MapReduce by Google, which actually introduced it.

MR applications at least must provide the following three input parameters:

1. Location of the input data (e.g. a directory consisting of a single (rare) or multiple input files).
2. Programming implementations of Map, Reduce functions and their configuration (e.g. a Java JAR file which is distributed to workers)
3. Location of the output data (e.g. `/tmp/hadoop-output/`)

You must remember this always: All input and output in MR is based on <key, value> pairs. It is everywhere, the input to the Map function, its output, the input to the Reduce function and its output are all in <key, value> pairs.

MapReduce Example in Java

To wrap our minds around MapReduce, let us consider an example. Suppose you have just become the Development Lead for a company which specializes in reading seismic data which measure earthquake magnitudes around the world. There are thousands of such sensors deployed around the world recording earthquake data in log files, the following format:

nc,71920701,1,”Saturday, January 12, 2013 19:43:18 UTC”,38.7865,-122.7630,1.5,1.10,27,“Northern California”

Each entry consists of lot of details. The items in red are the magnitude of the earthquake and the name of region where the reading was taken, respectively.

There are millions of such log files available. In addition, logs also contain erroneous entries such as when the sensor became faulty and went in an infinite loop dumping thousands of lines a second. The input data is stored on 50 machines and all the log files combined are about 10 Terabytes in size.  Your Director of Software asks you to perform a simple task: for every region where sensors were deployed, find out the highest magnitude of the earthquake recorded.

Now, let’s think about this for a moment. This tasks sounds rather simple. You could use your trusted linux tools like `grep`, `sort`, or even `awk` to accomplish this if the logfiles were available on a single computer. But they are not – they are scattered across 50 computers. Processing data on each computer manually and combing results will be too inefficient (for a Lead Developer, that is).

This is the kind of problem where you can use Hadoop. Let us see how you would do it:

1. First you will deploy HDFS on the 50 machines where the input data is stored so that all data could be seen by all machines. Let us say you put all logfiles of data in a folder on HDFS called input/.
2. Next, you will write a Java application providing implementations of Map & Reduce functions.

       /**
        * The `Mapper` function. It receives a line of input from the file,
        * extracts `region name` and `earthquake magnitude` from it, and outputs
        * the `region name` and `magnitude` in <key, value> manner.
        * @param key - The line offset in the file - ignored.
        * @param value - This is the line itself.
        * @param context - Provides access to the OutputCollector and Reporter.
        * @throws IOException
        * @throws InterruptedException
        */
       @Override
       public void map(LongWritable key, Text value, Context context) throws
               IOException, InterruptedException {
   
           String[] line = value.toString().split(",", 12);
   
           // Ignore invalid lines
           if (line.length != 12) {
               System.out.println("- " + line.length);
               return;
           }
   
           // The output `key` is the name of the region
           String outputKey = line[11];
   
           // The output `value` is the magnitude of the earthquake
           double outputValue = Double.parseDouble(line[8]);
   
           // Record the output in the Context object
           context.write(new Text(outputKey), new DoubleWritable(outputValue));
       }
   

       /**
        * The `Reducer` function. Iterates through all earthquake magnitudes for a
        * region to find the maximum value. The output is the `region name` and the
        * `maximum value of the magnitude`.
        * @param key - The name of the region
        * @param values - Iterator over earthquake magnitudes in the region
        * @param context - Used for collecting output
        * @throws IOException
        * @throws InterruptedException
        */
       @Override
       public void reduce(Text key, Iterable values,
               Context context) throws IOException, InterruptedException {
   
           // Standard algorithm for finding the max value
           double maxMagnitude = Double.MIN_VALUE;
           for (DoubleWritable value : values) {
               maxMagnitude = Math.max(maxMagnitude, value.get());
           }
   
           context.write(key, new DoubleWritable(maxMagnitude));
       }
   

3. Next you will configure MapReduce to be run processing on all 50 computers. This achieves data locality – the logfiles are processed on the same computer where they are located and only the results are sent back to be reduced, saving bandwidth.
4. Run `hadoop` passing it the location of the input folder on the HDFS (input/), MapReduce program, and the location where the output is to be produced (output/). And that’s all you need to do.

Example Project on GitHub

You can find the complete source code for the above example on GitHub repository I have created.

17.0
1.2.167.19
11.0.0
1.5.11 (1635)
1.6.0_37 
23.0.1271.101
6.0.0.2968
Mayan Apocalypse (96)
0.6.9
3.7.1 (M20110909-1335)
7.0 (Build 201104080000)  

As you can see, there is no concrete pattern here. The most common theme seems to be:

<major>.<minor>.<patch>.<build>

To be fair, there have been some attempts to standardize version numbers such as Semantic Versioning, Apache APR, even the older Apple Standards. However,…..

Not all software is created equal:

We have a wide variety of software from libraries and frameworks, to complex server software to simple mobile applications. Some software is characterized by a high build rate (e.g. few times a day) whereas on the other end of the spectrum, it’s easy to find cases where a release every few months or even years is common practice. Some software is released to the public, other stays in-house. It is hard to define a single universal scheme that works for all kinds of software. So let us divide software into the following three broad categories:

1. Libraries and Frameworks
2. Public Software
3. In-house or Hosted Software

Libraries and Frameworks

For library components and frameworks, which define their own API, Semantic Versioning (SemVer) looks promising. It requires that the software must define a public API to be able to use it. From their webpage:

“A normal version number MUST take the form X.Y.Z where X, Y, and Z are non-negative integers. X is the major version, Y is the minor version, and Z is the patch version. Each element MUST increase numerically by increments of one. For instance: 1.9.0 -> 1.10.0 -> 1.11.0.”

Web based frameworks like Express (for Node.js), Ruby On Rails, can benefit from it.

Public Software

When it comes to public software, things get complicated. Let’s take a moment to ask yourself why your users would ever use the version information? The only answer that comes to my mind is that’s how they decide if there’s a newer version available for purchase or not. Back in the day when I was using Windows 95 and heard about this fancy new thing called Windows 98, I knew it was time to upgrade.

Here’s the deal: your users don’t really care about cryptic version numbers. The only time detailed version information such as 4.1.2222A would come handy is when the user is experiencing issues and is on the phone, the support guy would want to know exactly what version of the software they are using right down to the build number.

Therefore, it is a good idea for public software to have a dual versioning scheme: The first version is a user-friendly name that is easy to remember and use in casual discussions, e.g. Lion.  The second is the detailed version to be used in times of crisis, e.g. 10.7.5 (11G63), such as when browsing the Internet for known vulnerabilities or calling Apple for support.

Microsoft seemed to have invented or at least popularized the dual versioning system with Windows 95. Windows 95, the immensely popular successor of Windows 3.1x, was actually called Windows 4.0. Windows 98 was 4.1. Apple picked this idea up starting from OS X 10.0 and started naming major releases after cats (Cheetah, Jaguar, Panther, to the latest Mountain Lion). You can apply your creative juices when coming up with a user-friendly versioning scheme and even involve the marketing group in the process.

For detailed versioning scheme, I personally prefer <major>.<minor>.<revision>.<build>. You could use <major>.<minor>.<revision>.<date>. For example:

2.15.45.30098312345
1.0.3 (December 21, 2012)

Embedding the date in the release is actually a very good idea. It conveys time the build was generated. Others prefer appending build Ids generated by build management systems such as SVN, Git or Mercurial. At my work, I use the following rules:

• Major: This is usually done at the end of full release cycle; the resulting product is a major upgrade. All stakeholders (Managers, VPs, Marketing Directors) are informed of the major increment and is usually followed by press releases and marketing.
• Minor: When additional functionality or new features are introduced. Software Development managers or Senior Developers typically approve such increments.
• Revision: Bug fixes, ad-hoc patches, any minor change. Developers increment this number each time they make a minor change or fix a bug.
• Build: This is the only component of the version that is automatically generated. There is a python script that generates this using the Git commit SHA every time the software is built.

This post from Alex Collins explain the rules are incrementing each component:

“You zero digits to the right of any you increment, so if you fix a bug and introduce a new feature after version 5.3.6 then the new version is 5.4.0. Unstated digits are assumed to be zero, so 5.4.0 is the same as 5.4.0.0 and 5.4.0.0.0.0.0.0.0…”

Bonus: I have written a Java class to hold Version information. It is available under the MIT license so you can freely use it. .NET users, have Version class which comes with the framework.

In-house or Hosted Software

For high rate builds using automated systems like Jenkins, it is a good idea to have versions automatically time stamped. Something like <year>.<month>.<day>.<time> is good for providing detailed information to support desk staff and for referencing build information to developers. E.g. 2012.12.29.0059 is the version of build generated on December 29, 2012 at 12:59 a.m.

Be Consistent

Whatever you do, please be consistent and stick to the system you pick. Sun Microsystems (acquired by Oracle), confused developers everywhere by changing their versioning scheme along the way.

Reblogged from JavaScript, JavaScript...:

In Cassandra terminology, these copies of data or rows are called replicas.

Replication Factor:

This parameter determine how many nodes in your cluster store copies of data. For example, if Replication Factor (RF) is set to 2, there will be two copies of every data stored on different nodes. As common sense dictates, the RF cannot be greater than the number of nodes in the cluster. You cannot say store 10 replicas of data (RF=10) when you only have 8 nodes available. If you try to do this, your writes will fail.

Replication Strategies:

Cassandra has two replication protocols or strategies which I will discuss below. The main difference between the two is whether you have a single data center or your Cassandra cluster spans multiple data centers.

• Simple Strategy: As the name indicates, this is the simplest strategy. Data Partitioner protocol (e.g. RandomPartitioner) picks out the first node for the first replica of data. SimpleStrategy places the second replica on the very next node in the ring.
• Network Topology Strategy: As the name indicates, this strategy is aware of the network topology (location of nodes in racks, data centers etc.) and is much intelligent than Simple Strategy. This strategy is a must if your Cassandra cluster spans multiple data centers and lets you specify how many replicas you want per data center. It tries to distribute data among racks to minimize failures. That is, when choosing nodes to store replicas, it will try to find a node on a different rack.Let’s talk about a few real-world scenarios from my experience: My team and I were designing a Cassandra cluster spanning 4 data centers (we called them sites). Please note that I’m changing several details about the actual scenario, however, the concepts are the same. Each site also had own application servers. See figure below.
  

  We had two concerns:
  1. Allow local reads in a data center to avoid cross data center latency. For example, the Application Server in Site #1 should read from the Cassandra cluster in the same site.
  2. Application should continue to work albeit with limited functionality if connectivity between data centers is disrupted.
  3. Tolerate entire cluster failures in two data centers and some node failures throughout all clusters in all data centers.

  To meet all the requirements, we used NetworkTopologyStrategy with 2 replicas per data center. That is every single row of data will be stored in 8 nodes in total ( 2 nodes per site x 4 sites). Hence we can tolerate failure a single node per site while still allowing local reads. We can also tolerate failure of an entire cluster but will have to incur the extra cost of cross data center latency in the Application server of the site where the cluster failed. For example, if the cluster in Site #2 dies completely, the Application server in that site can have to retrieve data from any of Site # 1, 2 or 3 with extra cost of inter site communication. In this example, we can still have our application working (with decreased performance) even if Cassandra clusters in 3 sites fail completely.

SimpleStrategy Versus NetworkTopologyStrategy – Use Cases:

I have read in several posts people suggesting that if you have a single data center, you should use Simple Strategy. However, it’s not so simple. Simple Strategy is completely oblivious to network topology, including rack configuration. If the replicas happen to be on the same rack, and the rack fail, you will suffer from data loss. Racks are susceptible to failures due to power, heat or network issues. So, I would recommend using Simple Strategy only when your cluster is in a single data center and on a single rack (or you don’t care about rack configuration).

How Cassandra knows your Network Topology?

There is no magic here. Cassandra doesn’t learn your topology automatically. You must define the topology such as assigning nodes to racks and data centers and give this information to Cassandra which then uses it. This mapping of nodes to racks and data center is done using Snitch, which I will discuss later.

~~~ Ignore below this line ~~~
PlanetCassandra