Friday, October 23, 2015

Create ElasticSearch cluster on single machine

I wanted to figure out how to create a multi-node ElasticSearch cluster on single machine. So i followed these instructions


  1. First i did download elasticsearch zip file.
  2. Create 3 folders in  like node1, node2 and node3
  3. Unzip elasticsearch.zip in each one of these folders and name the folder node1,node2,node3
  4. Then i opened the node1/config/elasticsearch.yml and i did change value of cluster.name to samarcluster and value of node.name to node1,node2 and node 3  as shown below.

Typical setup done with 3 nodes on a single machine

node1: 
cluster.name: samarcluster
node.name: "node1"
node.master: true
node.data: true
node2 : 
cluster.name: samarcluster
node.name: "node2"
node.master: false
node.data: true
node3 : 
cluster.name: samarcluster
node.name: "node3"
node.master: false
node.data: false

Next step was to install below Plugins
Marvel on each of the nodes by executing bin/plugin -i elasticsearch/marvel/latest
ElasticSearch-HQ on each of the nodes by executing bin/plugin -install royrusso/elasticsearch-HQ

Once all the 3 nodes are started, i could see them in marvel and elasticHQ as shown below.









Thursday, October 8, 2015

Elastic Search MongoDB Integration

Follow below steps to install elastic search, assuming mongoldb is already installed.


2) export ES_HOME=/Users/samaraarkotti/Documents/elasticsearch-1.7.2  
3) $ES_HOME/bin/plugin -install elasticsearch/elasticsearch-mapper-attachments/2.7.1   (  https://github.com/elastic/elasticsearch-mapper-attachments )
4) $ES_HOME/bin/plugin -install com.github.richardwilly98.elasticsearch/elasticsearch-river-mongodb/2.0.9 ( https://github.com/richardwilly98/elasticsearch-river-mongodb/ )
5) $ES_HOME/bin/plugin --install mobz/elasticsearch-head
6) $ES_HOME/bin/plugin -i elasticsearch/marvel/latest
7) start elastic search bin/elasticsearch

Optional Plugin to install 

$ES_HOME/bin/plugin --install lukas-vlcek/bigdesk ( To access Inquisitor, open http://localhost:9200/_plugin/bigdesk)
$ES_HOME/bin/plugin -install polyfractal/elasticsearch-inquisitor   ( To access Inquisitor, open http://localhost:9200/_plugin/inquisitor/

curl -XPUT 'http://localhost:9200/_river/mongodb/_meta' -d '{"type": "mongodb","mongodb": {"db": "analytics","collection": "Article"},"index": {"name": "articleindex","type": "article"}}'
curl -XPUT 'http://localhost:9200/_river/mongodb/_meta' -d '{"type": "mongodb","mongodb": {"db": "analytics","collection": "ViewsAndRatingDetails"},"index": {"name": "viewindex","type": "views"}}'



-- DELETE Plugins using curl or directly on web both articleindex and _river , once done , run above curl command to index again .


Reference 



--- if getting any error's converting your standalone mongodb to replicaset , follow below steps
1)start the server in your usual way (single instance)
2)connect to the instance with a similar command : ./mongo --port 27017
3)look at your databases : show dbs
4)if there is a db called local, connect to it:  use local
drop that database : db.dropDatabase()



--- if you are getting error saying _mongodb RIVER FAILED to start ,
1) Then delete the _river plugin ( curl -XDELETE 'http://localhost:9200/_river/mongodb' )
2) shutdown ES and restart


-- disable dynamic mapping  -- NOT WORKING

curl -XDELETE localhost:9200/articleindex    --- First delete the index
curl -XPUT localhost:9200/articleindex -d '{"index.mapper.dynamic": false}'

--added below in elasticsearch.yml under config folder 
you will also need to configure elastic search to accept requests from the browser using CORS ( cross origin site scripting). To enable CORS, add the following to elasticsearch's config file. Usually, this file is located near the elasticsearch  at config/elasticsearch.yml

http.cors:
  enabled: true
  allow-origin: /https?:\/\/localhost(:[0-9]+)?/
  
  

  
  
  

Thursday, April 16, 2015

ElasticSearch and Mongo through river – scalable data store and search engine platform

1. MONGODB
MongoDB is an open-source document database with built in replication, high availability, auto-sharding and map/reduce mechanisms.Install MongoDB from 
http://docs.mongodb.org/manual/tutorial/install-mongodb-on-os-x/

2. ELASTICSEARCH
Elastic search is a powerful open source, real-time search and analytics engine, designed from the ground up to be used in distributed environments with reliability and scalability as a must have. Looks great as a search engine.

Download Elastic Search from: https://www.elastic.co OR Use below command from terminal.
Export ES_HOME=/Users/xxx/Downloads/elasticsearch-1.4.2

3. MONGODB RIVER PLUGIN FOR ES
Elastic search provides ability to enhance the basic functionality by plugins, which are easy to use and develop. They can be used for analysis, discovery, monitoring, data synchronization and many others. Rivers is a group of plugins used for data synchronization between database and elastic search
The first is a dependency called Mapper Attachments. You can install via the ES plugin script:
$ES_HOME/bin/plugin -install elasticsearch/elasticsearch-mapper-attachments/2.4.3

The second plugin is the ES 'river' for Mongo. The syntax to install it is slightly different as it's a third-party plugin:
$ES_HOME/bin/plugin -install com.github.richardwilly98.elasticsearch/elasticsearch-river-mongodb/2.0.9

Restart the server 
                        sh elastic search restart

Elastic Search is on board and running – 3 minutes in my case. It’s recommended to use the second machine to avoid sharing resources, but for test deployments a single one is good enough.

Tell elastic search to index the “person” collection in testmongo database by issuing the following command in your terminal.

    "type": "mongodb",
    "mongodb": {
        "db": "testmongo",
        "collection": "person"
    },
    "index": {
        "name": "mongoindex",
        "type": "person"
    }
}'

We’ve got ElasticSearch automatically synchronizing data with MongoDB
2 minutes in my case.

4. Setup MONGODB As Replica Set.

MongoDB -- setting up replica set on the local host( Mac OS X)

mongod --port 27017 --dbpath /data/db --replSet rs0
rs.initiate()  -- > this will initiate new replica set.

Create a Mongo Database: testmongo
Create a Mongo Collection under testmongo: person

5. Now finally start the Search.

Use this command to search the data from terminal or go to URL directly to see below response

Response from Elastic Search :

{
  "took": 42,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 1.4054651,
    "hits": [
      {
        "_index": "mongoindex",
        "_type": "person",
        "_id": "552ee22d9f829d905d5f180f",
        "_score": 1.4054651,
        "_source": {
          "lastName": "Doe",
          "_id": "552ee22d9f829d905d5f180f",
          "firstName": "John"
        }
      }
    ]
  }
}
After inserting document in MongoDB configured as replica set, it is also stored in oplog collection.The mentioned collection is operations log configured as capped collection, which keeps a rolling record of all operations that modify the data stored in databases. River plugin monitors this collection and forwards new operations to elasticsearch according to its configuration. That means that all insert, update and delete operations are forwarded to elasticsearch automatically..

We can easily check what we have in ES using head plugin, which can be installed with the help of command:
./plugin -install mobz/elasticsearch-head

Some elasticsearch plugins provide web interface that can be reached using endpoint /_plugin:





Summarizing, we have MongoDB configured as replica set, Elasticsearch with River that pulls data from database to index, and finally everything is prepared for sharding and replication.

Tuesday, April 14, 2015

Singleton Pattern

Singleton Pattern in Java

Singleton Pattern ensures a class has only one instance and provides a global point of access to it.
The default constructor of the class is made private, which prevents the direct instantiation of the object by other classes.

A static modifier is applied to the instance method that returns the object as it then makes this method a class level method that can be accessed without creating an object.

public class BookingFactory {
 
private static BookingFactory instance;
// An instance attribute.
private int data = 0;
private BookingFactory() {
  //initiate any other attributes if needed.
}
 
public static BookingFactory getInstance(){
  if(instance == null
      instance = new BookingFactory();
  return instance;
}

public int getData() {
  return data;
}

public void setData(int data) {
  this.data = data;
}
// other methods....
}

In future if you get a requirement for having more than one instance then singleton allows multiple instances without affecting a singleton class's clients. Just you need to do is make a small change in Singleton Class, which doesn't effect the client's code.

Note that the singleton instance is only created when needed. This is called lazy instantiation.

public class SingletonDemo {
 
public static void main(String args[]) {
// Get a reference to the single instance of Singleton.
BookingFactory bookingFactory = BookingFactory.getInstance();
 
// Set the data value.
bookingFactory.setData(34);
 
System.out.println("First reference: " + bookingFactory);
System.out.println("Singleton data value is: " +bookingFactory.getData());
}
 
}

How can we break Singleton:

1. It could happen that the access method may be called twice from 2 different classes at the same time and hence more than one object being created. This could violate the design patter principle. 

Solution :

In order to prevent the simultaneous invocation of the getter method by 2 threads or classes simultaneously we add the synchronized keyword to the method declaration.

Make the Instance access method Synchronized to prevent concurrent thread access.

public static synchronized BookingFactory getInstance()

Synchronization is expensive, however, and is really only needed the first time the unique instance is created.

Do an eager instantiation of the instance rather than a lazy instantiation.

You can also instantiate the Singleton as soon as the class loads. i.e You can place it in static block to ensure that it happens only once.

public class BookingFactory {
 
private static BookingFactory instance;
// An instance attribute.
private int data = 0;
 
static {
  instance = new BookingFactory();
}
 
private BookingFactory() {
  //initiate any other attributes if needed.
}
 
public static BookingFactory getInstance()
{
  return instance;
}

public int getData() {
  return data;
}

public void setData(int data) {
  this.data = data;
}
// other methods....
}

Instead of synchronizing the whole method you can also make the instance variable as static final. It is thread-safe because static member variables created when declared are guaranteed to be created the first time they are accessed. You get a thread-safe implementation that automatically employs lazy instantiation.

public class BookingFactory {

  private final static BookingFactory instance = new BookingFactory();

  private BookingFactory() {
        // Exists only to defeat instantiation.
   }

}

But you loose the flexibility of having more than one instance in future without changing client's code.

2. if you are using multiple class-loaders  this could defeat the Singleton implementation and result in multiple instances. 

Solution:

Because multiple class-loaders are commonly used in many situations—including servlet containers—you can wind up with multiple singleton instances no matter how carefully you've implemented your singleton classes. If you want to make sure the same class-loader loads your singletons, you must specify the class-loader yourself; for example:

private static Class getClass(String classname) 
        throws ClassNotFoundException {
ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
if(classLoader == null)
classLoader = Singleton.class.getClassLoader();
return (classLoader.loadClass(classname));
}

}

The preceding method tries to associate the classloader with the current thread; if that classloader is null, the method uses the same classloader that loaded a singleton base class. The preceding method can be used instead of Class.forName().

3. If SingletonClass implements the java.io.Serializable interface, the class's instances can be serialized and deserialized. However, if you serialize a singleton object and subsequently deserialize that object more than once, you will have multiple singleton instances. 

Solution:

To avoid the above you need to implement readResolve() method.

private Object readResolve() {
    return INSTANCE;

}

The previous singleton implementation returns the lone singleton instance from the readResolve() method; therefore, whenever the Singleton class is deserialized, it will return the same singleton instance.

4. We will be able to create a copy of the Object by cloning it using the Object’s clone method.

Solution:

Override the Object clone method to prevent cloning, this can be done as shown below

SingletonDemo clonedObject = (SingletonDemo) obj.clone();

This again violates the Singleton Design Pattern’s objective. So to deal with this we need to override the Object’s clone method which throws a CloneNotSupportedException exception.

public Object clone() throws CloneNotSupportedException {
throw new CloneNotSupportedException();

}

When to use:
We can use this while creating objects of thread pools, caches etc to avoid wasting resources.

Create ElasticSearch cluster on single machine

I wanted to figure out how to create a multi-node ElasticSearch cluster on single machine. So i followed these instructions First i did...