PRICING

HELP

BLOG

APPSTORE

COMPANY

LEGAL

LOGIN

In this article we will demonstrate how you can leverage your IzyCloud data to effectively and efficiently perform complex analysis on the organizations commucation archives including mails, text messages, slack message, etc. Each bit of communication across employees or between customers and CSMs will help you identify trends and patterns that can be beneficial in optimizing your business process.

Setup the Environment

You should enable the IzyCloud ElasticSearch service from your dashboard. If you do not have an enterprise dashboard, you should first setup ES. If you do, then skip to the next step.

Setup Elastic Search for Non-Enterprise Environment

Please follow these steps for Setting Up Elastic Search. Then you should be able to run elasticsearch.

The easiest way to install it is by using docker:

docker pull docker.elastic.co/elasticsearch/elasticsearch:6.3.2

add -d below to run in the background

docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:6.3.2

Check this link for the elastic search server status.

Define Data Schema and Index

To consume the content in ES, the message payloads need to be packaged as documents. Documents in Elasticsearch are represented in JSON format. Also, documents are added to indices, and documents have a type.

The (index, type, document keys) design decision will affect how search queries may be constructed:

/_search?q=foo

/_search?q=key:value

/index/type/_search?q=key:value

/index1/type1,type2/_search?q=key:value

...

it will also affect how and what aggregation operations may be performed on the data.

To get better control over the fields, we should also define custom mappings. Mapping is the process of defining how a document, and the fields it contains, are stored and indexed. While eastic search will create the mapping automatically, sometimes it is beneficial to customize the mapping.

Handling Duplicate Data

Eliminating duplicates is a must. This will ensure that:

• The aggregate queries correctly represent the underlying data
• Reruns of existing data wont impact the index
• ..

Fortunately IzyCloud assigns each payload a unique URL that can be used as the document id for the ES index. To do this, simply refer to using your own ID. You may simply pass in the HEX value of the fingerprint field for the item in the INSERT call.

Data Schema

var esData = {

body: data.body,

from: data.sender,

to: [data.to],

subject: data.subject,

cc: [data.cc],

utcDate: data.date

};

Analysis

To count the number of documents, do:

http://localhost:9200/contentindex1/email/_count

You may want to use Kibana to get a better view.

Kibana

See this article for further information visualization using Kibana.

Setting up Kibana

First,

docker pull docker.elastic.co/kibana/kibana:6.3.2

If you are not sure about the comfiguration for the environment, you can try:

docker inspect docker.elastic.co/kibana/kibana:6.3.2

Under ContainerConfig, you can see:

"Cmd": [

"/bin/sh",

"-c",

"

(nop) ",

"CMD [\"/usr/local/bin/kibana-docker\"]"

]

Also, under Config you can see:

"ExposedPorts": {

"5601/tcp": {}

}

Make sure that networking is setup correctly:

add -d below to run in the background

docker run -e ELASTICSEARCH_URL=http://host.docker.internal:9200 -p 5601:5601 docker.elastic.co/kibana/kibana:6.3.2

You may see:

kibana docker Login is currently disabled Administrators should consult the Kibana logs for more details.

That is a misleading image because it can result from the elastic search service not being accessible from inside the docker image.

For trouble shooting you can do :

docker run -v /plat/p/apps/elasticsearch/kibana:/izyhostdir -i -t -e ELASTICSEARCH_URL=http://host.docker.internal:9200 -p 5601:5601 docker.elastic.co/kibana/kibana:6.3.2 /bin/bash -c "/izyhostdir/ping.sh"

Use ping.js node script:

const http = require('http');

http.get('http://host.docker.internal:9200', (resp) => {

var data = '';

// A chunk of data has been recieved.

resp.on('data', (chunk) => {

data += chunk;

});

// The whole response has been received. Print out the result.

resp.on('end', () => {

console.log('OK:' + data);

});

}).on("error", (err) => {

console.log("Error: " + err.message);

});

And /izyhostdir/ping.sh:

/usr/share/kibana/node/bin/node /izyhostdir/ping.js

Create IzyWare task to feed data into ES

Follow the instructions in the post manage-and-automate-your-day-to-day-business-tools-using-izycloud-tasks-part-i to setup a new task.

using your own ID: https://www.elastic.co/guide/en/elasticsearch/guide/current/index-doc.html

manage-and-automate-your-day-to-day-business-tools-using-izycloud-tasks-part-i: https://izyware.com/help/article/manage-and-automate-your-day-to-day-business-tools-using-izycloud-tasks-part-i

visualization using Kibana: https://medium.com/@orweinberger/analyze-and-visualize-your-gmail-inbox-using-elasticsearch-and-kibana-88cb4e373c13

Setting Up Elastic Search: https://www.elastic.co/guide/en/elasticsearch/reference/current/setup.html

elastic search server status: http://127.0.0.1:9200/

Izyware Blog