Added ElasticSearch Emitter to index Frames into an ElasticSearch Index

[maheshbabugorantla]

ElasticSearch Emitter

Resolves Issue #392

Summary of Changes

1. Added Elasticsearch Emitter
2. Added UnitTest to test formatting of crawler frames into elasticsearch document format

---

Results

Environment Setup

$ docker ps
CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS              PORTS                                            NAMES
e96b77434155        kibana:7.4.2          "/usr/local/bin/dumb…"   21 hours ago        Up 21 hours         0.0.0.0:5601->5601/tcp                           tender_dijkstra
a658b6357af5        elasticsearch:7.4.2   "/usr/local/bin/dock…"   6 months ago        Up 22 hours         0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   es_emitter_test

Crawling and Indexing frames in INVM Crawl Mode

$ sudo venv/bin/python crawler/crawler.py --url elastic://localhost:9200 --features os,disk,process,package --extraMetadata '{"iteration_number": 1, "hostname": "my_ubuntu_1804"}' --format json

$ sudo venv/bin/python crawler/crawler.py --url elastic://localhost:9200 --features os,disk,process,package --extraMetadata '{"iteration_number": 2, "hostname": "my_ubuntu_1804"}' --format json

Kibana Query (Filtering by extraMetadata fields)

hostname : "my_ubuntu_1804" and iteration_number : "1"

hostname : "my_ubuntu_1804" and iteration_number : "2"

[ricarkol]

Hi @maheshbabugorantla , thanks for the PR.

Could you sign your commits, please? this test is failing (https://github.com/cloudviz/agentless-system-crawler/blob/master/.validateDCO).

@sahilsuneja1 @nadgowdas we should disable this DCO check.

[sahilsuneja1]

I don't mind dropping the DCO check , but I thought that was a requirement?
@maheshbabugorantla: Thanks for this PR!! This fix is as simple as using the -s flag when you are committing a change: git commit -s -m "xxxx"

[maheshbabugorantla]

@sahilsuneja1 I have amended the commit and executed git push -f origin mbg-es-emitter and that sign-off the commit

I have also run the . ./validateDCO to confirm all the commits are signed

$ ./.validateDCO
TRAVIS_COMMIT_RANGE:
Commits: e5cf98bb4322cc326018663286a7d62bcdfeead7

Looking at commit: e5cf98bb4322cc326018663286a7d62bcdfeead7
Signed-off-by: Mahesh Babu Gorantla <[email protected]>

**** All commits are signed!

Can I reopen a new PR and close this one ?

[sahilsuneja1]

Hmm, I don't see any Signed-off-by: line in any of the commits associated with this PR.
The commit e5cf98b doesn't seem to be part of this PR.
Also, from what I remember, all commits would need to be signed.
Maybe squash them into one and sign it?
New PR is also ok.

[maheshbabugorantla]

Will Reopen a new PR

[maheshbabugorantla]

Updated PR