NiFi

NiFi is platform originally developed by the NSA as a data harvesting platform.

One can build a pipeline that can automate the harvesting data from different sources. We can use python for a more granular pipeline.

The platform is similar to Flink in that it has a sources, processors, and sinks. Being an Apache product, it is well integrated into the ecosystem.

Flink

Flink is a great platform for processing data at scale. It supports a python wrapper, Scala and Java.

Used by AliBaba, Flink can be setup to manage terabytes of data on a daily basis. It can allow for a company's data scientists to create modules that can be uploaded onto the platform making use of snapshots and large data processing.

Selenium and CyprusJS

Unit testing is part of software development. Along with ArtilleryJS, Selenium and CyprusJS help to test the fidelity of our code. We use a collection of libraries to query the result of our program and make sure that what we produced was really present.

We can help sure up our platform and have reportable results.