How do I use this library with Apache Spark?

[Ilya_B_Intel]

Intel® DAAL provides algorithms implementations for distributed computing.

All data management objects support Java serialization and can be elements of Apache Spark RDD collections.

Algorithms are divided into 2 or more steps which correspond to general map/reduce computation tasks (and in more complex cases sequence of map/reduce tasks).

Simpler scheme can be described in the following way:

• You do map on available portions of data stored in RDD

  • you call computation of 1st step of chosen algorithm for each element of RDD and obtain PartialResult object

  • return PartialResult in RDD

• You call collect() on PartialResults RDD 

  • you call computation of 2nd step of chosen algorithm and obtain Results object which contains algorithm results

More details for each particular algorithm can be found within programming guide.

Package is supplied with four Spark samples, which implements this scheme for PCA (Correlation and SVD method), QR and SVD decompositions. We will extend number of samples in further updates. Let us know if you are interested in some specific algorithms.