Privacy preserving collaborative filtering for SaaS enabling PaaS clouds

Motivating example

With data about individual’s personal preferences being increasingly stored by applications built atop various SaaS and PaaS cloud platforms, there is a growing need to ensure that privacy of such data is preserved while enabling efficient statistical operations on the data, even over multiple cloud applications belonging to multiple cloud sites. Consider the following example: Alice has seen and rated a number of films from the Japanese animation Studio Ghibli on a film rating web application built on a PaaS cloud. She intends to see the 2011 film: From Up on Poppy Hill next and would like the film rating website to give her a rating prediction for this film based on her ratings of the other films that she has rated as well as such ratings from the community. She is completely unaware of (and does not care) who else has rated the various films in this way apart from obtaining a reasonable rating for From Up on Poppy Hill. Alice also is unwilling to send her entire rating vector for films she has rated to any third party but is happy to send some in such a way that they are de-linked from her identity through some anonymising mechanism. If Alice is to obtain a rating for From Up on Poppy Hill, she would prefer confidentiality of the information and also does not want to reveal her identity in the prediction query. In the future, Alice may also change the ratings for any of her previously rated films.

Objectives

We also assume honest but curious user participation, although we discuss in this paper what happens if we give up this assumption. The formal problem statement is presented in Section “Problem Statement”.

To achieve this, in our approach, the user will assume the presence of anonymising techniques (e.g. IPv4 network address translation and dynamic IPs, anonymiser networks such as Tor, and pseudonymous identities; see: [3–9]) to de-identify himself/herself from his/her ratings sufficiently such that the complete rating vector for a user cannot be reconstructed. Thus our security guarantees are based upon the security guarantees provided by the underlying anonymising mechanism, and are bounded by it.

Contributions

The main contribution of this paper is that we build on our previously proposed work [2] and show that it is feasible to implement on both Amazon and Google Software-as-a-Service enabling Platform-as-a-Service clouds. We also include discussions on possible extensions of our idea. Note that other existing PPCF schemes do not have any cloud based implementations, hence our results are not comparable with implementation results of such PPCF schemes.

The rest of the paper is organised as follows: in §“Related Work”, we list some of the related work in privacy preserving collaborative filtering. In §“Background”, we introduce the building blocks for our proposed scheme, which is presented in §“Proposed Scheme”. We present the comparative evaluation of our implementation in §“Evaluation” preceded by practical implementation considerations in §“Implementation Considerations” before concluding with future directions in §“Conclusion and Future Work”.

Slope one predictors for collaborative filtering

The Slope One predictors due to Lemire and McLachlan [24] are item-based collaborative filtering schemes that predict the rating of an item for a user from a pair-wise deviations of item ratings. Slope One CF can be evaluated in two stages: pre-computation and prediction of ratings.

where ϕ_a,b is the count of the users who have rated both items while δ_i,a,b=r_i,a−r_i,bis the deviation of the rating of item a from that of item b both given by user i.

The matrices Δ and ϕ are called deviation and cardinality matrices respectively. These matrices are sparse matrices. We need to store the upper triangulars only because the leading diagonal contains deviations and cardinalities of ratings between the same items, which are irrelevant. The lower triangular for the deviation matrix is the additive inverse of the upper triangular while that of the cardinality matrix is the same as its upper triangular.

Anonymity and identifiability

The diagram in Figure 1 illustrates why a simple network address translator can provide a certain degree of anonymity by representing all IP addresses on its LAN side with just one IP address on its WAN side. A more complex and cryptographic pseudonymous method of anonymity was presented in [9].

Despite the importance of anonymity in this privacy preserving collaborative filtering technique, there is also a consensus on identifiability if there is a pressing need. In both the case of NAT and the case of pseudonyms, this identifiability can be obtained through information that the NAT administrator or pseudonymous group membership issuers provide. The process is not straightforward and requires offline intervention by administrators. Hence, the support for identifiability does not nullify the ability to anonymise the data.

Homomorphic cryptosystem

The Paillier cryptosystem with optimisations are described in three steps: key generation (algorithm 1), encryption (algorithm 2) and decryption (algorithm 3).

Algorithm 1

Paillier cryptosystem key generation.

           1: Generate two large prime numbers p and q, each with half the specified modulus bit length for the cryptosystem.

     Ensure: gcd(pq,(p−1)(q−1))=1 and p≠q.

           2: Modulus n=pq and pre-compute n².

           3: Compute $\lambda =\mathrm{lcm}(p-1,q-1)=\frac{(p-1)(q-1)}{\gcd (p-1,q-1)}$.

           4: $g\leftarrow (1+n)$. {Optimised here but originally: select random $g\in Z_{n^2}^{\ast }$ such that n divides the order of g.}

     Ensure: gcd(L(g ^λ mod n²),n)=1 where $L\left(u\right)=\frac{u-1}{n}$. {Optimisation: g ^λ mod n²=(1 + nλ) mod n².}

           5: Pre-compute the modular multiplicative inverse μ=L(g ^λ mod n²)⁻¹ mod n.

           6: return Public key: (n,n²,g) and private key: (λ,μ).

Algorithm 2

Paillier encryption algorithm.

       Require: Plaintext m∈Z _n .

         1: Choose random $r\in Z_n^{\ast }$.

         2: return: Ciphertext $c\leftarrow (1+\mathrm{mn})r^n\mathrm{mod}{n}^2$. {Optimised here but originally: $c\leftarrow g^m{r}^n\mathrm{mod}{n}^2$.}

Algorithm 3

Paillier decryption algorithm.

       Require: Ciphertext $c\in Z_{n^2}^{\ast }$.

         1: return: Plaintext $m\leftarrow L\left(c^{\lambda }\mathrm{mod}{n}^2\right)\mu \mathrm{mod}n$.

Our implementation of Paillier cryptosystem supports negative numbers for plaintext because quite often a deviation of two ratings can be negative. We have used an optimised version of a non-threshold Paillier cryptosystem. Due to the absence of the threshold servers, decryption is faster than the threshold version. To cater for negative plaintext inputs, we divide the ring of n in two parts and consider any plaintext $m\ge \frac{n}{2}$ as negative.

Problem statement

The problem can be formally defined in the following fashion:

Pre-computation

In the pre-computation phase, the plaintext deviation matrix and the plaintext cardinality matrix are computed. In the absence of full rating vectors from users and consistent user identification, the combination of the deviation and the cardinality matrices pose no privacy threat to the users’ private rating data. The collection of the rating data is done pair-wise and after the user identity is de-linked in the process through the use of known techniques, such as anonymising networks, mixed networks, pseudonymous group memberships, and so on. User submits a pair of ratings or the corresponding deviation to the cloud application at any point in time. Thus, if the user originally rated n items then $\frac{n(n-1)}{2}$ pair-wise ratings or deviations should be submitted. Since the user’s identity (e.g. a pseudonym or an IP address) can (rather, must) change between consecutive submissions, the cloud cannot deterministically link the rating vector to a particular user.

Algorithm 4

An algorithm for the addition of new ratings.

       Require: An item pair identified by a and b, ratings r _a and r _b , or the deviation δ_a,b=r _a −r _b has been submitted.

           1: Find the deviation Δ_a,band cardinality ϕ_a,v.

           2: {While looking for deviations and cardinalities, also look for their inverses, i.e. Δ_b,aand ϕ_b,abecause only the upper triangular is stored. If the inverses are retrieved then deviation must be inverted before operating on it.}

           3: if Δ_a,band ϕ_a,bnot found then

           4: ${\Delta }_{a,b}\leftarrow 0$ and ${\varphi }_{a,b}\leftarrow 0$.

           5: end if

           6: Update ${\Delta }_{a,b}^{″}\leftarrow {\Delta }_{a,b}+{\delta }_{a,b}$ and ${\varphi }_{a,b}^{″}\leftarrow {\varphi }_{a,b}+1$.

           7: Store ${\Delta }_{a,b}^{″}$ and ${\varphi }_{a,b}^{″}$.

     Ensure: While storing, write to the inverses ${\Delta }_{b,a}^{″}$ and ${\varphi }_{b,a}^{″}$ if these were initially retrieved. {If the inverses were retrieved then deviation must be inverted before storing it.}

           8: Audit this add operation in the datastore, e.g. using user’s IP address as the identity. {This is a typical insider threat in the cloud.}

Algorithm 5

An algorithm for the updates or deletions of existing ratings.

     Require: In case of update, an item pair identified by a and b, and ${\mathit{\text{diff}}}_{{\delta }_{a,b},{\delta }_{a,b}^{″}}$; or in case of deletion: an item pair identified by a and b, and −δ_a,b.

           1: Find the deviation Δ_a,band cardinality ϕ_a,b.

           2: {While looking for deviations and cardinalities, also look for their inverses, i.e. Δ_b,aand ϕ_b,abecause only the upper triangular is stored. If the inverses are retrieved then deviation must be inverted before operating on it.}

           3: if Δ_a,band ϕ_a,bnot found then

           4:       print error!

           5: end if

           6: In case of an update, ${\Delta }_{a,b}^{″}\leftarrow {\Delta }_{a,b}+{\mathit{\text{diff}}}_{{\delta }_{a,b},{\delta }_{a,b}^{″}}$; or in case of deletion: ${\Delta }_{a,b}^{″}\leftarrow {\Delta }_{a,b}-{\delta }_{a,b}$ and ${\varphi }_{a,b}^{″}\leftarrow {\varphi }_{x,y}-1$.

           7: Store ${\Delta }_{a,b}^{″}$ and ${\varphi }_{a,b}^{″}$, if ${\varphi }_{a,b}^{″}$ was changed in case of deletion.

     Ensure: While storing, write to the inverses ${\Delta }_{b,a}^{″}$ and ${\varphi }_{b,a}^{″}$ if these were initially retrieved. {If the inverses were retrieved then deviation must be inverted before storing it.}

8: Audit this update or deletion operation in the datastore, e.g. using user’s IP address as the identity. {This is a typical insider threat in the cloud.}

Prediction

The steps for the prediction is shown in algorithm 6.

Algorithm 6

An algorithm for the prediction of an item.

     Require: An item x for which the prediction is to be made, a vector $\overrightarrow{\mathrm{RE}}=\mathcal{ℰ}\left(r_{a|a\ne x}\right)$ of encrypted ratings for other items rated by the user (i.e. each item a|a≠x) and the public key p k _u of user u.

           1: total cardinality: $\mathrm{tc}\leftarrow 0$; total deviation: $\mathrm{td}\leftarrow 0$; total encrypted weight: $\mathrm{tew}\leftarrow \mathcal{ℰ}\left(0\right)$; total encrypted deviation: $\mathrm{ted}\leftarrow \mathcal{ℰ}\left(0\right)$.

           2: for$j=1\to \mathrm{length}\left(\overrightarrow{\mathrm{RE}}\right)$do

           3:         Find the deviation Δ_x,jand cardinality ϕ_x,j.

           4:        {While looking for deviations and cardinalities, also look for their inverses, i.e. Δ_j,xand ϕ_j,xbecause only the upper triangular is stored. If the inverses are retrieved then deviation must be inverted before operating on it.}

           5:        if Δ_x,jand ϕ_x,jfound then

           6:                $\mathrm{td}\leftarrow \mathrm{td}+{\Delta }_{x,j}$.

           7:               $\mathrm{tc}\leftarrow \mathrm{tc}+{\varphi }_{x,j}$.

           8:               $\mathrm{tew}\leftarrow \mathcal{ℰ}\left(\mathrm{tew}\right)\left(\mathcal{ℰ}{\left(r_j\right)}^{{\varphi }_{x,j}}\right)$. {This step involves a homomorphic addition and a homomorphic multiplication.}

           9:            end if

           10: end for

           11: $\mathrm{ted}\leftarrow \left(\mathcal{ℰ}\right(\mathrm{tew}\left)\mathcal{ℰ}\right(\mathrm{td}\left)\right)$. {This is a homomorphic addition.}

           12: return ted and tc. {User decrypts ted; the predicted result is $r_{u,x}=\frac{\mathcal{풟}\left(\mathrm{ted}\right)}{\mathrm{tc}}$}

In the scheme described above, there is, in fact, one privacy leakage in the prediction phase: the number of items in the user’s original rating vector. This can be addressed by computing the prediction at the user’s end with the necessary elements from the deviation and cardinality matrices obtained from the cloud. The user can mask the actual rating vector by asking the cloud for an unnecessary number of extra items. Note that the only privacy leakage in the prediction stage is the list of items in the query vector. The corresponding ratings can only be decrypted by the user’s private key. Further to this, the user is free to use a different key pair for each prediction request such that a single public key cannot be used by an adversary to link the queries together.

Vertical partition across multiple organisations

While the solution presented above works when each user knows their own ratings, it will not work if there are two or more cloud applications with disjoint item and user sets (e.g. Yahoo movies, Netflix and IMDB have their own cloud applications), the prior PPCF solution will work if we extend our scheme as follows. For example, assume the two applications know the values r _a and r _b respectively for two different items for the same user. To compute the deviation r _a −r _b for that user without revealing it to either or to any third site, assume that the first application randomly splits the value r _a into two shares (thus, randomly chooses z_{a 1}∈Z and compute z_{a 2}∈Z such that r _a =z_{a 1} + z_{a 2}). Similarly, the other application randomly chooses z_{b 1}and computes z_{b 2} such that r _b =z_{b 1} + z_{b 2}. The first application sends z_{a 2} to the other, and after receiving −z_{b 1} from it, computes c₁=z_{a 1}−z_{b 1}, while the other computes c₂=z_{a 2}−z_{b 2}. Finally, both can send c₁and c₂respectively in an unlinkable fashion to the cloud PPCF application, which obtains c₁ + c₂=z_{a 1}−z_{b 1} + z_{a 2}−z_{b 2}=r _a −r _b . It is possible to do this with k cloud applications as well, where each value is split into k splits, to make the process more resistant to collusion. Note that one problem with this approach is that the same deviation is split over the k splits – effectively, cardinality is increased by k instead of by 1. One approximate fix for this is simply to increase the deviation k-fold. Thus, each application will send in k∗c _i , instead of c _i . However, this will only approximate the effect since instead of adding the true deviation and 1, we end up adding k times the true deviation and k. Alternatively, the cardinalities can be fixed by exposing a reduction capability, so that anyone can reduce the cardinalities appropriately in an unlinkable fashion. Assuming semi-honest participants, this works fine. If this is not suitable, the cardinalities can also be computed correctly by flagging to the PPCF application that submissions of all c _k splits should be treated as one submission. However, in this case, the linkage between the splits will be established and now the security relies on the lack of collusion among all k applications. We will explore this issue more in the future, and also leave the experimental results for it for future work.

Database access and configurability

Efficiency and parallelism

Attack model: malicious cloud

The implementation of our model accepts, as input from a user, ratings for a pair of items, or the deviations of their ratings. The implementation quietly collects the user’s IP addresses without the user’s permission (constituting an example attack scenario). With the presence of IP address, a level of linkability between rating pairs can be attained. However, through the use of a network address translator (NAT), or an onion routing protocol (e.g. Tor) or dynamic IP addresses, the user’s IP addresses act as pseudonyms with a many-to-many relation between a pseudonym and a real user, i.e. one IP address may be re-used by many users and one user may have many IP addresses over time. This makes predictable linkability hard and often incorrect. If the user uses different IP addresses for each separate pair of ratings then at any point in time, the server can at most link only one pair of ratings to a user. In reality, it will link more (and these are wrong) because of the IP addresses being re-used by other users.

Note that the server can also use browser-based cookies to more uniquely identify a particular browser, hence a user. However, we limit our discussion of the attack model to IP addresses only because even with browser cookies, it is possible for the same user to be identified separately, e.g. using more than one browsers; or different users identified by the same browser, e.g. a browser on a publicly shared user account. In addition browser cookies can be selectively blocked and the users can be warned of such cookies, thus making the user identification process less transparent than the one using IPs.

Potential extension to vertical partitions

While we have not explicitly experimented with an implementation of the algorithms for vertical partitions (given in Section “Vertical Partition Across Multiple Organisations”), we now discuss potential implementation and performance. It is important to note that the communication between the applications is independent of the PPCF application residing in the PaaS cloud. The different applications, holding vertical partitions, will need to follow some protocol to exchange randomised values between each other. From the perspective of the PPCF application, the performance should be largely unaffected by the process because the addition of the different randomised parts can happen in parallel. Once submitted, those parts are used to update the internal deviation matrix. The query stage is completely unaffected by this process, and so is the time performance of the query. The only scope of a performance slowdown depends on how fast the individual applications collaborate with each other to generate the partial randomised deviations.

Pre-computation

In the pre-computation stage, there is no cryptographic operation on the cloud.

In the case of Google App Engine, the application latency is dominated by the time taken to complete a datastore write operation. Each such datastore write operation took between 80ms and 150ms using the High-Replication datastore and about half that time while using the Master-Slave datastore. We also performed bulk addition of pair-wise deviations. Figure 3 is a screenshot of the Google App Engine control panel showing the typical load generated during the bulk data addition procedure. The bulk adding client generated 32 threads to process a part of the MovieLens 100K^c dataset. The figure shows data of the 14 automatically allocated application instances^d. Each such instance can handle multiple requests and are pooled in memory. The QPS (queries per second) column shows the average number of queries per second an instance received while the latency is the average time taken by that instance to complete one such request. This simulates the scenario if many users concurrently submit many requests to add rating or deviation data.

In the case of the Amazon Elastic Beanstalk, we adopted a very different approach to loading the complete MovieLens 100K dataset. We had direct access to port 3306 of the MySQL server EC2 instance running as part of Amazon Relational Database Service (RDS). Consequently, we pre-computed the deviation and cardinality matrices from the 100K dataset user-item rating matrix and then used plain SQL to add that data. The process was faster and more reliable than adding data to Google App Engine’s datastore.

Prediction

The prediction stage involves one homomorphic encryption as well as several homomorphic multiplications. Therefore, increasing the size of the encrypted rating vector typically linearly increased the time taken to predict. It is not dependent on the size of the deviation and cardinality matrices. This is shown in Tables 1 and 2. Since the recent version of GAE/J allows front-end instance class configuration, we have also included prediction test results with the 1200MHz, 256MB RAM and the 2400MHz and 512MB RAM front-end instance classes in Tables 3 and 4 respectively. Note that the results presented in Table 1 were obtained using the previous version of the GAE/J which did not allow front-end instance class configuration. The front-end instance class from the previous version roughly compares to the 600MHz, 128MB RAM default instance class in terms of performance.

In terms of prediction performance, the less time it takes, the better. In a realistic example, the prediction of several items ought to be obtained with the user waiting to see some meaningful recommendations. While the prediction may be obtainable in parallel and reported asynchronously to the user (which falls in the remits of future work), users will still expect meaningful recommendations presented to them in reasonably quickly, which is often in order of just few seconds on a web browser or a mobile application.

Note that given a 2048-bit Paillier cryptosystem, the total prediction time with 10 encrypted ratings as the input vector is reasonably fast: about 5 seconds, while the prediction time improves by almost eight-fold on the Google App Engine if we use a 1024-bit cryptosystem. Sometimes, even if the input vector is large, pair-wise ratings between the queried for item and the items in the input vector may not exist, which will reduce the prediction time. Another factor impacting on performance is the availability of the deviation and cardinality matrix data on the distributed in-memory cache versus the datastore, although in our Amazon implementation we did not take advantage of Amazon’s in-memory cache – the ElastiCache. In addition, GAE/J instances may also perform better or worse depending on the shared resources availability on the Google’s cloud computing clusters. In contrast with the Google App Engine, the results on the Amazon Elastic Beanstalk are significantly faster, with almost a 5.5 fold performance increase for a 2048-bit cryptosystem and query vector size 10.

In addition, we note that the t1.micro instance on the AWS/EBS still outperforms the GAE/J 2400MHz front-end instance class when using the 2048-bit cryptosystem. This is explicable from the description of Amazon’s micro instances^e, which shows that not only do t1.micro instances have more memory (at 600MB) than the GAE/J but also have more powerful CPU available in short bursts. In fact, the micro instances are particularly suitable for applications requiring occasional short bursts of CPU, which is what our experiments do. Figure 4 presents a performance comparison of the various GAE/J instance classes as well as the t1.micro of AWS/EBS. The diagram shows each class has four different performance values. The leftmost is the one for 1024 bits with a query vector size of 5; followed by that with the query vector size of 10; and then followed by that with the query vector size of 5 for the 2048 bit cryptosystem and the rightmost using the 2048 bits cryptosystem with the query vector size 10.

Security