Efficient and privacy-preserving image classification using homomorphic encryption and chunk-based convolutional neural network

Abstract

Image feature categorization has emerged as a crucial component in many domains, including computer vision, machine learning, and biometrics, in the dynamic environment of big data and cloud computing. It is extremely difficult to guarantee image data security, privacy, and computing efficiency while also lowering storage and transmission costs. This paper introduces a novel method for classifying image features that combines multilevel homomorphic encryption and image data partitioning in an integrated manner. We employ a novel partitioning strategy to reduce computational complexity, significantly reducing computational load and improving classification accuracy. In the quest for increased data security and privacy, we introduce a novel, fully homomorphic encryption approach specialized to partitioned images. To counter the inherent complexity of encryption, we devise a compound encryption strategy that exploits the full potential of homomorphic computation, with an explicit objective to curtail computational and storage overheads. Evidently superior to conventional methods, our methodology showcases pronounced benefits in computational efficiency, storage and transmission cost reduction, and robust security and privacy preservation. Hence, the methodology put forth in this paper presents a pioneering and efficacious resolution to the multifaceted challenges of image feature classification within the intricate milieu of cloud computing and big data.

Introduction

Deep learning has become integral to image classification applications, including facial recognition, object detection, and information forensics. These applications demand substantial computational resources and large-scale data sets to ensure optimal performance (Fig. 1). Centralized training, wherein individual training samples are uploaded to a third-party entity with extensive computational resources (e.g., cloud servers) to develop the final model, is the conventional approach, as shown in Fig. 2a. However, this method raises significant privacy concerns, as these third-parties might misuse sensitive data from multiple users.

Fig. 1

It shows how a modified convolutional neural network is used to process the original images after they have been encrypted using the Completely Homomorphic Encryption (CKKS) technique

Fig. 2

Illustration of various deep learning models. a Centralized Training: Depicts a model where users upload their individual training samples to a third-party entity, typically a cloud server with substantial computational resources, which is responsible for generating the final model. b Federated Learning Systems: Demonstrates an approach where models are trained locally, with only the gradients being shared, thereby ensuring that private data remains unexposed. c Decentralized Learning System: Represents a system where users have the autonomy to exchange gradients with their neighboring users and independently update their models

To address these concerns, distributing training tasks to the users is a plausible solution. This approach enables users to train the model locally and share only the gradients, thus eliminating the need to disclose private data, as shown in Fig. 2b. While this reduces privacy risks, attackers can still infer the attributes of the samples from the shared gradients. This threat is especially pronounced in decentralized learning systems such as the one shown in Fig. 2c, as any user connected to honest users can access their gradients and potentially compromise their data privacy [1].

In the era of cloud computing and big data, the outsourcing of data to cloud servers is becoming increasingly prevalent, thereby reducing the consumption of local storage and computational resources [2]. However, with the proliferation of malicious users on the internet and the inherent unreliability of cloud service providers, privacy and security concerns are paramount. Consequently, the development of an efficient and privacy-centric deep learning framework is of utmost importance [3].

Homomorphic encryption, which allows computation on encrypted data, is a promising solution to this issue. However, its practical application in deep learning is limited due to its high computational complexity. This paper presents a novel privacy-protecting image feature extraction method that integrates homomorphic encryption with deep learning techniques. By leveraging the inherent sparsity of image data, we reduce the computational burden of fully homomorphic encryption while maintaining the privacy of sensitive image data.

Homomorphic encryption is utilized to protect the privacy of user data operated on in cloud computing. Current research does not explicitly outline the specific method of segmenting images, computing gradient values, applying homomorphic encryption only to blocks containing important features, and then classifying the encrypted blocks. Therefore, the method in this paper is an optimized solution for image classification problems.

In our study, we confirm the effectiveness of our proposed approach through an empirical case study involving the utilization of a modified convolutional neural network (CNN) model [4]. This model, known for its capabilities in image recognition tasks, is the backbone of our approach. We pre-process the input image. This critical stage ensures that the image is in the best format for subsequent processing. After pre-processing, we divide the image into different chunks. This allows us to apply our method to each chunk independently, thus reducing computational complexity and increasing scalability. Once the image is partitioned into chunks, we apply a fully homomorphic encryption scheme to the chunks whose gradients exceed a certain threshold [5]. This selective encryption strategy ensures that we focus our computational resources on the most information-rich parts of the image, thus improving the efficiency of our method without compromising the accuracy of the results.

Our work makes several significant contributions (Fig. 3).

• We introduce a novel image splitting encryption method that reduces the time spent on homomorphic encryption computations.

• We propose a composite encrypted image scheme to lower the computational complexity associated with fully homomorphic encryption, while also reducing computational and storage costs.

• We validate our approach using a custom image classification model and assess its performance in terms of accuracy and computational efficiency.

Fig. 3

Depiction of an advanced image processing pipeline. The process initiates with the partitioning of the original image, followed by the full homomorphic encryption of blocks with distinctive features. These encrypted image blocks are then processed by a modified convolutional neural network

Related work

Homomorphic encryption

Image privacy protection is an important issue in the fields of mobile computing and cloud services. With the proliferation of mobile devices and cloud services, people are increasingly relying on these platforms to store and process personal data, including images. However, this also brings the risk of privacy leakage, as users’ data might be accessed or abused by unauthorized third parties. In recent years, the intersection of Homomorphic Encryption (HE) and image classification has garnered widespread attention, primarily due to the growing demand for data privacy in deep learning applications. Some contemporary research has made significant contributions in this area [6].

In a mobile cloud computing environment, homomorphic encryption can serve as a service to protect the privacy of outsourced images. For instance, one study introduced the concept of “Homomorphic Encryption as a Service” for protecting the privacy of outsourced images in mobile cloud computing environments [7]. This setup allows rich mobile applications to run on various mobile devices, while all data processing and storage are conducted in cloud services outside the mobile devices [8].

Homomorphic encryption technology can process ciphertext data while preserving privacy, and can directly search, compute, and statistic ciphertext on the cloud. This technology has four main aspects of application in cloud computing: (1) Retrieving encrypted data in cloud computing [9]. This shows that by using homomorphic encryption, users can protect their image data, while still being able to conduct some basic processing and analysis on this data without decryption.

Cloud computing is widely adopted due to its low cost, high reliability, and generic services. However, as transactions between users and service providers are often asynchronous, data privacy could lead to a crisis of trust, thus hindering the expansion of cloud computing applications. In a paper, the authors proposed a data privacy protection scheme based on homomorphic encryption to address this issue [10].

Overall, homomorphic encryption provides a potent solution for image privacy protection in mobile computing and cloud services. By using homomorphic encryption, the security and privacy of user data can be ensured, while still being able to conduct necessary computations and analyses on the encrypted data.

Image privacy protection

The issue of privacy protection in image data, particularly facial images, has been a focal point of recent research [11]. Several innovative methodologies have been proposed to address this concern [12], leveraging advanced techniques such as federated learning, ensemble models, and generative adversarial networks (GANs).

Yang et al. [13] proposed a transferable face image privacy protection method based on federated learning and ensemble models. Their approach utilizes a federated learning model established on distributed datasets and a local facial recognition model obtained through local face data training. The method demonstrated its effectiveness in generating private face images with high transferability and practicality.

In a different approach, Yu et al. [14] proposed a GAN-based differential private image privacy protection framework for the Internet of Multimedia Things (IoMT). Their method identifies privacy-sensitive content in images using deep neural network techniques and protects it with synthetic content generated by GANs with differential privacy. The proposed framework was shown to effectively protect users’ privacy while maintaining image utility.

Another significant contribution was made by Yang et al. [15] who proposed a facial image privacy protection method based on the principal components of adversarial segmented image blocks. Their method adds minor perturbation to the principal components of facial images to protect users’ privacy and prevent distinct face-related features of the images from being easily extracted. The proposed method outperformed other similar methods in terms of generated image quality, operation speed, and target recognition network accuracy.

These studies highlight the ongoing efforts in the field of image privacy protection and provide a solid foundation for further research in this area [16].

Homomorphic encryption for image classification

Currently, Homomorphic Encryption (HE) has been employed in image classification tasks, but due to its high computational complexity, a balance usually needs to be struck between choosing datasets and ensuring classification accuracy.

In a study, the aim was to analyze data from The Cancer Genome Atlas (TCGA) dataset, which comprises 3,622 samples, covering 11 types of cancer, extracting genetic features from 25,128 genes. Through preprocessing methods, the number of genes was reduced to 4,096 or fewer, and a microAUC value of 0.9882 (85% accuracy) was achieved using a 1-layer shallow neural network [17].

Another study demonstrated that by optimizing the activation functions, the classification accuracy of CIFAR-10 could be improved by 4.27%. Moreover, pretraining the activation function optimization for Fashion-MNIST and CIFAR-10, even in different networks and datasets, could enhance the classification accuracy [18].

From these, it can be seen that homomorphic encryption has already achieved a certain level of accuracy in image classification tasks, though some optimization may be needed to enhance efficiency.

Methods

Preliminary preparation

Our proposed preliminary research and planning phase for constructing an encrypted classification model includes two main components: data preprocessing and building the initial Convolutional Neural Network (CNN) model [19]. Data preprocessing makes the data more suitable for analysis and modeling, and can improve the accuracy of the model. The subsequent image splitting CNN and encrypted fast CNN models for processing encrypted data need to be modified based on the initial CNN model.

Image preprocessing

This initial step is crucial for normalizing and enhancing the input data of the CNN model, thereby ensuring effective feature learning. We use the NEU-CLS dataset and perform a series of transformations [20].

• 1. Image Size Adjustment: Using the OpenCV library, we adjust all images to a uniform size of 112x112 pixels, thereby ensuring the consistency of the input data, which is a prerequisite for CNN models that require fixed-size input.

• 2. NEU-CLS Dataset: The NEU-CLS dataset of steel surface defects is a surface defect dataset that collects six typical surface defects of hot-rolled steel strip, namely rolling scale (RS), patches (Pa), cracks (Cr), pitted surface (PS), inclusions (In), and scratches (Sc). The dataset contains 1800 grayscale images, six different types of typical surface defects, each type of defect contains 300 samples. For defect detection tasks, the dataset provides annotations indicating the category and location of defects in each image [21].

• 3. Contrast Enhancement: We use histogram equalization to enhance image contrast, thereby improving the intensity distribution of the image, making hidden details more clearly visible, and improving the overall image quality. This stage is particularly important because it helps the model accurately distinguish features during the training process. The formula for histogram equalization is [22]:

  $$\begin{aligned} H(v)= \sum \limits _{i=0}^v h(i) \end{aligned}$$

  (1)

  where H(v) is the cumulative distribution function and h(i)is the histogram of pixel intensities.

• 4. Data Augmentation: To enhance the robustness of the model and prevent overfitting, we use data augmentation techniques, including rotation ($\pm 15^{\circ }$), horizontal flipping, and translation (±10%). Data augmentation mainly produces changes in the image, enriches the dataset, and allows our model to learn from a wider range of examples.

Original image classification model

Given the complexity of the NEU-CLS dataset, we construct a simple neural network for classification. Our aim is to validate the accuracy of classification after image block encryption, hence we opt for a Convolutional Neural Network (CNN) for training and testing, which sufficiently demonstrates our method. With fewer parameters, we can train an excellent model and effectively avoid overfitting [23].

• 1. Network Design The adjusted neural network targets encrypted data, starting with a convolutional layer with 7x7 convolution kernels, followed by an activation function layer and a fully connected layer with 256 input neurons and 128 output neurons. Finally, there is an activation function layer and an output layer with 128 input neurons and 6 output neurons. These features are combined for encrypted block image classification [24]. The calculation method of the convolutional layer is shown in Eq. 2:

  $$\begin{aligned} (F * I)(i, j)=\sum _{m} \sum _{n} F(m, n)(i-m, j-n) \end{aligned}$$

  (2)

  where, F is the filter or kernel, I is the input image, and (i, j) are the spatial coordinates.

• 2. Activation Function After the convolutional layer and pooling layer, the Rectified Linear Unit (ReLU) activation function is applied. ReLU introduces non-linearity without affecting the receptive field of the convolutional layer. The Rectified Linear Unit (ReLU) activation function is represented as Eq. 3:

  $$\begin{aligned} f(x)=max(0,x) \end{aligned}$$

  (3)

• 3. Optimization and Training We use cross-entropy as the loss function, which is the standard choice for classification problems. In terms of network optimization, we adopt the Adam optimizer, which can adaptively adjust the learning rate of each weight. The model is trained on the training set and validated on the validation set to evaluate its performance. The Adam optimizer updates the weights based on the adaptive estimation of the low-order moments [25]. The update rule of Adam can be represented as:

  $$\begin{aligned} \theta _{t+1}=\theta _t-\frac{\eta }{\sqrt{\hat{v}_{t+1}}+\epsilon }\hat{m}_{t+1} \end{aligned}$$

  (4)

  where, $\theta$ is the parameter (weight), $\eta$ is the learning rate, $\hat{v}_{t+1}$ and $\hat{m}_{t+1}$ are the estimates of the first and second moments of the gradient, and $\epsilon$ is a small constant to avoid division by zero.

Chunks classification model

Initially, we constructed a simple CNN model to process raw plaintext image data. However, to handle chunked image data, it was necessary to modify the original CNN model to accommodate chunked data [26].

• 1. Network Design The architecture of our CNN model was inspired by the classic design principles of convolutional networks [27]. Similar to the original CNN model, the adjusted neural network targets encrypted data, starting with a convolutional layer with 7x7 kernels, followed by a fully connected layer with 256 input neurons and 128 output neurons, and finally an activation function layer and a fully connected layer with 128 input neurons and 6 output neurons [28]. These features are combined to classify encrypted chunked images. The purpose of these layers is to learn hierarchical spatial features by applying filters to the input images and to capture increasingly complex patterns as the network deepens. They reduce the spatial size of the feature maps while retaining key information, thus lowering the computational complexity of the network and providing translational invariance. The fully connected layer is a major part of the CNN architecture, adjusted to accommodate 256 input neurons. This layer combines the high-level features learned in the preceding layers to perform the image classification task [29]. Finally, an output layer comprising six neurons is implemented, corresponding to the six categories in our dataset. Each neuron in this layer represents a different category, and the activation of these neurons provides a probability distribution of the possible categories for a given input image [30].

• 2. Activation Function After the convolutional and pooling layers, the Rectified Linear Unit (ReLU) activation function is applied, introducing non-linearity without affecting the receptive field of the convolutional layer [31]. The structure of the chunked convolutional neural network is shown in the figure:

• 3. Optimization and Training During network training, we use cross-entropy as the loss function [32]. Cross-entropy is a standard choice for classification problems as it measures the difference between predicted class probabilities and true class labels [33]. For network optimization, we employ the Adam optimizer, an algorithm capable of adaptively adjusting the learning rate of each weight in the network [34]. This optimizer combines the advantages of two other popular optimization algorithms, AdaGrad and RMSProp, providing an effective and efficient optimization method.

As shown in Fig. 4, the pre-classified chunk set is input into the modified adaptive chunk CNN model for training and testing. The classification results are then compared with the actual results.

Fig. 4

Demonstration of the model structure performing convolutional operations on segments of the original image. The depicted convolutional network comprises one convolutional layer and two fully connected layers. The Rectified Linear Unit (ReLU) activation function is employed to introduce non-linearity, without impacting the receptive field of the convolutional layer

The above data preprocessing and the construction steps of the original image CNN model and chunked image CNN model lay the foundation for subsequent homomorphic encryption and encrypted image classification stages.

Image chunking and encryption strategy

The second phase of our proposed method involves dividing the image into chunks and performing gradient calculations on these chunks. Homomorphic encryption (HE) is applied to image chunks that exceed a threshold value based on the calculation results, with the primary goal of protecting data privacy without hindering subsequent classification accuracy processes [35].

Image splitting

 

• 1. Image Splitting Each preprocessed image in the dataset is divided into small subregions of 28x28 pixels. This division simplifies the task of identifying different features in the image and helps to reduce the complexity of the problem [36].

• 2. Local Gradient Calculation Upon segmenting the image, we commence the computation of local gradients for each segment. This computation is executed utilizing the Sobel operator, a mature tool in image processing, capable of determining the edge direction and edge strength at every point within the image. These edge features are pivotal in image analysis and interpretation as they often correspond to significant boundaries within the image, such as the contours of objects [36]. The Sobel operator operates by taking the weighted difference of grayscale values of the pixels in the four neighboring regions (up, down, left, right) of each pixel in the image, reaching a maximum at the edges hence detecting them. It is a discrete differential operator used to approximate the gradient of the image intensity function. Applying this operator at any point within the image will generate the corresponding gradient vector or its normal vector. The Sobel operator used to calculate the local gradient can be represented as follows: For the x direction

  $$\begin{aligned} G_x= \left[ \begin{array}{lll} -1 &{} 0 &{} +1\\ -2 &{} 0 &{} +2\\ -1 &{} 0 &{} +1 \end{array}\right] *I \end{aligned}$$

  (5)

  For the y direction

  $$\begin{aligned} G_y= \left[ \begin{array}{ccc} -1 &{} -2 &{} -1\\ 0 &{} 0 &{} 0\\ +1 &{} +2 &{} +1 \end{array}\right] *I \end{aligned}$$

  (6)

  where $G_x$ and $G_y$ are the gradients in the x and y directions respectively, and I is the image. The star (*) represents the convolution operation. The formula for calculating the gradient magnitude is

  $$\begin{aligned} G=\sqrt{G_x^2+G_y^2} \end{aligned}$$

  (7)

• 3. Feature Information Recognition Following the computation of local gradients, we proceed to identify segments that encompass crucial feature information. We set the gradient threshold at 100, a value chosen based on multiple tests [37]. If a segment’s gradient value surpasses this threshold, it is considered to contain ample feature information; otherwise, it is deemed to lack critical feature information. This criterion aids us in distinguishing segments with significant features from those without, allowing us to concentrate computational resources on segments that are more likely to contribute to the overall image classification [38]. The entire chunking and gradient computation process above is shown in Fig. 5.

• 4. Chunk Indexing Finally, assign an index to each chunk. Chunks identified as containing feature information are given a positive index, while chunks that do not contain feature information are given a negative index. This indexing scheme helps track split chunks and their related information, providing a systematic method for managing the large amount of data generated during the image splitting process. It also aids subsequent steps in our method, such as encrypting feature-rich split chunks and inputting these split chunks into our improved ciphertext convolutional neural network classification model [39, 40].

Fig. 5

Illustration of the process of filtering chunks based on feature values. The value signifies the content of features, with chunks containing a greater number of features exhibiting larger values. Chunks that surpass a predetermined threshold are deemed to contain feature chunks

Chunks encryption

 

• 1. Encrypting Feature-Rich Image Chunks Once chunks containing important feature information are identified, we begin encrypting these fragments using the fully homomorphic Cheon-Kim-Kim-Song (CKKS) scheme. CKKS is a form of homomorphic encryption that allows complex operations to be performed on encrypted data. This feature is particularly beneficial for computations involving image data, as image data often involves non-linear operations such as convolution. By using the CKKS scheme, CKKS can directly encode, encrypt, and operate on double-precision floating-point real numbers or even complex numbers, thus protecting the privacy of image data while still performing meaningful computations [41].

• 2. Encryption Process Parameter Selection: Choosing appropriate parameters is crucial, including the degree of the polynomial, the modulus, and the error distribution, among others. These parameters impact the security and efficiency of the encryption scheme [29]. When using multiple polynomial moduli, different sizes of images require different sizes of keys, as shown in Table 1.

Table 1 Key size required for two sizes of images using different polynomial modules

Key Generation: Public and Private Key Generation: A random private key is generated, and the public key is computed using the private key. The private key is typically a short vector, while the public key is a matrix formed by combining the private key with some random elements [42].

Encryption: Message Encoding: The message to be encrypted is encoded into a polynomial. This involves mapping the message to a complex domain or integer ring and includes some scaling and adjustment operations to maintain precision. Actual Encryption: The public key and the encoded message, along with some random values, are used to create the encrypted ciphertext. The encryption process in the CKKS scheme involves polynomial multiplication and modulus reduction operations [43], as illustrated in Fig. 6.

Fig. 6

Representation of the application of full homomorphic encryption in image processing. The process initiates with the KeyGen function generating an encryption key (EncKey) and a decryption key (DecKey). The EncKey is utilized to encrypt image chunks with distinctive features, while the DecKey is ultimately employed to decrypt these encrypted chunks

Computation: Computations are performed on the ciphertext. These computations are executed on the encrypted data, without the need for decryption. In the CKKS scheme, complex polynomial computations can be carried out, enabling more complex operations to be performed on encrypted data (Fig. 7).

Fig. 7

Illustration of the process of subjecting the encrypted image to a convolution operation. The model performs an image-to-column convolution, effectively transforming the 2D convolution into a singular matrix multiplication operation

Decryption: The private key is used to decrypt the encrypted ciphertext to recover the original message [44]. In the CKKS scheme, the decryption process involves polynomial multiplication and modulus reduction operations. Decryption is only performed after the necessary computations on the encrypted data have been executed, ensuring that the original image data is revealed only when necessary.

Message Decoding: The decrypted polynomial is decoded back into the original message. This involves inverse mapping operations, as well as some adjustments and scaling to restore the original precision.

The CKKS and encryption scheme we use in our method is complex and involves various mathematical operations. A simplified representation of the encryption process is as follows:

$$\begin{aligned} E(m)=(m+r)\mod q \end{aligned}$$

(8)

where E(m) is the encrypted message, m is the original message, r is a random number, and q is a large prime number.

Chunks merge and storage

 

• 1. Image Chunk Merging After encryption, the feature and non-feature fragments are merged into block groups [45]. The order of these block groups reflects the order of the original images, with each block group corresponding to an original image.

• 2. Block Group Storage Once the block groups are formed, they are stored in arrays or lists. This storage format is chosen because in subsequent stages of our method, it is necessary to conveniently access and manipulate these block groups. Arrays and lists allow efficient access to their elements and allow easy manipulation of data, making them ideal for our purposes. By storing block groups in this way, we can easily retrieve and process encrypted image data in the later stages of our method [46].

Through this process, we prepare the dataset for subsequent steps using homomorphic encryption technology, thereby ensuring data privacy. These steps include modifying the convolutional neural network (CNN) model to handle encrypted data and training this modified model on the encrypted dataset. By performing these operations on encrypted data, we can maintain the privacy of image data while obtaining accurate image classification results. This balance between data privacy and computational efficiency is a key aspect of our method and demonstrates the potential of homomorphic encryption in privacy-protecting image classification tasks.

Encrypted chunks classification model

The final stage of our method involves training an adjusted Convolutional Neural Network (CNN) on encrypted data and evaluating its performance. In this stage, we modify the image chunking CNN model to handle encrypted data and assess the efficiency and effectiveness of our method.

Classification model modification

 

• 1. Homomorphic encryption operations Replace the original layers (convolutional layer, pooling layer, and fully connected layer) of the CNN model with homomorphic encryption operations that can handle encrypted data [47, 48]. The adjusted neural network targets encrypted data, starting with a convolutional layer with 7x7 convolution kernels, followed by a fully connected layer with 256 input neurons and 128 output neurons. These features are combined for encrypted chunk image classification. Finally, an output layer with 6 neurons is implemented, corresponding to the 6 categories in our dataset. The structure of the chunked convolutional neural network is shown in the Fig. 8:

• 2. Activation and loss functions Current homomorphic encryption schemes require careful selection of the activation function and loss function for the neural network. In our method, due to the complexity of the NEU-CLS dataset, we use the square activation function and cross-entropy loss function to be compatible with encryption technology.

  $$\begin{aligned} y=x^2 \end{aligned}$$

  (9)

Fig. 8

Depiction of the model structure executing convolutional operations on image chunks. The illustrated convolutional network comprises a convolutional layer, a 2-layer activation function using a square function to introduce non-linearity, and two fully connected layers. The square function is employed to introduce non-linearity without impacting the receptive field of the convolutional layer

The cross-entropy loss function for a binary classification problem can be defined as:

$$\begin{aligned} L(y,\hat{y})=-\frac{1}{N}\sum _{i=1}^N[y_i\log (\hat{y_i})+(1-y_i)\log (1-\hat{y_i})] \end{aligned}$$

(10)

where y is the true label, $\hat{y}$ is the predicted label, and N is the number of samples.

Training and testing with encrypted chunks

 

• 1. Testing with encrypted data After the model is trained, we test its performance on the encrypted test set. The testing process includes inputting the encrypted image data into the trained model and comparing the model’s prediction results with the true labels of the images. Due to the complexity of homomorphic encryption operations, this testing process takes longer than testing with unencrypted data [17]. However, our scheme reduces the encryption computation time without losing accuracy. The testing process rigorously evaluates the performance of our model, allowing us to assess the degree of generalization of our model to unseen data. This generalization performance is an important aspect of any machine learning model, as it indicates the model’s ability to make accurate predictions on new data, not limited to the specific examples it was trained on [49].

• 2. Recording performance metrics After the testing process, we record key performance metrics such as accuracy and loss value. The accuracy metric provides the proportion of images correctly classified by the model, giving us a direct understanding of the model’s performance. On the other hand, the loss value can measure the confidence of the model in its predictions. The smaller the loss value, the more confident the model is in its predictions, which usually means better performance [50].

These metrics provide valuable information for understanding the performance of the model and the effectiveness of our method. By recording these metrics, we can track the progress of the model over time, identify areas that need improvement, and compare our method with other methods. This recording process is an important part of our methodology, as it allows us to critically evaluate our method and continually strive for improvement [30].

Classification results and performance evaluation

 

• 1. Performance metric comparison We compare the performance metrics-accuracy and loss value-of the original data and encrypted data. This comparison helps to understand the impact of encryption on model performance:

  $$\begin{aligned} \text {Accuracy}=\frac{\text {Number of correct predictions}}{\text {Total number of predictions}}. \end{aligned}$$

  (11)

  By comparing the accuracy and loss values when the model is trained on original data and encrypted data, we can gain a deeper understanding of the trade-offs involved in privacy-preserving image classification using homomorphic encryption technology.

• 2. Encryption impact analysis In addition, we conduct a comprehensive analysis of the impact of encryption operations on model performance and computational complexity. This includes evaluating the impact of homomorphic encryption operations on the model’s testing time and the model’s accuracy. By studying these impacts, we can better understand the computational costs associated with our privacy-preserving method and identify potential areas for optimization.

• 3. Privacy protection evaluation Based on experimental results, we evaluate the effectiveness of our method in protecting data privacy [51]. This evaluation is crucial for confirming whether our method has achieved the dual goals of data privacy protection and efficient image classification. By demonstrating that our method can maintain high classification accuracy while protecting data privacy, we can validate the effectiveness of our method. Through these comprehensive steps, we aim to leverage the capabilities of deep learning and homomorphic encryption to design a robust and privacy-preserving image classification method. Our method demonstrates the potential of combining advanced encryption technology with deep learning models to achieve secure and efficient image classification [52].

Result and discussion

Preliminary work for encrypted chunks classification model

We performed a series of preprocessing steps on images from the NEU-CLS dataset. These steps include resizing the images to a uniform size, converting the images to grayscale to reduce computational complexity, applying histogram equalization to enhance image contrast, and implementing data augmentation techniques to increase the diversity of the dataset [53]. After the preprocessing stage, we constructed a Convolutional Neural Network (CNN) model. This model includes two convolutional layers, two pooling layers, two fully connected layers, and an output layer. We used Rectified Linear Units (ReLU) as the activation function to introduce non-linearity into the model and used the Adam optimizer for efficient parameter updates. This model laid a solid foundation for our subsequent modifications and adaptation to encrypted image chunking computations, adopting a simple neural network to better adapt to our computations on complex datasets.

Modification of encrypted chunks classification model

We adjusted the original CNN model to handle encrypted chunk image data [54]. This adjustment involves modifying the input of the fully connected layer to adapt to the changes brought about by encryption operations. This adjusted model aims to handle chunked and encrypted image data, allowing us to maintain data privacy while performing image classification [55]. The encrypted image chunking convolutional neural network we constructed can be applied to various datasets, and image classification of various encrypted image chunks can be achieved by simply adjusting the input and output of the fully connected layer.

Image splitting and chunks encryption

After adjusting the model, we split each image into subregions of 28x26 pixels. Then, we calculated the local gradients of each subregion using the Sobel operator, a common technique for edge detection in image processing. Based on the calculated gradient values, we divided the regions into areas with or without important features. Then, we used the Complex Key-Value Storage (CKKS) homomorphic encryption scheme to homomorphically encrypt the areas with important features. After encryption, the regions were merged into chunk groups, maintaining their original order to preserve the spatial relationships between different parts of the image. In the analysis process, we found that the time required to encrypt only the chunks containing features using our complete method was the shortest compared to the time required to encrypt the entire image using a generic model, encrypt all chunks using our incomplete method, and encrypt only the chunks containing features using our complete method, as shown in Table 2.

Table 2 Time taken to encrypt a whole image using the generic model, to encrypt all chunks using our incomplete method, and to encrypt only chunks containing features using our full method

Results of classification and model performance

The convolutional layer, pooling layer, and fully connected layer of the original CNN model were replaced with homomorphic encryption operations that can handle encrypted data. This transformation required the use of a square activation function and cross-entropy loss function compatible with the CKKS scheme. Then, we retrained the adjusted network with the encrypted training set. We validated the model on the encrypted validation set, thereby evaluating the model’s performance on unseen data.

During the encryption process, we chunked according to the combination of image chunk groups. The encryption speed of this combination is on average 0.1 seconds faster than the original image without chunking, showing a significant time advantage, especially on large datasets.

The classification results of the original image chunking model and the adapted encrypted model are shown in the attached Fig. 9. The overall classification accuracy after encryption has significantly improved compared to the original image classification accuracy.

Fig. 9

Illustration of the comparative results of the original image chunking model and the adaptive encryption model for classification. The depicted values represent the accuracy of original image classification and encrypted image chunk classification for the six NEU-CLS images, respectively

The pre-classified chunk groups are input into the modified adaptive chunking CNN model for testing, and the classification results are compared with the actual results, as shown in Fig. 10a. The pre-classified encrypted chunk groups are input into the modified adaptive encrypted chunking CNN model for testing, and the classification results are compared with the actual results, as shown in Fig. 10b.

Fig. 10

Comparative representation of evaluation results from the plaintext model and the chunked ciphertext model. The diagonal elements represent the test accuracy values. a Illustrates the outcomes of the classification test conducted on the plaintext dataset, juxtaposed with the true classification results. b Depicts the outcomes of the classification test conducted on the chunked ciphertext dataset, compared with the true classification results

Comparing our method with existing research shows the superiority of the accuracy of our method. The comparison results are shown in Table 3.

Finally, we compared the performance of models trained on original data and encrypted data. The focus of the comparison was on accuracy, loss value, and the impact of encryption operations on model performance and computational complexity. We also evaluated the effectiveness of the encrypted neural network in data privacy protection.

Table 3 The accuracy comparison of the method [17] and the optimized activation function method [18] with our method for analyzing The Cancer Genome Atlas (TCGA) data set is shown

Summary

Our research presents a novel approach to image classification that maintains data privacy by utilizing encrypted image chunks and an adjusted Convolutional Neural Network (CNN). The study is divided into four sections, each detailing a specific stage of our method and its results.

In the initial stage, we preprocessed images from the NEU-CLS dataset, which included resizing, grayscale conversion, histogram equalization, and data augmentation. A simple CNN model was then constructed, laying the groundwork for subsequent modifications to handle encrypted image chunks.

The second stage involved adjusting the original CNN to handle encrypted chunked image data. This adjustment mainly involved modifying the inputs of the fully connected layer to accommodate changes brought about by encryption operations. The adjusted model is designed to handle chunked and encrypted image data, enabling image classification while preserving data privacy.

In the third stage, we segmented each image into 28x26 pixel subregions and calculated local gradients using the Sobel operator. Based on the computed gradient values, we divided the regions into those with or without significant features. Homomorphic encryption was then applied to regions with significant features using the Complex-Valued Cheon-Kim-Kim-Song (CKKS) scheme. The encrypted regions were merged into chunk groups, maintaining their original order to preserve spatial relationships between different parts of the image.

In the final stage, we replaced the convolutional layer, pooling layer, and fully connected layer of the original CNN model with homomorphic encryption operations compatible with the CKKS scheme. The adjusted network was then retrained with the encrypted training set and validated on the encrypted validation set.

Our results showed that the time required for encrypting only the feature-containing chunks using our complete method was the shortest compared to encrypting the entire image using a generic model or encrypting all chunks using our incomplete method. Furthermore, the performance of the models trained on original and encrypted data was compared, focusing on accuracy, loss values, and the impact of encryption operations on model performance and computational complexity. The effectiveness of the encrypted neural network in data privacy protection was also evaluated. Our method demonstrated significant time advantage, especially on large datasets, and maintained high classification accuracy while protecting data privacy.

Conclusion

In this research, we presented a pioneering approach to image classification that seamlessly integrates accuracy and data privacy by utilizing encrypted image chunks and an adjusted Convolutional Neural Network (CNN). Our method demonstrated significant time advantages, especially on large datasets, and maintained high classification accuracy. The emphasis on encrypting only the feature-rich chunks not only ensures data privacy but also offers computational benefits. The flexibility of the CNN model in accommodating encrypted data structures underscores its potential in evolving data privacy landscapes. Future research directions include further optimization of the CNN model, exploration of alternative encryption schemes, and application to diverse datasets. The findings of this study have broad implications for secure and efficient image processing across various domains.