Federated Learning is a machine learning technique where independent devices (clients) cooperatively train a machine learning model by working on decentralized training data. A fundamental challenge in federated learning is the client model aggregation. The goal is to combine and preserve the knowledge that each client has acquired during its local training phase and generate an aggregated model with superior performance than any of the clients. Most current state-of-the-art algorithms for model aggregation still rely on simple weight parameter averaging. Though effective, these methods do not consider the internal structure of a deep neural network. In addition, the element-wise aggregation of the model parameters may create a conflict and loss of knowledge if a particular weight parameter serves different classification tasks over other learning clients. To decrease the probability of such conflicts, we propose FedGrid, a method that involves rotated (shifted) weight parameter grids during the clients’ local training phase, which stimulates the models to build non-overlapping neural network structures. FedGrid can be applied to any existing algorithm that performs parameter averaging to improve the aggregation of deep NN models. To estimate the method’s performance, we generate an ensemble of datasets with preset heterogeneity, measured via entropy balance metric, ranging from 0.1 to 1.0. Our results indicate that FedGrid outperforms the state-of-the-art methods when applied to deep neural network models. Moreover, the proposed method is simple and does not add computational complexity to the learning process. We believe that our effective method for grid application provides a new field for future research and improvement of the federation of deep neural network models.