Python Forum
visualizing huge correation matrix - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: Data Science (https://python-forum.io/forum-44.html)
+--- Thread: visualizing huge correation matrix (/thread-35250.html)



visualizing huge correation matrix - erdemath - Oct-12-2021

I have a huge correlation matrix with the dimension of (654345,2,2). It was generated by Pearson's coefficients, i.e. numpy.corrcoef. What is the best way
of visualizing it? Below is a small part of the entire code;

    def __init__(self):
        self.LayerIdx = ['CA3', 'DG', 'EC'] # propagation order

    def get_correlation(self, Layers, LayersPos):

        corr_xy = []
        for layers_count in range(self.LayerNo - 1):
            x1 = Layers[layers_count]
            x2 = Layers[layers_count + 1]
            y1 = LayersPos[layers_count]
            y2 = LayersPos[layers_count + 1]
            for ind1 in range(len(y1)):
                for ind2 in range(len(y2)):
                    corr_xy.append(np.corrcoef(x1[y1[ind1]], x2[y2[ind2]]))

            x_labels = [v for v in self.LayerIdx]
            y_labels = [v for v in self.LayerIdx]
            x_to_num = {p[1]: p[0] for p in enumerate(x_labels)}
            y_to_num = {p[1]: p[0] for p in enumerate(y_labels)}
            fig, ax = plt.subplots(figsize=(10, 6))
            fig.subplots_adjust(left=0.0625, right=0.95, wspace=0.1)
            for i in range(np.shape(corr_xy)[0]): #corr_xy is the collection of Pearsons coefficients
                sns.heatmap(corr_xy[i], annot=True, fmt='.2f')
                ax.grid(False)
                ax.set_xticks([x_to_num[v] for v in x_labels])  # [x_to_num[v] for v in x_labels]
                ax.set_yticks([y_to_num[v] for v in y_labels])
                cbar = ax.figure.colorbar(im, ax=ax, format='% .2f')
            plt.show()



RE: visualizing huge correation matrix - Larz60+ - Oct-12-2021

There are quite a few blogs on this, Google: plot numpy.corrcoef (Pearson's coefficients) correlation matrix python for list


RE: visualizing huge correation matrix - erdemath - Oct-12-2021

I have been googling for the last 9.5 hours. I think the problem is about how I generate the matrix. To be pragmatic, I do list by using ".append".

(Oct-12-2021, 04:22 PM)Larz60+ Wrote: There are quite a few blogs on this, Google: plot numpy.corrcoef (Pearson's coefficients) correlation matrix python for list



RE: visualizing huge correation matrix - erdemath - Oct-13-2021

Besides, many of those examples are for extremely cases.