Dynamically plotting graphs with matplotlib

[CAD79]

I'm trying to create an application that formats data from a data logger that I have coded using an Arduino. I have the code finished and the output data is, for example:

Time,Sw,28C5
53782,1,27.89
53792,1,26.34
53802,1,27.89
53812,1,27.89
53822,1,25.95
53832,0,27.89
53842,0,26.42
53852,0,24.49
53862,1,27.89
53872,1,27.89
53882,0,27.89
53892,1,30.07
53902,1,27.89
53912,1,24.67
53922,1,33.93
53932,1,31.45
53942,1,27.89
53952,0,24.68
53962,1,22.11
53972,1,21.79

I use pandas to put it into an excel sheet, but to plot the data as a graph is the part of which I am not sure of. As I want to use this application for any data logger I need to create (there will be more devices I'm making, which will all have different output data), I do not know the correct syntax to dynamically plot the data. As time will always be the X-axis, that part of the code is okay, but- for example- for another datalogger, I may have to plot more than 1 sensor (28C5, in this circumstance, is a DS18B20 temperature sensor). Could anyone help me with how to plot a graph for any number of columns of data? I have already used something like that in my code, but I'm not sure how to implement it into matplotlib. Here is my code (unfinished, just trying to make it functional for the moment):

import os
import tkinter as tk
from tkinter import ttk
from tkinter import messagebox
import pandas as pd
import matplotlib.pyplot as plt


class DataFormatter():


    def __init__(self):


        read_path=os.listdir('E:/DataLogging/')

        self.root=tk.Tk()
        self.root.geometry('350x150')
        self.root.title('Data Formatter')
        self.root.iconbitmap('Y:/ENGINEERING/DATALOGGING/Files/datalog.ico')

        self.mainframe=tk.Frame(self.root,background='white')
        self.mainframe.pack(fill='both',expand=True)

        self.type_label=ttk.Label(self.mainframe,text='Select data type: ',background='white',pad=10)
        self.type_label.grid(row=0, column=0, padx=70)

        self.type_box=ttk.Combobox(self.mainframe, values=read_path)
        self.type_box.grid(row=1,column=0)

        format_button=ttk.Button(self.mainframe,text='Format',command=self.format_data)
        format_button.grid(row=2,column=2)

        self.graph_check=tk.Checkbutton(self.mainframe, text='Add a graph', background='white')
        self.graph_check.grid(row=2,column=0, pady=10)

        self.root.mainloop()

        return

    def format_data(self):


        data_type=self.type_box.get()

        read_path = f'E:/DataLogging/{data_type}'
        new_file = (f'Y:/ENGINEERING/DATALOGGING/{data_type}')

        if os.path.exists(new_file)==False:

            os.mkdir(new_file)

        else:

            os.chdir(new_file)

        write_text_path = f'{new_file}/Text'
        write_excel_path = f'{new_file}/Excel'
        file_list = os.listdir(f'{read_path}/')

        for file in file_list:

            write_path = f'{write_text_path}/Parsed_{file}'

            if os.path.exists(write_text_path):

                os.chdir(write_text_path)
                os.chdir(write_excel_path)

            else:

                os.mkdir(write_text_path)
                os.mkdir(write_excel_path)

            with open(f'{read_path}/{file}', 'r') as read_file, open(write_path, 'w') as write_file:

                read_file = read_file.readlines()

                for line in read_file:

                    line = line.strip()
                    if line.find("ERROR:"):
                        line.strip()

                    columns = line.split(',')

                    formatted_line = ''.join(f'{column:>15}' for column in columns)

                    write_file.write(f'{formatted_line}\n')

                write_file.close()

            with open(write_path, 'r') as data:

                txt_to_xl = file.replace('.txt', '.xlsx')
                excel_file = f'{new_file}/Excel/Parsed_{txt_to_xl}'
                df = pd.read_csv(data, delim_whitespace=True)
                df = df.set_index("Time")

                df.to_excel(excel_file)

                print(excel_file)
            XL_data = pd.read_excel(excel_file)
            plt.plot(XL_data.index,  #what do I put here for dynamic plotting?)
            plt.show()




        if write_text_path and write_excel_path:
          messagebox.showinfo('Datalog Formatter', 'Formatting successful!')

        else:
            messagebox.showerror('Datalog Formatter', 'Formatting failed! Check all used files and directories.')

if __name__ == '__main__':
    DataFormatter()

[Larz60+]

Is there any reason you want to store that data as excel?
This will be a lot slower than ploting directly from a csv file (your data is already in CSV format)
see: Plotting data from a file

If the data is already loaded, and you don't need to save it, you can plot directly from your data list.

The recommended way of plotting data from a file is therefore to use dedicated functions such as numpy.loadtxt or pandas.read_csv to read the data. These are more powerful and faster. Then plot the obtained data using matplotlib.

[deanhystad]

You already have the data in a pandas dataframe. You can plot a dataframe directly from pandas.

https://pandas.pydata.org/pandas-docs/st....plot.html

I don't understand what you mean by "dynamic". Do you mean the plot data might change over time after the plot is drawn, or do you mean that the signals might be different from plot to plot?

[CAD79]

Is there any reason you want to store that data as excel?
This will be a lot slower than ploting directly from a csv file (your data is already in CSV format)
see: Plotting data from a file

If the data is already loaded, and you don't need to save it, you can plot directly from your data list.

The recommended way of plotting data from a file is therefore to use dedicated functions such as numpy.loadtxt or pandas.read_csv to read the data. These are more powerful and faster. Then plot the obtained data using matplotlib.

I'm just using this for my code now:

            with open(write_path, 'r') as data:

                txt_to_xl = file.replace('.txt', '.xlsx')
                excel_file = f'{new_file}/Excel/Parsed_{txt_to_xl}'
                df = pd.read_csv(data, delim_whitespace=True)
                df = df.set_index("Time")

                df.to_excel(excel_file)

                print(excel_file)

                plt.plot(df)
                plt.show()

Which was much easier.

I have only recently started coding in python, and for some reason, any tutorial I could find on youtube, they were passing an X and Y value into the plt.plot() function, which made it a little confusing to work with.

Would there be a way I could automatically title the graph too? Edit: never mind, I got it, youtube tutorials aren't good lol, I'll just check the documentation next time.

[CAD79]

You already have the data in a pandas dataframe. You can plot a dataframe directly from pandas.

https://pandas.pydata.org/pandas-docs/st....plot.html

I don't understand what you mean by "dynamic". Do you mean the plot data might change over time after the plot is drawn, or do you mean that the signals might be different from plot to plot?

I just meant the number of columns would change between excel files, as I want to use the application for multiple data loggers, however, I've realised that passing just the dataframe to plt.plot() was all I needed. I was just using youtube videos to learn how to use matplotlib and every video I found passed an X and a Y argument, as opposed to just passing a dataframe, which is what I was confused about.

[considina2]

How can we use Python's pandas library to plot a dataframe directly, without needing to specify X and Y arguments, and handle dynamic changes in the number of columns when working with multiple data loggers?

[CAD79]

How can we use Python's pandas library to plot a dataframe directly, without needing to specify X and Y arguments, and handle dynamic changes in the number of columns when working with multiple data loggers?

yes.. that is what I wanted

[Larz60+]

please provide some sample data (not much) in 'original format', which I believe you said was text.

You can just paste 10 items or so between output tags.
I'm looking for data prior to any conversion.
Need something to play with.

[deanhystad]

please provide some sample data (not much) in 'original format', which I believe you said was text.

See original post which appears to be some kind of timestamp, a switch (0/1) and something call 28C5 that I think is temperature data.

Plotting 0, 1 on the same chart as 27.89 will be problematic. The range of the values is insignificant compared to the difference in ranges. You could plot each column against the time column, each on a separate plot, but you probably want to see similar signals on the same plot.

I don't think generic plotting will work well. The datalogger needs to know something about the data. Maybe you could structure the signal names to indicate the signal time. 0/1 signals are all switches. Temperature signal names all starts with Temp. Then you could use filtering to create a dataframe of switch signals and a dataframe of temperature signals and so on.

[CAD79]

please provide some sample data (not much) in 'original format', which I believe you said was text.

See original post which appears to be some kind of timestamp, a switch (0/1) and something call 28C5 that I think is temperature data.

Plotting 0, 1 on the same chart as 27.89 will be problematic. The range of the values is insignificant compared to the difference in ranges. You could plot each column against the time column, each on a separate plot, but you probably want to see similar signals on the same plot.

I don't think generic plotting will work well. The datalogger needs to know something about the data. Maybe you could structure the signal names to indicate the signal time. 0/1 signals are all switches. Temperature signal names all starts with Temp. Then you could use filtering to create a dataframe of switch signals and a dataframe of temperature signals and so on.

Would it be possible to write the code so that any boolean values (0, 1) are in their own subplot, and any other data is all in another subplot? Edit: I would actually like that or just create a subplot for each signal, which I can imagine would use a for loop to create a subplot for each column. I'll maybe need a little help with the syntax as I haven't been programming for long, but I'll try and figure it out.

I will be using this code for data other than temperature data, such as ozone meter data, which means I cannot use any specific words such as 'temp', which makes things a little more complicated. So, I'd probably just find it best to write the code so that one plot is boolean values, and another plot is any other data.

Also, the timestamp is the time of day in seconds, and 28C5 is a DS18B20 temperature sensors hexadecimal address to the first 4 characters.