Only getting last record saved...Why

[Milfredo]

The following code works. It writes to disk all 17 files I an trying to write to file. Problem is, only the last record in each file is being saved. I could use some help.

# DEFINE LOAD RACE FILE FUNCTION
def load_racefiles() :
	global Race_date
	global master_race_file
	global track_name
	global order
	global h
	global line
	field = ['track','race_number','todays_surface']
	order= []
	track_name = " "
	master_race_file = []
	h = []
	count = 0
	df_filespath = ('C://guis/f_racefiles/')
	rows = 100
	columns = 100
	horse_array = [0 for i in range(rows)] 
	filename = StringVar()
	f_name = StringVar()
	line = ()
	
	fd =    open("dirpath", "r") 
	file_dir = (fd.read())
	fd.close()
	os.chdir(file_dir)
	print(file_dir)
	for f in os.listdir(file_dir): 
		file_name, file_ext = os.path.splitext(f)
		f_name = file_name + file_ext
		file_name = file_name.strip()[3:]
		line_counter = 0
		if file_name == Race_date and file_ext == ".jcp":
			with open(f_name, "r") as csv_file:
				csv_reader = csv.reader(csv_file)
				
				for line in csv_reader:
					
											

				
					
					h = {'track' : line[0],
					'race_number' : line[2], 
					'todays_surface' : line[6],
					
					
					

 				
 						}

				track_name = line[0]
 					#os.chdir(df_filespath)	
				filename = df_filespath + track_name + '.csv'
				with open(filename, 'a') as csv_file:
					d_writer = csv.DictWriter(csv_file, fieldnames = field)
					d_writer.writerow(h)
				
					
		
			
			
				
	my_labeldone= Label( text ="Downloading Complete", fg= "black", font = ("sans_serif" , 16)).place(x=500, y=500)
					

Thank You

[bowlofred]

At line43, you create a datastructure for the currently read line. Then on line 56 you write that data out. But you do so by opening (and overwriting) any data that exists. So only the last line is visible. You don't want to open the output file within the per-line loop.

[Milfredo]

Thank you. I will try and fix that.

[buran]

Then on line 56 you write that data out. But you do so by opening (and overwriting) any data that exists.

That is not the cause of their problem - they open the file in append mode.
The problem is in the loop on lines 37-51. You virtually loop over the file and read each line, every time replacing h before you have written it to a file. So, only the last row of the source file is written. You either need to move the part where you write (lines 53-58) inside loop or create a list of dicts and then write all of them in one go using csv.DictWriter.writerows() method

[Milfredo]

I have tried to fix the problem. So looked up the os commands and saw the following used. Problem is I can't get the indentation right no matter where I put it...that line is track_name = line[0] here is the new code.

for f in os.listdir(file_dir): 
        file_name, file_ext = os.path.splitext(f)
        f_name = file_name + file_ext
        file_name = file_name.strip()[3:]
        line_counter = 0
        if file_name == Race_date and file_ext == ".jcp":
            with open(f_name, "r") as csv_file:
                csv_reader = csv.reader(csv_file)
                 
                for line in csv_reader:
                     
                                             
 
                 
                     
                    h = {'track' : line[0],
                    'race_number' : line[2], 
                    'todays_surface' : line[6],
                     
                     
                     
 
                 
                        }
 
           				track_name = line[0]
        path_dir:  str= r"C:\guis\daily_racefiles"      
        filename : str =   (track_name + ".csv")
        path_file = os.sep.join([path_dir, filename])
        with open(path_file, 'a') as csv_file:
            d_writer = csv.DictWriter(csv_file, fieldnames = field)
            d_writer.writerow(h)
                 
    my_labeldone= Label( text ="Downloading Complete", fg= "black", font = ("sans_serif" , 16)).place(x=500, y=500    

I am very frustrated. I have been working on this for 6 hours or so.

[buran]

You've been advised to move lines 53-58 (now lines 26-32) inside the loop. you moved just line 26 and actually moved rest of them even more out.

[Milfredo]

I just moved the code lines you specified into the loop. Here is the error it throws when I run the code.


Milford@LAPTOP-3NUEM60C MINGW64 /c/guis
$ python hello2.py
File "hello2.py", line 1225
track_name = line[0]
^
TabError: inconsistent use of tabs and spaces in indentation

[buran]

the error is pretty clear - you mix tab and spaces for indentation. don't do that. Usually IDE has an option to convert tab to certain number of spaces

[metulburr]

check out the section here called tab and space mixing towards the bottom of the first post.

[Milfredo]

Thanks very much you guys. This 70 year old brain really appreciates your time and help.

Milfredo