NOTICE: please fist run basic SWA and SGD, and save relevant checkpoints, since most of the following code need a model to load from. 


utils.py :			running related code

train_swa_solid.py: 		for basic training for ResNet on CIFAR-10

demo_h.py : 		for basic training a DenseNet-100

explore_new_sgd.py : 	for exploring on a specific direction. it is used in 
			exploring training loss surface on a specific direction of a SGD solution 
			verifying assumption 1
			only used in ResNet!

explore_new_dense.py :	for exploring on a specific direction. it is used in 
			only used in DenseNet!

explore_new_conv.py :	for exploring on a specific Non-BN direction. it is used in:
			BN section
			only used in ResNet!

explore_new_onlybn.py :	for exploring on a specific BN direction. it is used in:
			BN section
			only used in ResNet!

explore_new_dense_bn.py :	for exploring on a specific Non-BNdirection. it is used in:
			BN section
			only used in DenseNet

explore_new_dense_conv.py :	for exploring on a specific BN direction. it is used in:
			BN section
			only used in DenseNet

train_average_compare_randconv.py: 
			for comparing averaging performance of BN layer and CONV layer.

rand_shift_calculate.py : 	for calculating the shift between training loss and test loss
			used in verifying assumption 1
			need train loss and test loss data

ratio_calculate.py: 		for calculating different shift and correspoding \xi
			need train loss and test loss data

explore_new_sgd_simple.py : 	for verifying assumption 2		

trajectory_slice.py : 		for exploring loss surface neighbor along the SGD trajectory:
			checkpoints are needed
			only used in ResNet!

interpulation.py :		for calculating interpolation between SGD and SWA solutions
			used in ResNet
			be careful when choosing the loading model KEY!

sgd_after_swa.py : 		for running SGD after a given SWA checkpoint
			used in ResNet
			notice the learning rate should be smaller than SWA learning rate

sgd_after_swa_dense.py:	for running SGD after a given SWA checkpoint
			used in DenseNet
			notice the learning rate should be smaller than SWA learning rate

interpulation_dense.py :	for calculating interpolation between SGD and SWA solutions
			used in DenseNet


randomray.py : 		for exploring a random direction
			used in ResNet

high_dimension_find.py :	for verifying Theorem 2
			need the SGD solution after SWA solution as mentioned in paper
			used in ResNet

train_47.py :		for calculating ResNet-8.

logist_reg.py :		for finding asymmetric direction in a logistic regression task. 





	

