Myujth

I have a csv file test.csv. It have 5000 columns. Some of columns (example 50 columns), have same value in all rows. How can I find how many column have same value and print those columns in separate csv.
Example,

A B C D
1 2 2 3
1 2 3 3
1 2 4 3
1 2 5 3
1 2 7 3

I want to find columns which have similar values/elements such as A,B and D . Then print those A,B and D in separate CSV file and C in separate CSV.

Thank you.

I recommend using pandas. You can solve your problem with something like the below (which should get you started).

You'll need to review this link which will give you an overview of 10 minutes to pandas (i.e. reading in/manipulating data)

import pandas as pd

data = 
 'A': [1] * 5
 , 'B': [1] * 5
 , 'C': [1] * 5
 , 'D': [i for i in range(2, 7)]



df = pd.DataFrame(data)

# loop through each column
for col in df.columns.tolist():
 # check if every value in the column is equal to the first value
 if (df[col] == df[col][0]).all():
 print('all values match in col'.format(col=col))
 else:
 print('col has non-uniform values'.format(col=col))

Just find the columns which have only 1 unique value:

Create a DataFrame, I'm creating with some dummy data, you can read the csv with pd.read_csv.

>>> import pandas as pd
>>> df = pd.DataFrame(data='A': [1,1,1,1,1,1,1], 'B': [2,2,2,2,2,2,2], 'C': [1,2,3,4,5,6,7])
>>> df
 A B C
0 1 2 1
1 1 2 2
2 1 2 3
3 1 2 4
4 1 2 5
5 1 2 6
6 1 2 7

Find those columns which have only 1 unique value:

>>> equal_cols = [c for c in df.columns if len(df[c].unique()) == 1]
>>> equal_cols
['A', 'B']

Write those columns to sample1.csv, and all others to sample2.csv.

>>> df[equal_cols].to_csv('sample1.csv')
>>> df[c for c in df.columns if c not in equal_cols].to_csv('sample2.csv')

You can use pandas for pretty IO.
Just write a function to test a column and select the good ones :

Input :

import pandas as pd
df=pd.read_csv()

A short circuit function,which compare all values only if necessary:

from numba import njit
@njit # optional, for efficiency
def equal(arr):
 ref=arr[0]
 for x in arr[1:]:
 if x != ref : return False
 return True

Output:

mask=df.apply(equal,axis=0,raw=True)
#[ True, True, False, True ]
df.loc[:,mask].to_csv('equal.csv',sep=' ',index=False)
df.loc[:,~mask].to_csv('notequal.csv',sep=' ',index=False)

For:

A B D
1 2 3
1 2 3
1 2 3
1 2 3
1 2 3

and

C
2
3
4
5
7