Myujth

import findspark
findspark.init('C:spark')
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

a = 
i=1880
while i<2018:
 a.append(str(i)+'/'+str(i)+'verr.csv')
 i = i+1

dataset1 = spark.read.format('csv').option('header','true').load('C://venq/uyh/'+ a)

i run code and i get the error;
dataset1 = spark.read.format('csv').option('header','true').load('C://venq/uyh/'+ a)
TypeError: can only concatenate str (not "list") to str

i have a "C:venquyh18801880verr.csv" format a loop.
I have csv files in nested folders. I want to read them all with spark. however I get the following error. How can I solve this? thanks

variable 'a' is a list of files.

dataset1 = spark.read.format('csv').option('header','true').load('C://venq/uyh/'+ a)

Here you are trying to concatenate a string 'C://venq/uyh/' with 'a' which is a list - which throws the error. Try

root = r"C://venq/uyh/"

while i<2018:
 a.append(root + str(i)+'/'+ str(i)+'verr.csv')
 i = i+1

and then use a directly

dataset1 = spark.read.format('csv').option('header','true').load(a)