Myujth

I have a glue schema that works for most CSVs. but sometimes clients upload a CSV with CP1252 encoded fields. When I try to do anything with a dynamicFrame from such a file, I get an error:
datasource0.toDF().show(1)

An error occurred while calling o221.toDF.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 5.0 failed 4 times, most recent failure: Lost task 1.3 in stage 5.0 (TID 17, ip-172-31-11-73.ec2.internal, executor 2): com.amazonaws.services.glue.util.FatalException: Unable to parse file: xxxx11142018.csv

I tried decoding/encoding each line. but it doesn't work:

def sanitizeString(rawValue):
 value = re.sub('\s+', ' ', rawValue)
 value = re.sub('\\', '', value)
 value = value.decode('CP1252').encode('utf-8')
 return value

def sanitizeLine(rawLine):
 line = rawLine.map(sanitizeString)

datasourceRaw = glueContext.create_dynamic_frame.from_catalog(database = "glue_db", table_name = "glue_table", transformation_ctx = "datasourceRaw")
datasource0 = datasourceRaw.map(sanitizeLine)
# datasource3.toDF().show(1)

It gives the exact same error

I've also tried loading it through RDD, using

rdd0 = sc.textFile(filepath, use_unicode=False)
rdd0 = rdd0.map(sanitizeLine) # have to add an extra line to sanitizeLine() to split() the line
rdd0.toDF().show(2)

this works, but there's 2 problems:

1. For whatever reason, Its waaayy slower than using the DynamicFrame


2. It gives me the whole line as a string to map through. I suppose I can split on , but then I want to turn it back into a DynamicFrame and for that I need a schema. I tried getting that from the catalog, but (you guessed it), I get the same error:

-

datasourceRaw = glueContext.create_dynamic_frame.from_catalog(database = "glue_db", table_name = "glue_table", transformation_ctx = "datasourceRaw")
print datasourceRaw.schema()

An error occurred while calling o434.schema.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 9.0 failed 4 times, most recent failure: Lost task 1.3 in stage 9.0 (TID 47, ip-172-31-11-73.ec2.internal, executor 5): com.amazonaws.services.glue.util.FatalException: Unable to parse file: xxxx_11142018.csv