CSV Parser

Most of our data files are in CSV format. Although the String.split('\t') approach can handle a lot cases, there are CSV files which has quotes. In that case if a delimiter character is in between of quotes, split method will fail. In other words, we do need a good CSV parser which can work well with Spark.

Since Scala so far works fine with me, the first try is to use some RegexParser based approach within Scala. There is a small project called scala-csv which is doing exactly this. However, with some simple performance test, this parser is just too SLOW. The problem is actually from some performance issues with RegexParser on the new JVM. Then I found this java library opencsv. Some benchmarks showed really promising performance results. Here is what the code for using opencsv with Spark

import au.com.bytecode.opencsv.CSVParser
    def mLine(line:String)={
      val parser=new CSVParser('\t')
    val hist = myRDD.map(mLine(_).size).mapPartitions(histFeed(_)).reduce(_::_)
    hist.toSeq.sortBy(_._1).foreach{case (k,v)=>println(k+" "+v)}

Here I used the histogram helper to calculate the histogram on the field counts of an RDD.

The performance of opencsv parser is just AMAZING! On my single machine case, it’s sometimes even faster than the split method. Most of the times, it is almost as fast as split.

This entry was posted in Scala, Spark. Bookmark the permalink.

4 Responses to CSV Parser

  1. Pingback: Optimize map performamce with mapPartitions | Big Data Analytics with Spark

  2. Pingback: Calculate Running Sums | Big Data Analytics with Spark

  3. Pingback: Spark: Parse CSV file and group by column value at Mark Needham

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s