Hi, I'm importing about a million records and I've noticed it takes a long time just to build up the SQL insert statement.

I'm in a somewhat special scenario where a lot of column values are repeated and found out a slight improvement for the ActiveRecord::Extensions::ConnectionAdapters::MysqlAdapter#values_sql_for_column_names_and_attributes method.

Basically the quoting phase eats up a lot of time and we could skip it if we encounter the same column value twice. I'm now temporarily caching the quoted column values in a Hash and try to reuse them if possible.

your version

  def values_sql_for_column_names_and_attributes( columns, array_of_attributes )
    values = []
    array_of_attributes.each do |arr|
      my_values = []
      arr.each_with_index do |val,j|
        my_values << quote( val, columns[j] )
      end
      values << my_values
     end
     values.map{ |arr| '(' + arr.join( ',' ) + ')' }
    end
  end

with quoted string caching:

  def values_sql_for_column_names_and_attributes( columns, array_of_attributes )
    quoted_vals_cache = {}
    values = []
    array_of_attributes.each do |arr|
      my_values = []
      arr.each_with_index do |val,j|
        quoted_vals_cache[[val,j]] ||= quote( val, columns[j] )
        my_values << quoted_vals_cache[[val,j]]
      end
      values << my_values
    end
    values.map{ |arr| '(' + arr.join( ',' ) + ')' }
  end

The hash cache shouldn't affect memory consumption as it's just keeping references for existing objects.

Here a simple benchmark I've made, loading 500000 records:

         user         system   total       real
  normal 453.980000   2.760000 456.740000 (461.029449)
  cached 237.910000   1.200000 239.110000 (242.958087)

I know my case it's probably an edge one as I have many records sharing the same contents, but I think it could help anyways.

Let me know if you need more details about this.