Convert diff module to gem

Author: Adam Gardiner

.gitignore

@@ -0,0 +1,4 @@
+.DS_Store
+doc
+pkg
+.yardoc

LICENSE

@@ -0,0 +1,22 @@
+Copyright (c) 2013, Adam Gardiner
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this
+  list of conditions and the following disclaimer.
+* Redistributions in binary form must reproduce the above copyright notice,
+  this list of conditions and the following disclaimer in the documentation
+  and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

README.md

@@ -0,0 +1,76 @@
+# ColorConsole
+
+ColorConsole is a small cross-platform library for outputting text to the console.
+
+
+## Usage
+
+ColorConsole is supplied as a gem, and has no dependencies. To use it, simply:
+```
+gem install color-console
+```
+
+ColorConsole provides methods for outputting lines of text in different colors, using the `Console.write` and `Console.puts` functions.
+
+```ruby
+require 'color-console'
+
+Console.puts "Some text"                    # Outputs text using the current console colours
+Console.puts "Some other text", :red        # Outputs red text with the current background
+Console.puts "Yet more text", nil, :blue    # Outputs text using the current foreground and a blue background
+
+# The following lines output BlueRedGreen on a single line, each word in the appropriate color
+Console.write "Blue ", :blue
+Console.write "Red ", :red
+Console.write "Green", :green
+```
+
+## Features
+
+In addition to `Console.puts` and `Console.write` for outputting text in color, ColorConsole also supports:
+* __Setting the console title__: The title bar of the console window can be set using `Console.title = 'My title'`.
+* __Status messages__: Status messages (i.e. a line of text at the current scroll position) can be output and
+  updated at any time. The status message will remain at the current scroll point even as new text is output
+  using `Console.puts`.
+* __Progress bars__: A progress bar can be rendered like a status message, but with a pseudo-graphical representation
+  of the current completion percentage:
+
+    ```ruby
+    (0..100).do |i|
+        Console.show_progress('Processing data', i)
+    end
+    Console.clear_progress
+    ```
+    Output:
+    ```
+    [==============    35%                   ]  Processing data
+    ```
+* __Tables__: Data can be output in a tabular representation:
+
+    ```ruby
+    HEADER_ROW = ['Column 1', 'Column 2', 'Column 3', 'Column 4']
+    MIXED_ROW = [17,
+                 'A somewhat longer column',
+                 'A very very very long column that should wrap multple lines',
+                 'Another medium length column']
+    SECOND_ROW = [24,
+                  'Lorem ipsum',
+                  'Some more text',
+                  'Lorem ipsum dolor sit amet']
+
+    Console.display_table([HEADER_ROW, MIXED_ROW, SECOND_ROW], width: 100,
+                          col_sep: '|', row_sep: '-')
+    ```
+    Output:
+    ```
+    +----------+--------------------------+-----------------------------+-----------------------------+
+    | Column 1 | Column 2                 | Column 3                    | Column 4                    |
+    +----------+--------------------------+-----------------------------+-----------------------------+
+    |       17 | A somewhat longer column | A very very very long       | Another medium length       |
+    |          |                          | column that should wrap     | column                      |
+    |          |                          | multple lines               |                             |
+    +----------+--------------------------+-----------------------------+-----------------------------+
+    |       24 | Lorem ipsum              | Some more text              | Lorem ipsum dolor sit amet  |
+    +----------+--------------------------+-----------------------------+-----------------------------+
+    ```
+

Rakefile

@@ -0,0 +1,10 @@
+require 'rubygems'
+require 'rubygems/package_task'
+
+load 'color-console.gemspec'
+
+Gem::PackageTask.new(GEMSPEC) do |pkg|
+    pkg.need_tar = false
+end
+
+task :default => :gem

csv-diff.gemspec

@@ -0,0 +1,29 @@
+GEMSPEC = Gem::Specification.new do |s|
+    s.name = "csv-diff"
+    s.version = "0.1"
+    s.authors = ["Adam Gardiner"]
+    s.date = "2014-05-30"
+    s.summary = "CSV Diff is a library for generating diffs from data in CSV format"
+    s.description = <<-EOQ
+        This library performs diffs of CSV files.
+
+        Unlike a standard diff that compares line by line, and is sensitive to the
+        ordering of records, CSV-Diff identifies common lines by key field(s), and
+        then compares the contents of the fields in each line.
+
+        Data may be supplied in the form of CSV files, or as an array of arrays. The
+        diff process provides a fine level of control over what to diff, and can
+        optionally ignore certain types of changes (e.g. changes in position).
+
+        CSV-Diff is particularly well suited to data in parent-child format. Parent-
+        child data does not lend itself well to standard text diffs, as small changes
+        in the organisation of the tree at an upper level can lead to big movements
+        in the position of descendant records. By instead matching records by key,
+        CSV-Diff avoids this issue, while still being able to detect changes in
+        sibling order.
+    EOQ
+    s.email = "[email protected]"
+    s.homepage = 'https://github.com/agardiner/csv-diff'
+    s.require_paths = ['lib']
+    s.files = ['README.md', 'LICENSE'] + Dir['lib/**/*.rb']
+end

lib/csv-diff.rb

@@ -0,0 +1,4 @@
+require 'csv-diff/csv_source'
+require 'csv-diff/algorithm'
+require 'csv-diff/csv_diff'
+

lib/csv-diff/algorithm.rb

@@ -0,0 +1,122 @@
+class CSVDiff
+
+    # Implements the CSV diff algorithm.
+    module Algorithm
+
+        # Diffs two CSVSource structures.
+        #
+        # @param left [CSVSource] A CSVSource object containing the contents of
+        #   the left/from input.
+        # @param right [CSVSource] A CSVSource object containing the contents of
+        #   the right/to input.
+        # @param key_fields [Array] An array containing the names of the field(s)
+        #   that uniquely identify each row.
+        # @param diff_fields [Array] An array containing the names of the fields
+        #   to be diff-ed.
+        def diff_sources(left, right, key_fields, diff_fields, options = {})
+            left_index = left.index
+            left_values = left.lines
+            left_keys = left_values.keys
+            right_index = right.index
+            right_values = right.lines
+            right_keys = right_values.keys
+            parent_fields = left.parent_fields.length
+
+            include_moves = options.fetch(:include_moves, true)
+            include_deletes = options.fetch(:include_deletes, true)
+
+            diffs = Hash.new{ |h, k| h[k] = {} }
+            right_keys.each_with_index do |key, right_row_id|
+                key_vals = key.split('~')
+                parent = key_vals[0...parent_fields].join('~')
+                child = key_vals[parent_fields..-1].join('~')
+                left_parent = left_index[parent]
+                right_parent = right_index[parent]
+                left_value = left_values[key]
+                right_value = right_values[key]
+                left_idx = left_parent && left_parent.index(key)
+                right_idx = right_parent && right_parent.index(key)
+
+                id = {}
+                id[:row] = right_row_id + 1
+                id[:sibling_position] = right_idx + 1
+                key_fields.each do |field_name|
+                    id[field_name] = right_value[field_name]
+                end
+                if left_idx && right_idx
+                    if include_moves
+                        left_common = left_parent & right_parent
+                        right_common = right_parent & left_parent
+                        left_pos = left_common.index(key)
+                        right_pos = right_common.index(key)
+                        if left_pos != right_pos
+                            # Move
+                            diffs[key].merge!(id.merge(:action => 'Move',
+                                              :sibling_position => [left_idx + 1, right_idx + 1]))
+                            #puts "Move #{left_idx} -> #{right_idx}: #{key}"
+                        end
+                    end
+                    if changes = diff_row(left_values[key], right_values[key], diff_fields)
+                        diffs[key].merge!(id.merge(changes.merge(:action => 'Update')))
+                        #puts "Change: #{key}"
+                    end
+                elsif right_idx
+                    # Add
+                    diffs[key].merge!(id.merge(right_values[key].merge(:action => 'Add')))
+                    #puts "Add: #{key}"
+                end
+            end
+
+            # Now identify deletions
+            if include_deletes
+                (left_keys - right_keys).each do |key|
+                    # Delete
+                    key_vals = key.split('~')
+                    child = key_vals.pop
+                    parent = key_vals.join('~')
+                    left_parent = left_index[parent]
+                    left_value = left_values[key]
+                    left_idx = left_parent.index(key)
+                    next unless left_idx
+                    id = {}
+                    id[:row] = left_keys.index(key) + 1
+                    id[:sibling_position] = left_idx + 1
+                    key_fields.each do |field_name|
+                        id[field_name] = left_value[field_name]
+                    end
+                    diffs[key].merge!(id.merge(:action => 'Delete'))
+                    #puts "Delete: #{key}"
+                end
+            end
+            diffs
+        end
+
+
+        # Identifies the fields that are different between two versions of the
+        # same row.
+        #
+        # @param left_row [Hash] The version of the CSV row from the left/from
+        #   file.
+        # @param right_row [Hash] The version of the CSV row from the right/to
+        #   file.
+        # @return [Hash<String, Array>] A Hash whose keys are the fields that
+        #   contain differences, and whose values are a two-element array of
+        #   [left/from, right/to] values.
+        def diff_row(left_row, right_row, fields)
+            diffs = {}
+            fields.each do |attr|
+                right_val = right_row[attr]
+                right_val = nil if right_val == ""
+                left_val = left_row[attr]
+                left_val = nil if left_val == ""
+                if left_val != right_val
+                    diffs[attr] = [left_val, right_val]
+                    #puts "#{attr}: #{left_val} -> #{right_val}"
+                end
+            end
+            diffs if diffs.size > 0
+        end
+
+    end
+
+end

lib/csv-diff/csv_diff.rb

@@ -0,0 +1,112 @@
+# This library performs diffs of flat file content that contains structured data
+# in fields, with rows provided in a parent-child format.
+#
+# Parent-child data does not lend itself well to standard text diffs, as small
+# changes in the organisation of the tree at an upper level (e.g. re-ordering of
+# two ancestor nodes) can lead to big movements in the position of descendant
+# records - particularly when the parent-child data is generated by a hierarchy
+# traversal.
+#
+# Additionally, simple line-based diffs can identify that a line has changed,
+# but not which field(s) in the line have changed.
+#
+# Data may be supplied in the form of CSV files, or as an array of arrays. The
+# diff process process provides a fine level of control over what to diff, and
+# can optionally ignore certain types of changes (e.g. changes in order).
+class CSVDiff
+
+    # @return [CSVSource] CSVSource object containing details of the left/from
+    #    input.
+    attr_reader :left
+    alias_method :from, :left
+    # @return [CSVSource] CSVSource object containing details of the right/to
+    #    input.
+    attr_reader :right
+    alias_method :to, :right
+    # @return [Array<Hash>] An array of differences
+    attr_reader :diffs
+
+
+    # Generates a diff between two hierarchical tree structures, provided
+    # as +left+ and +right+, each of which consists of an array of lines in CSV
+    # format.
+    # An array of field indexes can also be specified as +key_fields+;
+    # a minimum of one field index must be specified; the last index is the
+    # child id, and the remaining fields (if any) are the parent field(s) that
+    # uniquely qualify the child instance.
+    #
+    # @param left [Array<Array<String>>] An Array of lines, each of which is in
+    #   turn an Array containing fields.
+    # @param right [Array<Array<String>>] An Array of lines, each of which is in
+    #   turn an Array containing fields.
+    # @param options [Hash] A hash containing options.
+    # @option options [Array<String>] :field_names An Array of field names for
+    #   each field in +left+ and +right+. If not provided, the first row is
+    #   assumed to contain field names.
+    # @option options [Boolean] :ignore_header If true, the first line of each
+    #   file is ignored. This option can only be true if :field_names is
+    #   specified.
+    # @options options [Array] :ignore_fields The names of any fields to be
+    #   ignored when performing the diff.
+    def initialize(left, right, options = {})
+        @left = CSVSource.new(left, options)
+        raise "No field names found in left (from) source" unless @left.field_names && @left.field_names.size > 0
+        @right = CSVSource.new(right, options)
+        raise "No field names found in right (to) source" unless @right.field_names && @right.field_names.size > 0
+        @warnings = []
+        @diff_fields = get_diff_fields(@left.field_names, @right.field_names, options.fetch(:ignore_fields, []))
+        @key_fields = @left.key_fields.map{ |kf| @diff_fields[kf] }
+        diff(options)
+    end
+
+
+    # Performs a diff with the specified +options+.
+    def diff(options = {})
+        @diffs = diff_sources(@left, @right, @key_fields, @diff_fields, options)
+    end
+
+
+    # Returns a summary of the number of adds, deletes, moves, and updates.
+    def summary
+        summ = Hash.new{ |h, k| h[k] = 0 }
+        @diffs.each{ |k, v| summ[v[:action]] += 1 }
+        summ
+    end
+
+
+    [:adds, :deletes, :updates, :moves].each do |mthd|
+        define_method mthd do
+            action = mthd.to_s.chomp('s')
+            @diffs.select{ |k, v| v[:action].downcase == action }
+        end
+    end
+
+
+    # @return [Array<String>] an array of warning messages generated during the
+    #    diff process.
+    def warnings
+        @left.warnings + @right.warnings + @warnings
+    end
+
+
+    private
+
+
+    # Given two sets of field names, determines the common set of fields present
+    # in both, on which members can be diffed.
+    def get_diff_fields(left_fields, right_fields, ignore_fields)
+        diff_fields = []
+        right_fields.each do |fld|
+            if left_fields.include?(fld)
+                diff_fields << fld unless ignore_fields.include?(fld)
+            else
+                @warnings << "Field '#{fld}' is missing from the left (from) file, and won't be diffed"
+            end
+        end
+        diff_fields
+    end
+
+
+    include Algorithm
+
+end