Skip to content

Add PartitionedFileLoader #61

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 32 commits into from
Jun 7, 2018
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
a07091b
Add PartitionedFileLoader
tyclintw May 7, 2018
67a358c
Roll back to the original DataType.
tyclintw May 7, 2018
0bc8a2b
Address comments.
tyclintw May 11, 2018
ce3edce
Add exception handling for failed loader.
tyclintw May 11, 2018
eebf207
Merge branch 'master' into tyclintw/partitionedloader
tyclintw May 11, 2018
bcd4aad
Fix Generator issues. This is change is a hack and will be addressed …
tyclintw May 11, 2018
748ffe7
Address comments.
tyclintw May 11, 2018
4549388
Remove unused namespaces.
tyclintw May 11, 2018
781a45e
Move subLoader to a byteArray so we aren't recreating with args.
tyclintw May 15, 2018
e54698a
Save and load ISchema directly instead of the Column [].
tyclintw May 15, 2018
885ff30
Update help text for clarity.
tyclintw May 15, 2018
bbf8de8
Fix linux test failures.
tyclintw May 15, 2018
9a2d641
Merge branch 'master' into tyclintw/partitionedloader
tyclintw May 16, 2018
225b7ee
Force path output to be unix formatted for consistency between OS tests.
tyclintw May 16, 2018
4497c35
Fix ZBaselines release folder name.
tyclintw May 16, 2018
1e01903
Sort file listings to guarantee "Expand" ordering across operating sy…
tyclintw May 16, 2018
10f47b4
Whitespace.
tyclintw May 16, 2018
90bedc4
Move test files from Samples to test/data
tyclintw May 22, 2018
c0467e6
Address comments.
tyclintw May 22, 2018
674b5cb
Modify exception handling to use Contracts instead.
tyclintw May 23, 2018
5265c90
Add UnescapeDataString to realtive path method.
tyclintw May 23, 2018
a040b51
Rename PathUtils to prevent name conflicts.
tyclintw May 23, 2018
fe6ca03
Address comments.
tyclintw May 23, 2018
097086f
Fix ExceptParam call.
tyclintw May 24, 2018
fe3229f
Merge branch 'master' into tyclintw/partitionedloader
tyclintw May 30, 2018
d3997cb
Merge branch 'master' into tyclintw/partitionedloader
tyclintw May 30, 2018
3c5d6d8
Merge branch 'tyclintw/partitionedloader' of https://github.com/tycli…
tyclintw May 30, 2018
5edd446
Merge branch 'master' into tyclintw/partitionedloader
tyclintw May 31, 2018
c1a5897
Move ZBaselines to new test\BaselineOutput location.
tyclintw May 31, 2018
7d09e32
Merge branch 'tyclintw/partitionedloader' of https://github.com/tycli…
tyclintw May 31, 2018
d9905bb
address comments
tyclintw Jun 1, 2018
ec92ecd
Modify all Exceptions to use Contracts.Exception.
tyclintw Jun 5, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Address comments.
  • Loading branch information
tyclintw committed May 11, 2018
commit 748ffe7d53fe94aba53071b0e7bed5f03922c4fb
6 changes: 4 additions & 2 deletions src/Microsoft.ML.Core/Utilities/PathUtils.cs
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@ public static string MakePathRelative(string basepath, string path)

if (baseUri.Scheme != uri.Scheme)
{
throw new ArgumentException("Paths cannot be made relative as they are of different schemas.");
throw new ArgumentException("Paths cannot be made relative as they are of different schemes.");
}

string relativePath;
Expand Down Expand Up @@ -196,8 +196,10 @@ public static string MakePathRelative(string basepath, string path)
/// <returns>An enumerable list of all non-empty directories.</returns>
public static IEnumerable<string> SplitDirectories(string path)
{
char [] separators = { Path.DirectorySeparatorChar };

var cleanPath = path.Replace(Path.AltDirectorySeparatorChar, Path.DirectorySeparatorChar);
return cleanPath.Split(new char[] { Path.DirectorySeparatorChar }, StringSplitOptions.RemoveEmptyEntries);
return cleanPath.Split(separators, StringSplitOptions.RemoveEmptyEntries);
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -175,7 +175,7 @@ public PartitionedFileLoader(IHostEnvironment env, Arguments args, IMultiStreamS
_host.CheckValue(files, nameof(files));

_pathParser = args.PathParserFactory.CreateComponent(_host);
_host.CheckValue(_pathParser, nameof(_pathParser), "Factory failed to create a FilePathSpec");
_host.CheckValue(_pathParser, nameof(_pathParser), "Failed to create the FilePathSpec.");

_subLoader = args.Loader;
_files = files;
Expand Down