Skip to content

[Documentation] Deploy Worker and Udf Binaries #329

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 20 commits into from
Nov 19, 2019
Merged

[Documentation] Deploy Worker and Udf Binaries #329

Changes from 1 commit
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
e6ca264
Merge pull request #1 from dotnet/master
elvaliuliuliu Sep 6, 2019
1eea8f6
doc for deploy worker and udf
elvaliuliuliu Nov 8, 2019
e98d6f5
Merge branch 'master' into elva/deployworker-udf
elvaliuliuliu Nov 8, 2019
c0b0e4e
Change workder link
elvaliuliuliu Nov 8, 2019
1213fac
Merge branch 'master' into elva/deployworker-udf
rapoth Nov 8, 2019
ecb66b9
resolve comments
elvaliuliuliu Nov 8, 2019
d13c575
typo
elvaliuliuliu Nov 8, 2019
7407797
resolve comments
elvaliuliuliu Nov 9, 2019
2793106
update
elvaliuliuliu Nov 9, 2019
6faa69b
update
elvaliuliuliu Nov 9, 2019
c3c2027
update
elvaliuliuliu Nov 9, 2019
4befea8
Update
elvaliuliuliu Nov 9, 2019
779e933
Add examples
elvaliuliuliu Nov 9, 2019
19e5397
Add yarn E2E example
elvaliuliuliu Nov 9, 2019
e23da68
Merge branch 'master' into elva/deployworker-udf
elvaliuliuliu Nov 11, 2019
04afe83
resolve comments
elvaliuliuliu Nov 12, 2019
17360bd
update format
elvaliuliuliu Nov 14, 2019
7e8ab94
Change to backslash
elvaliuliuliu Nov 19, 2019
c2d2879
Merge branch 'master' into elva/deployworker-udf
elvaliuliuliu Nov 19, 2019
ca9ae5f
update path
elvaliuliuliu Nov 19, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
resolve comments
  • Loading branch information
@elvaliuliuliu
elvaliuliuliu authored Nov 12, 2019
commit 04afe83d8a2541e027ec7b638010e8bdf27f6f64
16 changes: 11 additions & 5 deletions docs/deploy-worker-udf-binaries.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
# Deploy Worker and UDF Binaries General Instruction

This how-to provides general instructions on how to deploy Worker and UDF (User-Defined Function) binaries, including which Environment Variables to set up and some commonly used parameters when launching applications with `spark-submit`.
This how-to provides general instructions on how to deploy Worker and UDF (User-Defined Function) binaries,
including which Environment Variables to set up and some commonly used parameters
when launching applications with `spark-submit`.

## Configurations

Expand Down Expand Up @@ -53,15 +55,19 @@ Once the Spark application is [bundled](https://spark.apache.org/docs/latest/sub
<tr>
<td><b>--files</b></td>
<td>Comma-separated list of files to be placed in the working directory of each executor.</br>
<li>Please note that this option is only applicable for yarn mode.</li>
<li>It supports specifying file names with # similar to Hadoop.</br>
<ul>
<li>Please note that this option is only applicable for yarn mode.</li>
<li>It supports specifying file names with # similar to Hadoop.</br>
</ul>
<i>e.g. <code>myLocalSparkApp.dll#appSeen.dll</code>. Your application should use the name as <code>appSeen.dll</code> to reference <code>myLocalSparkApp.dll</code> when running on YARN.</i></li></td>
</tr>
<tr>
<td><b>--archives</b></td>
<td>Comma-separated list of archives to be extracted into the working directory of each executor.</br>
<li>Please note that this option is only applicable for yarn mode.</li>
<li>It supports specifying file names with # similar to Hadoop.</br>
<ul>
<li>Please note that this option is only applicable for yarn mode.</li>
<li>It supports specifying file names with # similar to Hadoop.</br>
</ul>
<i>e.g. <code>hdfs://&lt;path to your worker file&gt;/Microsoft.Spark.Worker.net461.win-x64-0.6.0.zip#worker</code>. This will copy and extract the zip file to <code>worker</code> folder.</i></li></td>
</tr>
<tr>
Expand Down