-
Notifications
You must be signed in to change notification settings - Fork 132
feat: add support for BatchWriteAtLeastOnce #2520
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
7ff7cb2
43c4f63
abedd15
21d8f67
4858965
81743a7
65abc96
8d76346
a3569ae
4cb0ada
5401b7d
5103b5c
95c3326
5678207
347e9c3
4c7251f
1deaa29
acece36
5f04154
8e15c5c
56f3929
356849e
07f3bfb
1ca0d31
c3e3d7b
a036ced
d8ef791
7c4c73d
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
- Loading branch information
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -194,75 +194,58 @@ CommitResponse writeAtLeastOnceWithOptions( | |
Iterable<Mutation> mutations, TransactionOption... options) throws SpannerException; | ||
|
||
/** | ||
* Batches the supplied mutations in a collection of efficient transactions. The mutations are | ||
* applied non-atomically in an unspecified order and thus, they must be independent of each | ||
* other. Partial failure is possible, i.e., some mutations may have been applied successfully, | ||
* while some may have failed. The results of individual batches are streamed into the response as | ||
* and when the batches are applied. | ||
* Applies batch of mutation groups in a collection of efficient transactions. The mutation groups | ||
* are applied non-atomically in an unspecified order and thus, they must be independent of each | ||
* other. Partial failure is possible, i.e., some mutation groups may have been applied | ||
* successfully, while some may have failed. The results of individual batches are streamed into | ||
* the response as and when the batches are applied. | ||
* | ||
* <p>Since this method does not feature replay protection, it may attempt to apply {@code | ||
* mutations} more than once; if the mutations are not idempotent, this may lead to a failure | ||
* being reported when the mutation was applied once. For example, an insert may fail with {@link | ||
* ErrorCode#ALREADY_EXISTS} even though the row did not exist before this method was called. For | ||
* this reason, most users of the library will prefer to use {@link #write(Iterable)} instead. | ||
* However, {@code batchWriteAtLeastOnce()} method may be appropriate for non-atomically | ||
* committing multiple mutations in a single RPC with low latency. | ||
* mutation groups} more than once; if the mutation groups are not idempotent, this may lead to a | ||
* failure being reported when the mutation group was applied once. For example, an insert may | ||
rajatbhatta marked this conversation as resolved.
Show resolved
Hide resolved
|
||
* fail with {@link ErrorCode#ALREADY_EXISTS} even though the row did not exist before this method | ||
* was called. For this reason, most users of the library will prefer to use {@link | ||
* #write(Iterable)} instead. However, {@code batchWriteAtLeastOnce()} method may be appropriate | ||
* for non-atomically committing multiple mutation groups in a single RPC with low latency. | ||
* | ||
* <p>Example of BatchWriteAtleastOnce | ||
* <p>Example of BatchWriteAtLeastOnce | ||
* | ||
* <pre>{@code | ||
* long singerId = my_singer_id; | ||
* Mutation mutation = Mutation.newInsertBuilder("Singers") | ||
* .set("SingerId") | ||
* .to(singerId) | ||
* .set("FirstName") | ||
* .to("Billy") | ||
* .set("LastName") | ||
* .to("Joel") | ||
* .build(); | ||
* ServerStream<BatchWriteResponse> responses = | ||
* dbClient.batchWriteAtLeastOnce(Collections.singletonList(mutation)); | ||
* dbClient.batchWriteAtLeastOnceWithOptions( | ||
* ImmutableList.of(MUTATION_GROUP1, MUTATION_GROUP2)); | ||
rajatbhatta marked this conversation as resolved.
Show resolved
Hide resolved
|
||
* for (BatchWriteResponse response : responses) { | ||
* // Do something when a response is received. | ||
* } | ||
* }</pre> | ||
* | ||
* @return ServerStream\<BatchWriteResponse> | ||
rajatbhatta marked this conversation as resolved.
Show resolved
Hide resolved
|
||
*/ | ||
ServerStream<BatchWriteResponse> batchWriteAtLeastOnce(Iterable<Mutation> mutations) | ||
ServerStream<BatchWriteResponse> batchWriteAtLeastOnce(Iterable<MutationGroup> mutationGroups) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. When should I create multiple MutationGroups instead of just one large mutation group? What's the tradeoff between a large number of small MutationGroups vs a few large MutationGroups? Is there any limits on the number of mutations that I may have in a MutationGroup? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Dependent mutations (to be committed together) should be part of a single MutationGroup. For example, the following should be part of same MutationGroup:
If there's a need to create just one large MutationGroup, it's better to use the existing Commit RPC for it.
Depends on the use case. The bottomline is we should only put mutations meant to be committed together in a group, otherwise, should prefer a different group.
Considering that in the base case, each mutation group could be committed into its own transaction, which has a limit of 40k mutations, the same limit would apply to the MutationGroup also. |
||
throws SpannerException; | ||
|
||
/** | ||
* Batches the supplied mutations in a collection of efficient transactions. The mutations are | ||
* applied non-atomically in an unspecified order and thus, they must be independent of each | ||
* other. Partial failure is possible, i.e., some mutations may have been applied successfully, | ||
* while some may have failed. The results of individual batches are streamed into the response as | ||
* and when the batches are applied. | ||
* Applies batch of mutation groups in a collection of efficient transactions. The mutation groups | ||
* are applied non-atomically in an unspecified order and thus, they must be independent of each | ||
* other. Partial failure is possible, i.e., some mutation groups may have been applied | ||
* successfully, while some may have failed. The results of individual batches are streamed into | ||
* the response as and when the batches are applied. | ||
* | ||
* <p>Since this method does not feature replay protection, it may attempt to apply {@code | ||
* mutations} more than once; if the mutations are not idempotent, this may lead to a failure | ||
* being reported when the mutation was applied once. For example, an insert may fail with {@link | ||
* ErrorCode#ALREADY_EXISTS} even though the row did not exist before this method was called. For | ||
* this reason, most users of the library will prefer to use {@link #write(Iterable)} instead. | ||
* However, {@code batchWriteAtLeastOnce()} method may be appropriate for non-atomically | ||
* committing multiple mutations in a single RPC with low latency. | ||
* mutation groups} more than once; if the mutation groups are not idempotent, this may lead to a | ||
* failure being reported when the mutation group was applied once. For example, an insert may | ||
* fail with {@link ErrorCode#ALREADY_EXISTS} even though the row did not exist before this method | ||
* was called. For this reason, most users of the library will prefer to use {@link | ||
* #write(Iterable)} instead. However, {@code batchWriteAtLeastOnce()} method may be appropriate | ||
* for non-atomically committing multiple mutation groups in a single RPC with low latency. | ||
* | ||
* <p>Example of BatchWriteAtleastOnceWithOptions | ||
* <p>Example of BatchWriteAtLeastOnceWithOptions | ||
* | ||
* <pre>{@code | ||
* long singerId = my_singer_id; | ||
* Mutation mutation = Mutation.newInsertBuilder("Singers") | ||
* .set("SingerId") | ||
* .to(singerId) | ||
* .set("FirstName") | ||
* .to("Billy") | ||
* .set("LastName") | ||
* .to("Joel") | ||
* .build(); | ||
* ServerStream<BatchWriteResponse> responses = | ||
* dbClient.batchWriteAtLeastOnce( | ||
* Collections.singletonList(mutation), | ||
* Options.priority(RpcPriority.LOW)); | ||
* dbClient.batchWriteAtLeastOnceWithOptions( | ||
* ImmutableList.of(MUTATION_GROUP1, MUTATION_GROUP2), | ||
* Options.tag("batch-write-tag")); | ||
* for (BatchWriteResponse response : responses) { | ||
* // Do something when a response is received. | ||
* } | ||
|
@@ -279,7 +262,7 @@ ServerStream<BatchWriteResponse> batchWriteAtLeastOnce(Iterable<Mutation> mutati | |
* @return ServerStream\<BatchWriteResponse> | ||
*/ | ||
ServerStream<BatchWriteResponse> batchWriteAtLeastOnceWithOptions( | ||
rajatbhatta marked this conversation as resolved.
Show resolved
Hide resolved
|
||
Iterable<Mutation> mutations, TransactionOption... options) throws SpannerException; | ||
Iterable<MutationGroup> mutationGroups, TransactionOption... options) throws SpannerException; | ||
|
||
/** | ||
* Returns a context in which a single read can be performed using {@link TimestampBound#strong()} | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,39 @@ | ||
package com.google.cloud.spanner; | ||
|
||
import com.google.common.base.Preconditions; | ||
import com.google.spanner.v1.BatchWriteRequest; | ||
import java.util.ArrayList; | ||
import java.util.Arrays; | ||
import java.util.List; | ||
|
||
public class MutationGroup { | ||
private final List<Mutation> mutations; | ||
|
||
private MutationGroup(List<Mutation> mutations) { | ||
this.mutations = mutations; | ||
} | ||
|
||
public static MutationGroup of(Mutation... mutations) { | ||
rajatbhatta marked this conversation as resolved.
Show resolved
Hide resolved
rajatbhatta marked this conversation as resolved.
Show resolved
Hide resolved
|
||
Preconditions.checkArgument(mutations.length > 0, "Should pass in at least one mutation."); | ||
return new MutationGroup(Arrays.asList(mutations)); | ||
rajatbhatta marked this conversation as resolved.
Show resolved
Hide resolved
|
||
} | ||
|
||
public List<Mutation> getMutations() { | ||
rajatbhatta marked this conversation as resolved.
Show resolved
Hide resolved
|
||
return mutations; | ||
} | ||
|
||
static BatchWriteRequest.MutationGroup toProto(final MutationGroup mutationGroup) { | ||
List<com.google.spanner.v1.Mutation> mutationsProto = new ArrayList<>(); | ||
Mutation.toProto(mutationGroup.getMutations(), mutationsProto); | ||
return BatchWriteRequest.MutationGroup.newBuilder().addAllMutations(mutationsProto).build(); | ||
olavloite marked this conversation as resolved.
Show resolved
Hide resolved
|
||
} | ||
|
||
static List<BatchWriteRequest.MutationGroup> toListProto( | ||
final Iterable<MutationGroup> mutationGroups) { | ||
List<BatchWriteRequest.MutationGroup> mutationGroupsProto = new ArrayList<>(); | ||
for (MutationGroup group : mutationGroups) { | ||
mutationGroupsProto.add(toProto(group)); | ||
} | ||
return mutationGroupsProto; | ||
} | ||
} |
Uh oh!
There was an error while loading. Please reload this page.