Skip to content

MODBULKOPS-337 - Stage 3b - FQM Performance (FQL): switching matched records flow to use mod-bulk-operations capabilities #423

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 26 commits into from
Jun 4, 2025
Merged
Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
c337f6f
MODBULKOPS-333 Test loan type
obozhko-folio May 28, 2025
735fc45
MODBULKOPS-333 Test loan type
obozhko-folio May 28, 2025
362e2b3
MODBULKOPS-333 Test loan type
obozhko-folio May 28, 2025
91b8eca
MODBULKOPS-333 Test loan type
obozhko-folio May 28, 2025
5ff6148
MODBULKOPS-333 Use deserializer
obozhko-folio May 28, 2025
a947c39
MODBULKOPS-333 Use getter/setter
obozhko-folio May 29, 2025
4e244d2
MODBULKOPS-333 Fixed tests
obozhko-folio May 29, 2025
f597197
MODBULKOPS-333 Fixed missing reference data names
obozhko-folio May 30, 2025
b550414
Merge branch 'master' into MODBULKOPS-333
obozhko-folio May 30, 2025
b0a5806
MODBULKOPS-333 Fixed used tenants for ui
obozhko-folio May 30, 2025
c46cddb
MODBULKOPS-333 Fixed tests
obozhko-folio May 30, 2025
829240b
MODBULKOPS-333 Added tests
obozhko-folio May 30, 2025
59169d8
MODBULKOPS-333 Added tests
obozhko-folio May 30, 2025
47031f7
MODBULKOPS-333 Fixed according to comments
obozhko-folio Jun 2, 2025
4e49e24
Merge branch 'master' into MODBULKOPS-333
obozhko-folio Jun 2, 2025
c6496ff
MODBULKOPS-333 Refactored
obozhko-folio Jun 3, 2025
4abbb6e
Merge remote-tracking branch 'origin/MODBULKOPS-333' into MODBULKOPS-333
obozhko-folio Jun 3, 2025
8eb5fc4
MODBULKOPS-333 Logging issue
obozhko-folio Jun 3, 2025
02a5c25
MODBULKOPS-333 Logging issue
obozhko-folio Jun 3, 2025
ae9c8c5
MODBULKOPS-333 Added property order
obozhko-folio Jun 3, 2025
2431e15
MODBULKOPS-333 Removed property order, added constructor
obozhko-folio Jun 3, 2025
7ecb1f7
MODBULKOPS-333 Fixed minor
obozhko-folio Jun 3, 2025
9a61d39
MODBULKOPS-333 Fixed error file name
obozhko-folio Jun 3, 2025
749fc59
MODBULKOPS-333 Rolled back jsonproperty
obozhko-folio Jun 4, 2025
0ed1d76
MODBULKOPS-333 Fixed tenant in notes
obozhko-folio Jun 4, 2025
ab439fd
MODBULKOPS-333 Fixed tenant in notes
obozhko-folio Jun 4, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,15 +1,20 @@
package org.folio.bulkops.client;

import org.folio.bulkops.configs.FeignClientConfiguration;
import org.folio.bulkops.domain.bean.MaterialType;
import org.folio.bulkops.domain.bean.MaterialTypeCollection;
import org.springframework.cloud.openfeign.FeignClient;
import org.springframework.http.MediaType;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestParam;

@FeignClient(name = "material-types", configuration = FeignClientConfiguration.class)
public interface MaterialTypeClient {

@GetMapping(produces = MediaType.APPLICATION_JSON_VALUE)
MaterialTypeCollection getByQuery(@RequestParam String query);

@GetMapping(value = "/{id}", produces = MediaType.APPLICATION_JSON_VALUE)
MaterialType getById(@PathVariable String id);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use here UUID type instead of String?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably yes, but in all other clients related to reference data we use String.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's stay as is.

}
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@
public interface BulkOperationsEntity {
String getIdentifier(IdentifierType identifierType);
Integer _version();
default void enrichItemWithReferenceDataNames() {
}
@JsonIgnore
default String getId() {
return null;
Expand All @@ -26,6 +28,10 @@ default String getSource() {
return StringUtils.EMPTY;
}
@JsonIgnore
default boolean isItem() {
return false;
}
@JsonIgnore
default BulkOperationsEntity getRecordBulkOperationEntity() {
return null;
}
Expand Down
36 changes: 36 additions & 0 deletions src/main/java/org/folio/bulkops/domain/bean/ExtendedItem.java
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
package org.folio.bulkops.domain.bean;

import static java.util.Objects.isNull;
import static java.util.Objects.nonNull;

import java.util.List;

import com.fasterxml.jackson.annotation.JsonProperty;
Expand All @@ -10,6 +13,7 @@
import lombok.With;
import org.folio.bulkops.domain.dto.IdentifierType;
import org.folio.bulkops.domain.dto.TenantNotePair;
import org.folio.bulkops.service.ItemReferenceHelper;

@Data
@With
Expand Down Expand Up @@ -63,4 +67,36 @@ public void setTenantToNotes(List<TenantNotePair> tenantNotePairs) {
public String getId() {
return entity.getId();
}

@Override
public boolean isItem() {
return true;
}

public void enrichItemWithReferenceDataNames() {
if (nonNull(entity.getMaterialType()) && isNull(entity.getMaterialType().getName())) {
var name = ItemReferenceHelper.service().getMaterialTypeById(entity.getMaterialTypeId(), tenantId).getName();
entity.getMaterialType().setName(name);
}
if (nonNull(entity.getPermanentLoanType()) && isNull(entity.getPermanentLoanType().getName())) {
var name = ItemReferenceHelper.service().getLoanTypeById(entity.getPermanentLoanTypeId(), tenantId).getName();
entity.getPermanentLoanType().setName(name);
}
if (nonNull(entity.getTemporaryLoanType()) && isNull(entity.getTemporaryLoanType().getName())) {
var name = ItemReferenceHelper.service().getLoanTypeById(entity.getTemporaryLoanTypeId(), tenantId).getName();
entity.getTemporaryLoanType().setName(name);
}
if (nonNull(entity.getPermanentLocation()) && isNull(entity.getPermanentLocation().getName())) {
var name = ItemReferenceHelper.service().getLocationById(entity.getPermanentLocationId(), tenantId).getName();
entity.getPermanentLocation().setName(name);
}
if (nonNull(entity.getTemporaryLocation()) && isNull(entity.getTemporaryLocation().getName())) {
var name = ItemReferenceHelper.service().getLocationById(entity.getTemporaryLocationId(), tenantId).getName();
entity.getTemporaryLocation().setName(name);
}
if (nonNull(entity.getEffectiveLocation()) && isNull(entity.getEffectiveLocation().getName())) {
var name = ItemReferenceHelper.service().getLocationById(entity.getEffectiveLocationId(), tenantId).getName();
entity.getEffectiveLocation().setName(name);
}
}
}
124 changes: 110 additions & 14 deletions src/main/java/org/folio/bulkops/domain/bean/Item.java
Original file line number Diff line number Diff line change
@@ -1,5 +1,13 @@
package org.folio.bulkops.domain.bean;

import static java.lang.Boolean.FALSE;
import static java.util.Collections.emptyList;
import static java.util.Objects.isNull;
import static org.apache.commons.lang3.StringUtils.EMPTY;

import java.util.Date;
import java.util.List;

import com.fasterxml.jackson.annotation.JsonIgnore;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.annotation.JsonTypeName;
Expand Down Expand Up @@ -33,14 +41,6 @@
import org.folio.bulkops.domain.dto.IdentifierType;
import org.folio.bulkops.domain.dto.TenantNotePair;

import java.util.Date;
import java.util.List;

import static java.lang.Boolean.FALSE;
import static java.util.Collections.emptyList;
import static java.util.Objects.isNull;
import static org.apache.commons.lang3.StringUtils.EMPTY;

@Data
@With
@Builder(toBuilder = true)
Expand Down Expand Up @@ -68,12 +68,27 @@ public class Item implements BulkOperationsEntity, ElectronicAccessEntity {
@UnifiedTableCell(visible = false)
private String holdingsData;

@JsonProperty("effectiveLocation")
@JsonIgnore
@CsvCustomBindByName(column = "Item effective location", converter = ItemLocationConverter.class)
@CsvCustomBindByPosition(position = 3, converter = ItemLocationConverter.class)
@UnifiedTableCell
private ItemLocation effectiveLocation;

@JsonProperty("effectiveLocationId")
public void setEffectiveLocationId(String id) {
this.effectiveLocation = new ItemLocation().withId(id);
}

@JsonProperty("effectiveLocationId")
public String getEffectiveLocationId() {
return effectiveLocation != null ? effectiveLocation.getId() : null;
}

@JsonProperty("effectiveLocation")
public ItemLocation getEffectiveLocation() {
return effectiveLocation;
}

@JsonProperty("effectiveCallNumberComponents")
@CsvCustomBindByName(column = "Effective call number", converter = EffectiveCallNumberComponentsConverter.class)
@CsvCustomBindByPosition(position = 4, converter = EffectiveCallNumberComponentsConverter.class)
Expand Down Expand Up @@ -131,12 +146,27 @@ public class Item implements BulkOperationsEntity, ElectronicAccessEntity {
@UnifiedTableCell(visible = false)
private List<String> administrativeNotes;

@JsonProperty("materialType")
@JsonIgnore
@CsvCustomBindByName(column = "Material type", converter = MaterialTypeConverter.class)
@CsvCustomBindByPosition(position = 13, converter = MaterialTypeConverter.class)
@UnifiedTableCell
private MaterialType materialType;

@JsonProperty("materialTypeId")
public void setMaterialTypeId(String id) {
this.materialType = new MaterialType().withId(id);
}

@JsonProperty("materialTypeId")
public String getMaterialTypeId() {
return materialType != null ? materialType.getId() : null;
}

@JsonProperty("materialType")
public MaterialType getMaterialType() {
return materialType;
}

@JsonProperty("copyNumber")
@CsvCustomBindByName(column = "Copy number", converter = StringConverter.class)
@CsvCustomBindByPosition(position = 14, converter = StringConverter.class)
Expand Down Expand Up @@ -247,18 +277,48 @@ public class Item implements BulkOperationsEntity, ElectronicAccessEntity {
@UnifiedTableCell(visible = false)
private List<ItemNote> notes;

@JsonProperty("permanentLoanType")
@JsonIgnore
@CsvCustomBindByName(column = "Permanent loan type", converter = LoanTypeConverter.class)
@CsvCustomBindByPosition(position = 32, converter = LoanTypeConverter.class)
@UnifiedTableCell
private LoanType permanentLoanType;

@JsonProperty("temporaryLoanType")
@JsonProperty("permanentLoanTypeId")
public void setPermanentLoanTypeId(String id) {
this.permanentLoanType = new LoanType().withId(id);
}

@JsonProperty("permanentLoanTypeId")
public String getPermanentLoanTypeId() {
return permanentLoanType != null ? permanentLoanType.getId() : null;
}

@JsonProperty("permanentLoanType")
public LoanType getPermanentLoanType() {
return permanentLoanType;
}

@JsonIgnore
@CsvCustomBindByName(column = "Temporary loan type", converter = LoanTypeConverter.class)
@CsvCustomBindByPosition(position = 33, converter = LoanTypeConverter.class)
@UnifiedTableCell
private LoanType temporaryLoanType;

@JsonProperty("temporaryLoanTypeId")
public void setTemporaryLoanTypeId(String id) {
this.temporaryLoanType = new LoanType().withId(id);
}

@JsonProperty("temporaryLoanTypeId")
public String getTemporaryLoanTypeId() {
return temporaryLoanType != null ? temporaryLoanType.getId() : null;
}

@JsonProperty("temporaryLoanType")
public LoanType getTemporaryLoanType() {
return temporaryLoanType;
}

@JsonProperty("status")
@CsvCustomBindByName(column = "Status", converter = ItemStatusConverter.class)
@CsvCustomBindByPosition(position = 34, converter = ItemStatusConverter.class)
Expand All @@ -277,18 +337,49 @@ public class Item implements BulkOperationsEntity, ElectronicAccessEntity {
@UnifiedTableCell(visible = false)
private List<CirculationNote> checkOutNotes;

@JsonProperty("permanentLocation")
@JsonIgnore
@CsvCustomBindByName(column = "Item permanent location", converter = ItemLocationConverter.class)
@CsvCustomBindByPosition(position = 37, converter = ItemLocationConverter.class)
@UnifiedTableCell(visible = false)
private ItemLocation permanentLocation;

@JsonProperty("temporaryLocation")
@JsonProperty("permanentLocationId")
public void setPermanentLocationId(String id) {
this.permanentLocation = new ItemLocation().withId(id);
}

@JsonProperty("permanentLocationId")
public String getPermanentLocationId() {
return permanentLocation != null ? permanentLocation.getId() : null;
}

@JsonProperty("permanentLocation")
public ItemLocation getPermanentLocation() {
return permanentLocation;
}

@JsonIgnore
@CsvCustomBindByName(column = "Item temporary location", converter = ItemLocationConverter.class)
@CsvCustomBindByPosition(position = 38, converter = ItemLocationConverter.class)
@UnifiedTableCell(visible = false)
private ItemLocation temporaryLocation;

@JsonProperty("temporaryLocationId")
public void setTemporaryLocationId(String id) {
this.temporaryLocation = new ItemLocation().withId(id);
}

@JsonProperty("temporaryLocationId")
public String getTemporaryLocationId() {
return temporaryLocation != null ? temporaryLocation.getId() : null;
}

@JsonProperty("temporaryLocation")
public ItemLocation getTemporaryLocation() {
return temporaryLocation;
}


@JsonProperty("electronicAccess")
@Valid
@CsvCustomBindByName(column = "Electronic access", converter = ItemElectronicAccessListConverter.class)
Expand Down Expand Up @@ -367,4 +458,9 @@ public void setTenantToNotes(List<TenantNotePair> tenantNotePairs) {
public void setTenant(String tenantId) {
this.tenantId = tenantId;
}

@Override
public boolean isItem() {
return true;
}
}
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
package org.folio.bulkops.processor.folio;

import static org.folio.bulkops.util.FolioExecutionContextUtil.prepareContextForTenant;

import lombok.RequiredArgsConstructor;
import org.folio.bulkops.client.ItemClient;
import org.folio.bulkops.domain.bean.ExtendedItem;
import org.folio.bulkops.domain.dto.EntityType;
Expand All @@ -11,10 +14,6 @@
import org.folio.spring.scope.FolioExecutionContextSetter;
import org.springframework.stereotype.Component;

import lombok.RequiredArgsConstructor;

import static org.folio.bulkops.util.FolioExecutionContextUtil.prepareContextForTenant;

@Component
@RequiredArgsConstructor
public class ItemUpdateProcessor extends FolioAbstractUpdateProcessor<ExtendedItem> {
Expand All @@ -40,7 +39,7 @@ public void updateRecord(ExtendedItem extendedItem) {
} else {
permissionsValidator.checkIfBulkEditWritePermissionExists(folioExecutionContext.getTenantId(), EntityType.ITEM,
NO_ITEM_WRITE_PERMISSIONS_TEMPLATE + folioExecutionContext.getTenantId());
itemClient.updateItem(item.withHoldingsData(null), item.getId());
itemClient.updateItem(item.withHoldingsData(null), item.getId());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inline: roll-back format.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

}
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -252,6 +252,9 @@ public void confirm(BulkOperationDataProcessing dataProcessing) {

while (iterator.hasNext()) {
var original = iterator.next();
if (original.isItem()) {
original.enrichItemWithReferenceDataNames();
}
if (INSTANCE_MARC.equals(operation.getEntityType()) && original instanceof ExtendedInstance extendedInstance
&& !MARC.equals(extendedInstance.getEntity().getSource())) {
var instance = extendedInstance.getEntity();
Expand Down Expand Up @@ -426,11 +429,17 @@ public void commit(BulkOperation operation) {

while (hasNextRecord(originalFileIterator, modifiedFileIterator)) {
var original = originalFileIterator.next();
if (original.isItem()) {
original.enrichItemWithReferenceDataNames();
}
if (INSTANCE_MARC.equals(operation.getEntityType()) && original instanceof ExtendedInstance extendedInstance
&& !MARC.equals(extendedInstance.getEntity().getSource())) {
continue;
}
var modified = modifiedFileIterator.next();
if (modified.isItem()) {
modified.enrichItemWithReferenceDataNames();
}
List<BulkOperationExecutionContent> bulkOperationExecutionContents = new ArrayList<>();

processedNumOfRecords++;
Expand Down
12 changes: 12 additions & 0 deletions src/main/java/org/folio/bulkops/service/ItemReferenceHelper.java
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,18 @@ public LoanType getLoanTypeByName(String name) {
return itemReferenceService.getLoanTypeByName(name);
}

public LoanType getLoanTypeById(String id, String tenantId) {
return itemReferenceService.getLoanTypeById(id, tenantId);
}

public MaterialType getMaterialTypeById(String id, String tenantId) {
return itemReferenceService.getMaterialTypeById(id, tenantId);
}

public ItemLocation getLocationById(String id, String tenantId) {
return itemReferenceService.getLocationById(id, tenantId);
}

private static ItemReferenceHelper service;

@Override
Expand Down
Loading