Skip to content

Allow a tsdb data stream to rolled over to a logsdb data stream #126640

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Apr 11, 2025
Merged

Allow a tsdb data stream to rolled over to a logsdb data stream #126640

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
2a93c8a
Allow a tsdb data stream to rolled over to a logsdb data stream
martijnvg Apr 10, 2025
be6801e
Merge remote-tracking branch 'es/main' into ds_rollover_tsdb_to_logsdb
martijnvg Apr 10, 2025
1ade709
Merge remote-tracking branch 'es/main' into ds_rollover_tsdb_to_logsdb
martijnvg Apr 11, 2025
078968b
added warning logging
martijnvg Apr 11, 2025
18f4503
Merge remote-tracking branch 'es/main' into ds_rollover_tsdb_to_logsdb
martijnvg Apr 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions server/src/main/java/org/elasticsearch/cluster/metadata/DataStream.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@
*/
package org.elasticsearch.cluster.metadata;

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.apache.lucene.document.LongPoint;
import org.apache.lucene.index.DocValuesSkipIndexType;
import org.apache.lucene.index.DocValuesSkipper;
Expand Down Expand Up @@ -71,6 +73,8 @@

public final class DataStream implements SimpleDiffable<DataStream>, ToXContentObject, IndexAbstraction {

private static final Logger LOGGER = LogManager.getLogger(DataStream.class);

public static final boolean FAILURE_STORE_FEATURE_FLAG = new FeatureFlag("failure_store").isEnabled();
public static final TransportVersion ADDED_FAILURE_STORE_TRANSPORT_VERSION = TransportVersions.V_8_12_0;
public static final TransportVersion ADDED_AUTO_SHARDING_EVENT_VERSION = TransportVersions.V_8_14_0;
Expand Down Expand Up @@ -610,6 +614,12 @@ public DataStream unsafeRollover(
} else if (dsIndexMode == IndexMode.LOGSDB && (indexModeFromTemplate == null || indexModeFromTemplate == IndexMode.STANDARD)) {
// Allow downgrading a time series data stream to a regular data stream
dsIndexMode = null;
} else if (dsIndexMode == IndexMode.TIME_SERIES && indexModeFromTemplate == IndexMode.LOGSDB) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is dsIndexMode the index mode of the write index?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It indicates whether a data stream is a tsdb data stream or a logsdb data stream. It isn't always the index mode of the most recent backing index, because the standard and lookup mode are never set as index mode of a data stream (there is no such thing as a lookup data stream). There is logic that checks the index mode (all checks are for time series iirc) of the data stream without having the to lookup the most recent IndexMetadata.

But if a data stream's index mode is logsdb or time series, then the most recent backing index's index mode should also be time series of logsdb.

dsIndexMode = IndexMode.LOGSDB;
LOGGER.warn("Changing [{}] index mode from [{}] to [{}]", name, indexModeFromTemplate, dsIndexMode);
} else if (dsIndexMode == IndexMode.LOGSDB && indexModeFromTemplate == IndexMode.TIME_SERIES) {
dsIndexMode = IndexMode.TIME_SERIES;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we be adding a warning too? Just in case this is done by mistake..

LOGGER.warn("Changing [{}] index mode from [{}] to [{}]", name, indexModeFromTemplate, dsIndexMode);
}

List<Index> backingIndices = new ArrayList<>(this.backingIndices.indices);
Expand Down
33 changes: 33 additions & 0 deletions server/src/test/java/org/elasticsearch/cluster/metadata/DataStreamTests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -277,6 +277,39 @@ public void testRolloverUpgradeToLogsdbDataStream() {
assertThat(rolledDs.getIndexMode(), equalTo(IndexMode.LOGSDB));
}

public void testRolloverFromTSdbToLogsdb() {
DataStream ds = DataStreamTestHelper.randomInstance().copy().setReplicated(false).setIndexMode(IndexMode.TIME_SERIES).build();
final var project = ProjectMetadata.builder(randomProjectIdOrDefault()).build();
var newCoordinates = ds.nextWriteIndexAndGeneration(project, ds.getDataComponent());

var rolledDs = ds.rollover(new Index(newCoordinates.v1(), UUIDs.randomBase64UUID()), newCoordinates.v2(), IndexMode.LOGSDB, null);
assertThat(rolledDs.getName(), equalTo(ds.getName()));
assertThat(rolledDs.getGeneration(), equalTo(ds.getGeneration() + 1));
assertThat(rolledDs.getIndices().size(), equalTo(ds.getIndices().size() + 1));
assertTrue(rolledDs.getIndices().containsAll(ds.getIndices()));
assertTrue(rolledDs.getIndices().contains(rolledDs.getWriteIndex()));
assertThat(rolledDs.getIndexMode(), equalTo(IndexMode.LOGSDB));
}

public void testRolloverFromLogsdbToTsdb() {
DataStream ds = DataStreamTestHelper.randomInstance().copy().setReplicated(false).setIndexMode(IndexMode.LOGSDB).build();
final var project = ProjectMetadata.builder(randomProjectIdOrDefault()).build();
var newCoordinates = ds.nextWriteIndexAndGeneration(project, ds.getDataComponent());

var rolledDs = ds.rollover(
new Index(newCoordinates.v1(), UUIDs.randomBase64UUID()),
newCoordinates.v2(),
IndexMode.TIME_SERIES,
null
);
assertThat(rolledDs.getName(), equalTo(ds.getName()));
assertThat(rolledDs.getGeneration(), equalTo(ds.getGeneration() + 1));
assertThat(rolledDs.getIndices().size(), equalTo(ds.getIndices().size() + 1));
assertTrue(rolledDs.getIndices().containsAll(ds.getIndices()));
assertTrue(rolledDs.getIndices().contains(rolledDs.getWriteIndex()));
assertThat(rolledDs.getIndexMode(), equalTo(IndexMode.TIME_SERIES));
}

public void testRolloverDowngradeFromTsdbToRegularDataStream() {
DataStream ds = DataStreamTestHelper.randomInstance().copy().setReplicated(false).setIndexMode(IndexMode.TIME_SERIES).build();
final var project = ProjectMetadata.builder(randomProjectIdOrDefault()).build();
Expand Down