feat: introduction to LeaseRead

[ariesdevil]

Checklist

• Updated guide with pertinent info (may not always apply).
• Squash down commits to one or two logical commits which clearly describe the work you've done.
• Unittest is a friend:)

---

This change is 

[drmingdrmer]

Reviewed 14 of 16 files at r1, all commit messages.
Reviewable status: 14 of 16 files reviewed, 3 unresolved discussions

---

openraft/src/config/config.rs line 96 at r1 (raw file):

    LeaseRead,
    ReadIndex,
}

It should not be a config entry. Let user decide which policy to use for read.

None is useless and should be removed.

Both policies are read-index based. The difference is that one of them use heartbeat to ensure leadership and the other use leader lease to ensure leadership. Use terminologies that reflect the purpose, such as Heartbeat and LeaderLease.

While there is not yet another ReadWritePolicy, ReadPolicy would be good enough for this enum than ReadOnlyPolicy.

Code quote:

#[derive(Clone, Debug, Display, PartialEq, Eq, ValueEnum)]
#[cfg_attr(feature = "serde", derive(serde::Deserialize, serde::Serialize))]
pub enum ReadOnlyPolicy {
    None,
    LeaseRead,
    ReadIndex,
}

---

openraft/src/config/config.rs line 146 at r1 (raw file):

    /// The range of `leader_lease_ratio` is (0, 100], default value is 90.
    #[clap(long, default_value = "90")]
    pub leader_lease_ratio: u64,

In OpenRaft, the leader_lease is set to the value of election_timeout_max.
A follower start to elect if it does not receive a heartbeat in leader_lease + election_timeout:

See:
https://github.com/datafuselabs/openraft/blob/50cb1cc3aa7e561361d018b8be17a93b7c57771a/openraft/src/core/raft_core.rs#L1509

There is no need to add such a leader_elase_ratio.

---

openraft/src/engine/engine_config.rs line 50 at r1 (raw file):

                election_timeout,
                smaller_log_timeout: Duration::from_millis(config.election_timeout_max * 2),
                leader_lease: config.leader_lease_timeout(rand_election_timeout),

The leader_lease can not be a random value on each node. Every node must agree on the same leader lease.

[drmingdrmer]

Reviewed 2 of 11 files at r2, 6 of 12 files at r3, 14 of 15 files at r4, 1 of 1 files at r5, all commit messages.
Reviewable status: 17 of 18 files reviewed, 7 unresolved discussions (waiting on @ariesdevil)

---

openraft/src/config/config.rs line 96 at r3 (raw file):

#[derive(Clone, Debug, Display, PartialEq, Eq, ValueEnum)]
#[cfg_attr(feature = "serde", derive(serde::Deserialize, serde::Serialize))]
pub enum ReadOnlyPolicy {

This type does not need ValueEnum or Serialize

Code quote:

#[derive(Clone, Debug, Display, PartialEq, Eq, ValueEnum)]
#[cfg_attr(feature = "serde", derive(serde::Deserialize, serde::Serialize))]
pub enum ReadOnlyPolicy {

---

tests/tests/client_api/t11_client_reads.rs line 265 at r5 (raw file):

    }

    // test LeaseRead policy

Split these two test cases into two function.

---

tests/tests/client_api/t11_client_reads.rs line 297 at r5 (raw file):

            .ensure_linearizable(leader, ReadPolicy::LeaseRead)
            .await
            .unwrap_or_else(|_| panic!("ensure_linearizable with `LeaseRead` failed for leader {}", leader));

This works because the append entries request during setting up the cluster behave as heartbeat.

The test itself should also assert that after a short while, LeaseRead will return an error.

And then trigger a round of heartbeat, with raft.trigger().heartbeat(). Then LeaseRead should return Ok again.

Code quote:

        router
            .ensure_linearizable(leader, ReadPolicy::LeaseRead)
            .await
            .unwrap_or_else(|_| panic!("ensure_linearizable with `LeaseRead` failed for leader {}", leader));

---

openraft/src/core/raft_core.rs line 280 at r3 (raw file):

            // Check if the lease is expired.
            if let Some(last_quorum_acked_time) = self.last_quorum_acked_time() {
                if now - last_quorum_acked_time < self.engine.config.timer_config.leader_lease {

last_quorum_acked_time may be greater than now and may underflow with the subtraction.
use now < last_quorum_acked_time + leader_lease instead.

Code quote:

            if let Some(last_quorum_acked_time) = self.last_quorum_acked_time() {
                if now - last_quorum_acked_time < self.engine.config.timer_config.leader_lease {

---

openraft/src/config/mod.rs line 9 at r5 (raw file):

pub use config::Config;
pub use config::ReadPolicy;

ReadPolicy is not config related. It's the argument of read APIs.
It should be in the raft mod.

[drmingdrmer]

LGTM

Reviewed 12 of 13 files at r6, 1 of 1 files at r7, 2 of 2 files at r8, all commit messages.
Reviewable status: all files reviewed, 3 unresolved discussions (waiting on @ariesdevil)

[drmingdrmer]

Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @ariesdevil)