Skip to content

scraper, project, superblock: Implement automated greylisting #2778

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
f6245a3
Initial machinery in project.h/cpp and blockchain.cpp for greylist st…
jamescowens Apr 28, 2024
9cda503
Scraper greylisting machinery (Part 1)
jamescowens May 1, 2024
c3c2936
Implement NotifyProjectChanged core signal
jamescowens Nov 25, 2024
322520e
Extend listprojects
jamescowens Nov 25, 2024
178b055
Enhance GUI to understand new greylisting status
jamescowens Jan 3, 2025
0daf7c1
Baseline changes for superblock v3 contracts
jamescowens Jan 4, 2025
2d05817
Initial implementation of AutoGreylist
jamescowens Jan 6, 2025
4dc35e6
Modify scraper code to accommodate all cpid total credit computation …
jamescowens Jan 7, 2025
ebf7858
Modify scraper and superblock code to support automatic greylisting (…
jamescowens Jan 12, 2025
c7ab778
Specify contract version 2 for superblocks in superblock_tests.cpp
jamescowens Jan 12, 2025
6510349
Modify scraper and superblock code (part 3)
jamescowens Jan 12, 2025
d540691
Add hidden entry for superblockv3height startup parameter for testing
jamescowens Jan 13, 2025
dc6cd75
Enhance SuperblockToJson for superblock v3
jamescowens Jan 13, 2025
79d7c3f
Add sending ProjectsAllCpidTotalCredits part to ScraperSendFileManife…
jamescowens Jan 13, 2025
8062bdd
Implement FromDoubleToString that allows specification of precision
jamescowens Jan 13, 2025
892c93c
Improve CScraperManifest::ToJson() to display project all cpid total …
jamescowens Jan 13, 2025
9c5da17
Additional changes to scraper from debugging
jamescowens Jan 13, 2025
39138cb
Changes to make clang initializer order checker happy
jamescowens Jan 13, 2025
a865103
Ensure auto greylist override in Whitelist::Snapshot doesn't modify u…
jamescowens Jan 14, 2025
1f9393e
Fix m_meets_greylisting_criteria determination
jamescowens Jan 15, 2025
406ac4a
Suppress greylisted projects on explainmagnitude output.
jamescowens Jan 15, 2025
3f93bcf
Fix NetworkWideStatsEntry initialization
jamescowens Jan 15, 2025
bba131a
Ensure AutoGreylist correctly deals with newly whitelisted projects
jamescowens Jan 15, 2025
648d97f
Add AUTO_GREYLIST_OVERRIDE to project status
jamescowens Jan 19, 2025
3efc659
Modify addkey and project constructor to deal with auto greylist over…
jamescowens Jan 19, 2025
5b9bafd
Add auto greylist update history and make other improvements
jamescowens Jan 19, 2025
69c6c4c
Wire in greylist history to getautogreylist
jamescowens Jan 19, 2025
08c1904
Change order of precedence for excluded project status in UI.
jamescowens Jan 19, 2025
16ee46e
Add documentation and other minor adjustments
jamescowens Jan 19, 2025
181c374
Change around the AutoGreylist initialization
jamescowens Jan 19, 2025
0477325
Update copyright on changed/added files.
jamescowens Jan 19, 2025
be3cfa4
Modify UpdateGreylistCandidateEntry and AutoGreylist::Reset()
jamescowens Jan 20, 2025
bb27ab8
Implement auto greylist boost unit test
jamescowens Jan 23, 2025
87563c3
Adjust vGreylistedProjects and include in convergencereport
jamescowens Jan 26, 2025
c7bb045
Add m_requires_ext_adapter to project entry and associated machinery
jamescowens Jan 26, 2025
9a4ef8c
Add version validation for superblock and project contracts
jamescowens Jan 26, 2025
9236467
Correct missing beaconChanged and researcherChanged signals in resear…
jamescowens Jan 27, 2025
7c1420a
Change iterator to reference to silence clang warning
jamescowens Feb 2, 2025
482d81e
Minor adjustments and documentation cleanup after self review
jamescowens Feb 3, 2025
d708d05
Add automated_greylisting_design_highlights.md to doc folder
jamescowens Feb 3, 2025
b8bce75
Adjust locking on Whitelist::GetProjectsFirstActive() to avoid potent…
jamescowens Feb 3, 2025
bfe4317
Correct Quorum::Hash on v3 serialized superblocks
jamescowens Feb 3, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add m_requires_ext_adapter to project entry and associated machinery
  • Loading branch information
jamescowens committed Apr 8, 2025
commit c7bb04573bc5881821448063f31bc684ba5a16c1
2 changes: 2 additions & 0 deletions src/chainparams.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ class CMainParams : public CChainParams {
consensus.DefaultConstantBlockReward = 10 * COIN;
consensus.ConstantBlockRewardFloor = 0;
consensus.ConstantBlockRewardCeiling = 500 * COIN;
consensus.ProjectV4Height = std::numeric_limits<int>::max();
consensus.SuperblockV3Height = std::numeric_limits<int>::max();
// Immediately post zero payment interval fees 40% for mainnet
consensus.InitialMRCFeeFractionPostZeroInterval = Fraction(2, 5);
Expand Down Expand Up @@ -194,6 +195,7 @@ class CTestNetParams : public CChainParams {
consensus.DefaultConstantBlockReward = 10 * COIN;
consensus.ConstantBlockRewardFloor = 0;
consensus.ConstantBlockRewardCeiling = 500 * COIN;
consensus.ProjectV4Height = std::numeric_limits<int>::max();
consensus.SuperblockV3Height = std::numeric_limits<int>::max();
// Immediately post zero payment interval fees 40% for testnet, the same as mainnet
consensus.InitialMRCFeeFractionPostZeroInterval = Fraction(2, 5);
Expand Down
7 changes: 7 additions & 0 deletions src/chainparams.h
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,13 @@ inline bool IsSuperblockV3Enabled(int nHeight)
return nHeight >= gArgs.GetArg("-superblockv3height", Params().GetConsensus().SuperblockV3Height);
}

inline bool IsProjectV4Enabled(int nHeight)
{
// The argument driven override temporarily here to facilitate testing.

return nHeight >= gArgs.GetArg("-projectv4height", Params().GetConsensus().ProjectV4Height);
}

inline int GetSuperblockAgeSpacing(int nHeight)
{
return (fTestNet ? 86400 : (nHeight > 364500) ? 86400 : 43200);
Expand Down
2 changes: 2 additions & 0 deletions src/consensus/params.h
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,8 @@ struct Params {
int PollV3Height;
/** Block height at which project v2 contracts are allowed */
int ProjectV2Height;
/** Block height at which project v4 contracts are allowed */
int ProjectV4Height;
/**
* @brief The default GRC paid for a constant block reward.
*
Expand Down
36 changes: 25 additions & 11 deletions src/gridcoin/project.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -49,30 +49,32 @@ ProjectEntry::ProjectEntry(uint32_t version)
, m_hash()
, m_previous_hash()
, m_gdpr_controls(false)
, m_requires_ext_adapter(false)
, m_public_key(CPubKey {})
, m_status(ProjectEntryStatus::UNKNOWN)
{
}

ProjectEntry::ProjectEntry(uint32_t version, std::string name, std::string url)
: ProjectEntry(version, name, url, false, ProjectEntryStatus::UNKNOWN, int64_t {0})
: ProjectEntry(version, name, url, false, false, ProjectEntryStatus::UNKNOWN, int64_t {0})
{
}

ProjectEntry::ProjectEntry(uint32_t version, std::string name, std::string url, bool gdpr_controls)
: ProjectEntry(version, name, url, gdpr_controls, ProjectEntryStatus::UNKNOWN, int64_t {0})
: ProjectEntry(version, name, url, gdpr_controls, false, ProjectEntryStatus::UNKNOWN, int64_t {0})
{
}

ProjectEntry::ProjectEntry(uint32_t version, std::string name, std::string url,
bool gdpr_controls, Status status, int64_t timestamp)
bool gdpr_controls, bool requires_ext_adapter, Status status, int64_t timestamp)
: m_version(version)
, m_name(name)
, m_url(url)
, m_timestamp(timestamp)
, m_hash()
, m_previous_hash()
, m_gdpr_controls(gdpr_controls)
, m_requires_ext_adapter(requires_ext_adapter)
, m_public_key(CPubKey {})
, m_status(status)
{
Expand Down Expand Up @@ -181,6 +183,17 @@ std::optional<bool> ProjectEntry::HasGDPRControls() const
return has_gdpr_controls;
}

std::optional<bool> ProjectEntry::RequiresExtAdapter() const
{
std::optional<bool> requires_ext_adapter;

if (m_version >= 4) {
requires_ext_adapter = m_requires_ext_adapter;
}

return requires_ext_adapter;
}

// -----------------------------------------------------------------------------
// Class: Project
// -----------------------------------------------------------------------------
Expand All @@ -193,27 +206,28 @@ Project::Project(uint32_t version)
}

Project::Project(std::string name, std::string url)
: ProjectEntry(1, name, url, false, ProjectEntryStatus::UNKNOWN, int64_t {0})
: ProjectEntry(1, name, url, false, false, ProjectEntryStatus::UNKNOWN, int64_t {0})
{
}

Project::Project(uint32_t version, std::string name, std::string url)
: ProjectEntry(version, name, url, false, ProjectEntryStatus::UNKNOWN, int64_t {0})
: ProjectEntry(version, name, url, false, false,ProjectEntryStatus::UNKNOWN, int64_t {0})
{
}

Project::Project(std::string name, std::string url, int64_t timestamp, uint32_t version)
: ProjectEntry(version, name, url, false, ProjectEntryStatus::UNKNOWN, timestamp)
: ProjectEntry(version, name, url, false, false, ProjectEntryStatus::UNKNOWN, timestamp)
{
}

Project::Project(uint32_t version, std::string name, std::string url, bool gdpr_controls)
: ProjectEntry(version, name, url, gdpr_controls, ProjectEntryStatus::UNKNOWN, int64_t {0})
: ProjectEntry(version, name, url, gdpr_controls, false, ProjectEntryStatus::UNKNOWN, int64_t {0})
{
}

Project::Project(uint32_t version, std::string name, std::string url, bool gdpr_controls, ProjectEntryStatus status)
: ProjectEntry(version, name, url, gdpr_controls, ProjectEntryStatus::UNKNOWN, int64_t {0})
Project::Project(uint32_t version, std::string name, std::string url, bool gdpr_controls,
bool requires_ext_adapter, ProjectEntryStatus status)
: ProjectEntry(version, name, url, gdpr_controls, requires_ext_adapter, ProjectEntryStatus::UNKNOWN, int64_t {0})
{
// The only two values that make sense for status using this constructor overload are MAN_GREYLISTED and
// AUTO_GREYLIST_OVERRIDE. The other are handled by the contract action context and the other overloads.
Expand All @@ -238,12 +252,12 @@ Project::Project(uint32_t version, std::string name, std::string url, bool gdpr_
}

Project::Project(uint32_t version, std::string name, std::string url, bool gdpr_controls, int64_t timestamp)
: ProjectEntry(version, name, url, gdpr_controls, ProjectEntryStatus::UNKNOWN, timestamp)
: ProjectEntry(version, name, url, gdpr_controls, false, ProjectEntryStatus::UNKNOWN, timestamp)
{
}

Project::Project(std::string name, std::string url, int64_t timestamp, uint32_t version, bool gdpr_controls)
: ProjectEntry(version, name, url, gdpr_controls, ProjectEntryStatus::UNKNOWN, timestamp)
: ProjectEntry(version, name, url, gdpr_controls, false, ProjectEntryStatus::UNKNOWN, timestamp)
{
}

Expand Down
19 changes: 16 additions & 3 deletions src/gridcoin/project.h
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ class ProjectEntry
//! ensure that the serialization/deserialization routines also handle all
//! of the previous versions.
//!
static constexpr uint32_t CURRENT_VERSION = 3;
static constexpr uint32_t CURRENT_VERSION = 4;

//!
//! \brief Version number of the serialized project format.
Expand All @@ -78,6 +78,7 @@ class ProjectEntry
uint256 m_hash; //!< The txid of the transaction that contains the project entry.
uint256 m_previous_hash; //!< The m_hash of the previous project entry with the same key.
bool m_gdpr_controls; //!< Boolean to indicate whether project has GDPR stats export controls.
bool m_requires_ext_adapter; //!< Boolean to indicate whether project requires external adapter.
CPubKey m_public_key; //!< Project public key.
Status m_status; //!< The status of the project entry. (Note serialization converts to/from int.)

Expand Down Expand Up @@ -113,10 +114,12 @@ class ProjectEntry
//! \param name. The key of the project entry.
//! \param url. The value of the project entry.
//! \param gdpr_controls. The gdpr control flag of the project entry
//! \param requires_ext_adapter. The flag that indicates whether the project requires an external adapter for stats.
//! \param status. the status of the project entry.
//! \param timestamp. The timestamp of the project entry that comes from the containing transaction
//!
ProjectEntry(uint32_t version, std::string name, std::string url, bool gdpr_controls, Status status, int64_t timestamp);
ProjectEntry(uint32_t version, std::string name, std::string url, bool gdpr_controls,
bool requires_ext_adapter, Status status, int64_t timestamp);

//!
//! \brief Determine whether a project entry contains each of the required elements.
Expand Down Expand Up @@ -184,6 +187,11 @@ class ProjectEntry
//!
std::optional<bool> HasGDPRControls() const;

//!
//! \brief Returns true if project requires an externel adapter for statistics collection.
//!
std::optional<bool> RequiresExtAdapter() const;

//!
//! \brief Comparison operator overload used in the unit test harness.
//!
Expand Down Expand Up @@ -300,7 +308,8 @@ class Project : public IContractPayload, public ProjectEntry
//! \param gdpr_controls Boolean to indicate gdpr stats export controls enforced
//! \param status ProjectEntryStatus to force project status.
//!
Project(uint32_t version, std::string name, std::string url, bool gdpr_controls, ProjectEntryStatus status);
Project(uint32_t version, std::string name, std::string url, bool gdpr_controls,
bool requires_ext_adapter, ProjectEntryStatus status);

//!
//! \brief Initialize a \c Project using data from the contract.
Expand Down Expand Up @@ -416,6 +425,10 @@ class Project : public IContractPayload, public ProjectEntry
READWRITE(m_previous_hash);
READWRITE(m_status);
}

if (m_version >= 4) {
READWRITE(m_requires_ext_adapter);
}
}
}; // Project (entry payload)

Expand Down
5 changes: 3 additions & 2 deletions src/gridcoin/scraper/scraper.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,8 @@ unsigned int SCRAPER_MISBEHAVING_NODE_BANSCORE GUARDED_BY(cs_ScraperGlobals) = 0
bool REQUIRE_TEAM_WHITELIST_MEMBERSHIP GUARDED_BY(cs_ScraperGlobals) = false;
/** Default team whitelist. Remember this will be overridden by appcache entries. */
std::string TEAM_WHITELIST GUARDED_BY(cs_ScraperGlobals) = "Gridcoin";
/** This is a short term place to hold projects that require an external adapter for the scrapers.*/
/** This is a short term place to hold projects that require an external adapter for the scrapers. This will be retired after
the block v13/superblock v3/project v4 mandatory */
std::string EXTERNAL_ADAPTER_PROJECTS GUARDED_BY(cs_ScraperGlobals) = std::string{};
/** This is the period after the deauthorizing of a scraper in seconds before the nodes will start
* to assign banscore to nodes sending unauthorized manifests.
Expand Down Expand Up @@ -6228,7 +6229,7 @@ UniValue convergencereport(const UniValue& params, bool fHelp)
for (const auto& entry : ConvergedScraperStatsCache.Convergence.vGreylistedProjects) {
ProjectEntry::Status status = GRC::EnumByte<GRC::ProjectEntryStatus>(entry.second);

ProjectEntry dummy_project(3, entry.first, "foo", false, status, 0);
ProjectEntry dummy_project(3, entry.first, "foo", false, false, status, 0);

UniValue greylisted_project_entry(UniValue::VOBJ);

Expand Down
3 changes: 3 additions & 0 deletions src/init.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -622,6 +622,9 @@ void SetupServerArgs()
// Temporary hidden option for block v13 height override to facilitate testing.
hidden_args.emplace_back("-blockv13height");

// Temporary hidden option for project v4 height override to facilitate testing.
hidden_args.emplace_back("-projectv4height");

// Temporary hidden option for superblock v3 height override to facilitate testing.
hidden_args.emplace_back("-superblockv3height");

Expand Down
4 changes: 3 additions & 1 deletion src/qt/researcher/researchermodel.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -593,7 +593,9 @@ std::vector<ProjectRow> ResearcherModel::buildProjectTable(bool extended) const
row.m_name = QString::fromStdString(project.DisplayName()).toLower();
row.m_magnitude = 0.0;

if (std::find(external_adapter_projects.begin(),
// the external adapter code below only appears here because if the project is in BOINC it does not need
// an external adapter.
if (!project.RequiresExtAdapter() && std::find(external_adapter_projects.begin(),
external_adapter_projects.end(),
project.m_name) == external_adapter_projects.end()) {
row.m_error = tr("Not attached");
Expand Down
Loading