Skip to content

[Data Liberation] Edit static files directly in a git repository #2109

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
4da97f0
Remount NODEFS and other mounts after hotswapPhpRuntime is called
adamziel Dec 23, 2024
9456b22
Preserve the /internal directory when hotSwapPhpRuntime is called
adamziel Dec 23, 2024
fc74f35
Test with 4 mounts
adamziel Dec 23, 2024
199bc5e
Local files browser in the post editor
adamziel Dec 23, 2024
e46e7e9
Add Editor UI and hooks to synchronize WordPress pages to local files
adamziel Dec 24, 2024
7b5b125
Implement Git sparse checkout in PHP
adamziel Dec 25, 2024
e821059
Plug in WP_Git_Filesystem to directly browse and edit the static file…
adamziel Dec 26, 2024
4d10e75
Prototype git push
adamziel Dec 27, 2024
45d7a88
Put all pack-related methods in WP_Git_Pack_Processor
adamziel Dec 27, 2024
0d8f88f
WP_Git_Filesystem that can read and write files from a Git repo
adamziel Dec 27, 2024
cefacc2
Tweak new file creation and metadata loading
adamziel Dec 27, 2024
18e73b8
Add specific exception for renaming directories
adamziel Dec 27, 2024
0da6447
Support renaming and deleting files
adamziel Dec 27, 2024
88b57b7
Tweak the user experience of the file picker. Enable drag&drop
adamziel Dec 27, 2024
74e671a
Replace recursive props with a context
adamziel Dec 27, 2024
7a4a0dc
Enable file uploading by drag&drop
adamziel Dec 27, 2024
e55b94a
Enable deleting file subtrees
adamziel Dec 27, 2024
718bbe6
Dsplay directories before files, use a more useful default commit
adamziel Dec 27, 2024
d9c0ead
Display a visually consistent selection in the Local Files tab without a
adamziel Dec 27, 2024
f945ec3
Use the vertical three dots icon for "more"
adamziel Dec 27, 2024
e5679d2
CSS tweaks
adamziel Dec 27, 2024
1fbfbfc
CSS tweaks
adamziel Dec 27, 2024
7e5e96a
Restrict the local files tab to the list view sidebar
adamziel Dec 28, 2024
6951723
WP_Git_Client: Use AsyncHttp\Client for HTTP requests
adamziel Dec 29, 2024
8b764c1
Fix disappearing editor, add post loading overlay
adamziel Dec 29, 2024
fdf9393
Add WP_Git_Cached_Index that stores objects in a local directory
adamziel Dec 29, 2024
63ed39a
Expand WP_Git_Cached_Index to support more operations and commits
adamziel Dec 30, 2024
59884ed
Streaming git read, ability to compute objects newly added in a commit
adamziel Dec 30, 2024
6200a4a
Implement git pull and push
adamziel Dec 30, 2024
d56adb5
Rename Git_Cached_Index to Git_Repository ane enable remote management
adamziel Dec 30, 2024
724e177
Support user.name and user.email git settings and auto push mode in t…
adamziel Dec 30, 2024
c07add9
Use serialize_blocks to store custom blocks in markdown
adamziel Dec 30, 2024
decd81e
Resolve minor papercuts to enable using the plugin with git
adamziel Dec 30, 2024
96f52a3
Store auto-saves on the disk
adamziel Dec 31, 2024
270737d
Improve Blocks/Markdown conversion
adamziel Dec 31, 2024
7bef2bc
Implement reparent, squash, amend
adamziel Dec 31, 2024
e368495
Send autosaves as commit amends
adamziel Dec 31, 2024
1676afa
Drag and drop from the editor to desktop
adamziel Jan 1, 2025
d33b84f
Preview static files
adamziel Jan 1, 2025
620755d
Only force pull at most once every 10 minutes, exclude .autosaves fro…
adamziel Jan 1, 2025
8594402
Reduce waiting in the UI – server provides each page's ID, the editor…
adamziel Jan 1, 2025
6956b58
Import attachments on boot and on upload
adamziel Jan 1, 2025
b2755d7
Suport UTF-8 filenames in download_file_endpoint
adamziel Jan 1, 2025
a9202bb
Rewrite WordPress URLs to relative URLs when serializing to markdown
adamziel Jan 2, 2025
802a6f0
Preserve images as blocks
adamziel Jan 2, 2025
875c89a
Upload media files to the static files filesystem
adamziel Jan 2, 2025
0077659
Use the correct paths in wp_handle_upload
adamziel Jan 2, 2025
11638c6
Delete files when backspace is pressed
adamziel Jan 2, 2025
a11d520
Store updated media files when they're, e.g., rotated
adamziel Jan 2, 2025
34e08ff
Media files preview
adamziel Jan 2, 2025
86ab57d
Derive WP_Block_Markup_Processor from WP_HTML_Tag_Processor, not WP_H…
adamziel Jan 2, 2025
6be33e3
Preserve the correct last href in Blocks -> Markdown conversion
adamziel Jan 2, 2025
1f27630
Don't explicitly force pull after writes
adamziel Jan 3, 2025
2ad968b
Rudimentary commit diffing and text diffing
adamziel Jan 3, 2025
12c3fb2
Clarify a TODO comment
adamziel Jan 3, 2025
b13b833
Implement three way merge
adamziel Jan 3, 2025
65c73bb
Support for the HTML format and changing file extensions
adamziel Jan 4, 2025
6357abd
Start implementing a git server
adamziel Jan 5, 2025
dd73f79
Parse Git Client messages
adamziel Jan 5, 2025
bb3928e
Add unit tests
adamziel Jan 5, 2025
b28b46a
Functional fetch support
adamziel Jan 6, 2025
4d90846
Implement git push in WP_Git_Server
adamziel Jan 6, 2025
32af23d
Remove debug comment
adamziel Jan 6, 2025
122fe94
Add TODO
adamziel Jan 6, 2025
149f8b4
git fetch WordPress pages
adamziel Jan 6, 2025
75f7601
Add readme for the git repo plugin
adamziel Jan 6, 2025
9b09454
Add a todo
adamziel Jan 6, 2025
8dd84b3
git push to update content
adamziel Jan 6, 2025
2d5da28
When fetching, fetch all the relevant objects without leaving out any
adamziel Jan 7, 2025
d4afa36
Comment out the failure in find_objects_added_in
adamziel Jan 7, 2025
bbea7ae
Handle content deletions in the git endpoint
adamziel Jan 8, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Parse Git Client messages
  • Loading branch information
adamziel committed Jan 5, 2025
commit dd73f79305ab01f11b241524ea97334f11bfb61e
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,18 @@ static private function wrap_object($type, $object) {
return "$type_name $length\x00" . $object;
}

static public function encode_packet_lines(array $payloads): string {
$lines = [];
foreach($payloads as $payload) {
if($payload === '0000' || $payload === '0001' || $payload === '0002') {
$lines[] = $payload;
} else {
$lines[] = self::encode_packet_line($payload);
}
}
return implode('', $lines);
}

static public function encode_packet_line(string $payload): string {
$length = strlen($payload) + 4;
return sprintf("%04x", $length) . $payload;
Expand Down Expand Up @@ -239,7 +251,7 @@ static public function decode_next_packet_line($pack_bytes, &$offset) {
case '0002':
return ['type' => '#response-end'];
default:
$length = intval($packet_length_bytes, 16);
$length = intval($packet_length_bytes, 16) - 4 ;
$payload = substr($pack_bytes, $offset, $length);
if(str_ends_with($payload, "\n")) {
$payload = substr($payload, 0, -1);
Expand Down
176 changes: 122 additions & 54 deletions packages/playground/data-liberation/src/git/WP_Git_Server.php
Original file line number Diff line number Diff line change
Expand Up @@ -21,15 +21,45 @@ public function __construct(WP_Git_Repository $repository) {
/**
* Handle Git protocol v2 ls-refs command
*
* ls-refs is the command used to request a reference advertisement in v2.
* Unlike the current reference advertisement, ls-refs takes in arguments
* which can be used to limit the refs sent from the server.
*
* Additional features not supported in the base command will be advertised as
* the value of the command in the capability advertisement in the form of a space
* separated list of features: "<command>=<feature 1> <feature 2>"
*
* ls-refs takes in the following arguments:
*
* symrefs
* In addition to the object pointed by it, show the underlying ref
* pointed by it when showing a symbolic ref.
*
* peel
* Show peeled tags.
*
* ref-prefix <prefix>
* When specified, only references having a prefix matching one of
* the provided prefixes are displayed. Multiple instances may be
* given, in which case references matching any prefix will be
* shown. Note that this is purely for optimization; a server MAY
* show refs not matching the prefix if it chooses, and clients
* should filter the result themselves.
*
* unborn
* The server will send information about HEAD even if it is a symref
* pointing to an unborn branch in the form "unborn HEAD symref-target:<target>".
*
* @see https://git-scm.com/docs/protocol-v2#_ls_refs
* @param array $request The parsed request data
* @return string The response in Git protocol v2 format
*/
public function handle_ls_refs_request($request) {
$parsed = $this->parse_ls_refs_request($request);
$parsed = $this->parse_message($request);
if(!$parsed) {
return false;
}
$prefix = $parsed['arguments']['ref-prefix'] ?? '';
$prefix = $parsed['arguments']['ref-prefix'][0] ?? '';

$refs = $this->repository->list_refs($prefix);
$response = '';
Expand Down Expand Up @@ -75,60 +105,29 @@ public function capability_advertise() {
"0000";
}

/**
* ls-refs is the command used to request a reference advertisement in v2.
* Unlike the current reference advertisement, ls-refs takes in arguments
* which can be used to limit the refs sent from the server.
*
* Additional features not supported in the base command will be advertised as
* the value of the command in the capability advertisement in the form of a space
* separated list of features: "<command>=<feature 1> <feature 2>"
*
* ls-refs takes in the following arguments:
*
* symrefs
* In addition to the object pointed by it, show the underlying ref
* pointed by it when showing a symbolic ref.
*
* peel
* Show peeled tags.
*
* ref-prefix <prefix>
* When specified, only references having a prefix matching one of
* the provided prefixes are displayed. Multiple instances may be
* given, in which case references matching any prefix will be
* shown. Note that this is purely for optimization; a server MAY
* show refs not matching the prefix if it chooses, and clients
* should filter the result themselves.
*
* unborn
* The server will send information about HEAD even if it is a symref
* pointing to an unborn branch in the form "unborn HEAD symref-target:<target>".
*
* @see https://git-scm.com/docs/protocol-v2#_ls_refs
*
* @param string $request_bytes Raw request bytes
* @return array Parsed request data
*/
public function parse_ls_refs_request($request_bytes) {
$parsed = [
'capabilities' => [],
'arguments' => [],
];

// Parse the capability advertisement part
public function parse_message($request_bytes) {
$offset = 0;
return [
'capabilities' => $this->parse_capabilities($request_bytes, $offset),
'arguments' => $this->parse_arguments($request_bytes, $offset),
];
}

private function parse_capabilities($request_bytes, &$offset=0) {
$capabilities = [];
while (true) {
$line = WP_Git_Pack_Processor::decode_next_packet_line($request_bytes, $offset);
if ($line === false || $line['type'] !== '#packet') {
break;
}

list($key, $value) = explode('=', $line['payload']);
$parsed['capabilities'][$key] = $value;
$capabilities[$key] = $value;
}
return $capabilities;
}

// Parse the optional arguments part
private function parse_arguments($request_bytes, &$offset=0) {
$arguments = [];
while (true) {
$line = WP_Git_Pack_Processor::decode_next_packet_line($request_bytes, $offset);
if ($line === false || $line['type'] !== '#packet') {
Expand All @@ -137,15 +136,84 @@ public function parse_ls_refs_request($request_bytes) {

$space_at = strpos($line['payload'], ' ');
if($space_at === false) {
$parsed['arguments'][$line['payload']] = true;
continue;
$key = $line['payload'];
$value = true;
} else {
$key = substr($line['payload'], 0, $space_at);
$value = substr($line['payload'], $space_at + 1);
}
$key = substr($line['payload'], 0, $space_at);
$value = substr($line['payload'], $space_at + 1);
$parsed['arguments'][$key] = $value;
}

return $parsed;
if(!array_key_exists($key, $arguments)) {
$arguments[$key] = [];
}
$arguments[$key][] = $value;
}
return $arguments;
}

/**
* Handle Git protocol v2 fetch command with "want" packets
*
* @param array $request The parsed request data
* @return string The response in Git protocol v2 format containing the pack data
*/
public function handle_fetch_request($request) {
$parsed = $this->parse_message($request);
if (!$parsed || empty($parsed['arguments']['want'])) {
return false;
}

$objects_to_send = [];
foreach ($parsed['arguments']['want'] as $want_hash) {
// For each wanted commit, find objects not present in any of the have commits
$new_objects = $this->repository->find_objects_added_in(
$want_hash,
$parsed['arguments']['have'] ?? ['0000000000000000000000000000000000000000']
);
$objects_to_send = array_merge($objects_to_send, $new_objects);
}
$objects_to_send = array_unique($objects_to_send);

// Pack the objects
$pack_objects = [];
foreach ($objects_to_send as $oid) {
$this->repository->read_object($oid);

// Apply blob filters if specified
if ($this->repository->get_type() === WP_Git_Pack_Processor::OBJECT_TYPE_BLOB) {
$filter = $parsed['arguments']['filter'] ?? null;
if ($filter) {
if ($filter['type'] === 'none') {
continue; // Skip all blobs
} else if ($filter['type'] === 'limit') {
$content = $this->repository->read_entire_object_contents();
if (strlen($content) > $filter['size']) {
continue; // Skip large blobs
}
}
}
}

$pack_objects[] = [
'type' => $this->repository->get_type(),
'content' => $this->repository->read_entire_object_contents(),
];
}

// Handle deepen if specified
if (isset($parsed['arguments']['deepen'])) {
// @TODO: Implement history truncation based on deepen value
// This would involve walking the commit history and including
// only commits within the specified depth
}

// Encode the pack
$pack_data = WP_Git_Pack_Processor::encode($pack_objects);

// Format the response according to protocol v2
return
WP_Git_Pack_Processor::encode_packet_line("packfile\n") .
WP_Git_Pack_Processor::encode_packet_line("\x01" . $pack_data) . // side-band channel 1
"0000";
}
}
Loading