Markdown in Drummer? #4

scripting · 2021-11-02T14:36:01Z

Just posted a change note about Markdown in Drummer. This is a place to discuss.

sabre23t · 2021-11-04T12:34:33Z

I'm not sure what "Markdown outline" features you are looking for Drummer, Dave. However, I know Logseq have export/import OPML features that could be used as benchmarks/samples.

Logseq Markdown outline flavour is quite unique to it, while being valid standard GFM Markdown. AFAIK, Obsidian doesn't natively support outline, instead relies on an outline plugin for that Markdown outline feature.

Key issue is Markdown have two main hierarchical constructs that can be used as outline hierarchy. The "#" headings and "-" unordered list items hierarchy. Originally Logseq used "#" exclusively for outline hierarchy. Now it use "-" as main outline hierarchy with some allowance to use "#" as top level outline headings (summit headlines).

I hope that's of some use. :-)

scripting · 2021-11-04T12:45:41Z

@sabre23t -- yes that is helpful. i'd like to be able to import LogSeq outlines in Markdown format and also export them. It would be helpful to have a few example files. Actual files exported from LogSeq. Could you paste the text from them into comments here?

sabre23t · 2021-11-04T13:54:30Z

Lets start with roundtripping your OPML spec example states.opml. Logseq v0.4.4 import that states.opml into states.opml.md with text contents as below.

title:: states.opml

- United States
	- Far West
		- Alaska
		- California
		- Hawaii
		- Nevada
		- Oregon
		- Washington
	- Great Plains
		- Kansas
		- Nebraska
		- North Dakota
		- Oklahoma
		- South Dakota
	- Mid-Atlantic
		- Delaware
		- Maryland
		- New Jersey
		- New York
		- Pennsylvania
		  id:: 614800f3-e336-45e5-849e-fe64ea361d35
	- Midwest
		- Illinois
		- Indiana
		- Iowa
		- Kentucky
		- Michigan
		- Minnesota
		- Missouri
		- Ohio
		- West Virginia
		- Wisconsin
	- Mountains
		- Colorado
		- Idaho
		- Montana
		- Utah
		- Wyoming
	- New England
		- Connecticut
		- Maine
		- Massachusetts
		- New Hampshire
		- Rhode Island
		- Vermont
	- South
		- Alabama
		- Arkansas
		- Florida
		- Georgia
		- Louisiana
		- Mississippi
		- North Carolina
		- South Carolina
		- Tennessee
		- Virginia
	- Southwest
		- Arizona
		- New Mexico
		- Texas
- United States
	- Far West
		- Alaska
		- California
		- Hawaii
		- Nevada
			- Reno
			- Las Vegas
			- Ely
			- Gerlach
		- Oregon
		- Washington
	- Great Plains
		- Kansas
		- Nebraska
		- North Dakota
		- Oklahoma
		- South Dakota
	- Mid-Atlantic
		- Delaware
		- Maryland
		- New Jersey
		- New York
		- Pennsylvania
	- Midwest
		- Illinois
		- Indiana
		- Iowa
		- Kentucky
		- Michigan
		- Minnesota
		- Missouri
		- Ohio
		- West Virginia
		- Wisconsin
	- Mountains
		- Colorado
		- Idaho
		- Montana
		- Utah
		- Wyoming
	- New England
		- Connecticut
		- Maine
		- Massachusetts
		- New Hampshire
		- Rhode Island
		- Vermont
	- South
		- Alabama
		- Arkansas
		- Florida
		- Georgia
		- Louisiana
		- Mississippi
		- North Carolina
		- South Carolina
		- Tennessee
		- Virginia
	- Southwest
		- Arizona
		- New Mexico
		- Texas
-

That "id:: 614800f3-e336-45e5-849e-fe64ea361d35" is an artifact of my Logseq graph (database) having a block reference to an item called "Pennsylvania". In general you see block level (headlines level) attributes are dropped. And only "title::" page level (head level) attribute was maintained.

Lets now round trip it back into OPML by exporting from states.opml.md page in Logseq v0.4.4 into OPML. The text contents as below:

<?xml version="1.0" encoding="UTF-8"?>
<opml version="2.0">
  <head>
    <title>
      untitled
    </title>
  </head>
  <body>
    <outline text=""/>
    <outline text="United States">
      <outline text="Far West">
        <outline text="Alaska"/>
        <outline text="California"/>
        <outline text="Hawaii"/>
        <outline text="Nevada"/>
        <outline text="Oregon"/>
        <outline text="Washington"/>
      </outline>
      <outline text="Great Plains">
        <outline text="Kansas"/>
        <outline text="Nebraska"/>
        <outline text="North Dakota"/>
        <outline text="Oklahoma"/>
        <outline text="South Dakota"/>
      </outline>
      <outline text="Mid-Atlantic">
        <outline text="Delaware"/>
        <outline text="Maryland"/>
        <outline text="New Jersey"/>
        <outline text="New York"/>
        <outline text="Pennsylvania"/>
      </outline>
      <outline text="Midwest">
        <outline text="Illinois"/>
        <outline text="Indiana"/>
        <outline text="Iowa"/>
        <outline text="Kentucky"/>
        <outline text="Michigan"/>
        <outline text="Minnesota"/>
        <outline text="Missouri"/>
        <outline text="Ohio"/>
        <outline text="West Virginia"/>
        <outline text="Wisconsin"/>
      </outline>
      <outline text="Mountains">
        <outline text="Colorado"/>
        <outline text="Idaho"/>
        <outline text="Montana"/>
        <outline text="Utah"/>
        <outline text="Wyoming"/>
      </outline>
      <outline text="New England">
        <outline text="Connecticut"/>
        <outline text="Maine"/>
        <outline text="Massachusetts"/>
        <outline text="New Hampshire"/>
        <outline text="Rhode Island"/>
        <outline text="Vermont"/>
      </outline>
      <outline text="South">
        <outline text="Alabama"/>
        <outline text="Arkansas"/>
        <outline text="Florida"/>
        <outline text="Georgia"/>
        <outline text="Louisiana"/>
        <outline text="Mississippi"/>
        <outline text="North Carolina"/>
        <outline text="South Carolina"/>
        <outline text="Tennessee"/>
        <outline text="Virginia"/>
      </outline>
      <outline text="Southwest">
        <outline text="Arizona"/>
        <outline text="New Mexico"/>
        <outline text="Texas"/>
      </outline>
    </outline>
    <outline text="United States">
      <outline text="Far West">
        <outline text="Alaska"/>
        <outline text="California"/>
        <outline text="Hawaii"/>
        <outline text="Nevada">
          <outline text="Reno"/>
          <outline text="Las Vegas"/>
          <outline text="Ely"/>
          <outline text="Gerlach"/>
        </outline>
        <outline text="Oregon"/>
        <outline text="Washington"/>
      </outline>
      <outline text="Great Plains">
        <outline text="Kansas"/>
        <outline text="Nebraska"/>
        <outline text="North Dakota"/>
        <outline text="Oklahoma"/>
        <outline text="South Dakota"/>
      </outline>
      <outline text="Mid-Atlantic">
        <outline text="Delaware"/>
        <outline text="Maryland"/>
        <outline text="New Jersey"/>
        <outline text="New York"/>
        <outline text="Pennsylvania"/>
      </outline>
      <outline text="Midwest">
        <outline text="Illinois"/>
        <outline text="Indiana"/>
        <outline text="Iowa"/>
        <outline text="Kentucky"/>
        <outline text="Michigan"/>
        <outline text="Minnesota"/>
        <outline text="Missouri"/>
        <outline text="Ohio"/>
        <outline text="West Virginia"/>
        <outline text="Wisconsin"/>
      </outline>
      <outline text="Mountains">
        <outline text="Colorado"/>
        <outline text="Idaho"/>
        <outline text="Montana"/>
        <outline text="Utah"/>
        <outline text="Wyoming"/>
      </outline>
      <outline text="New England">
        <outline text="Connecticut"/>
        <outline text="Maine"/>
        <outline text="Massachusetts"/>
        <outline text="New Hampshire"/>
        <outline text="Rhode Island"/>
        <outline text="Vermont"/>
      </outline>
      <outline text="South">
        <outline text="Alabama"/>
        <outline text="Arkansas"/>
        <outline text="Florida"/>
        <outline text="Georgia"/>
        <outline text="Louisiana"/>
        <outline text="Mississippi"/>
        <outline text="North Carolina"/>
        <outline text="South Carolina"/>
        <outline text="Tennessee"/>
        <outline text="Virginia"/>
      </outline>
      <outline text="Southwest">
        <outline text="Arizona"/>
        <outline text="New Mexico"/>
        <outline text="Texas"/>
      </outline>
    </outline>
    <outline text=""/>
  </body>
</opml>

Logseq forgets to export the "titlle::" attribute. There's also the erroneous additional blank headlines at the start and end of the outline.

Is that still useful? Is there more complex OPML that you'd like to see being round tripped? Is there a better way to show MD/OPML code blocks in Github comments?

scripting · 2021-11-13T13:45:20Z

I'm starting to experiment with code that supports this format.

A couple of questions.

Is indentation done with tabs?
At the end of a line is there a single \n character?

I hope to have something for you to look at soon. ;-)

scripting · 2021-11-13T15:59:49Z

Another question --

The states outline only has one summit.

Can these outlines have more than one summit?

Drummer outlines can have multiple summits.

scripting · 2021-11-13T16:31:18Z

Okay -- here's the first result.

https://github.com/scripting/tmp1/blob/main/demo1.md

That's states.opml as exported by Drummer using the format supported by various tools for thought apps (TFSBVTFTA).

The format needs a catchier name. 😀

sabre23t · 2021-11-14T03:40:03Z

Dave wrote:

The states outline only has one summit.
Can these outlines have more than one summit?

I see that http://hosting.opml.org/dave/spec/states.opml have one summit. The earlier import/export test shows two summits. Logseq does support one or more summits.

I ran the OPML import test again on Logseq v0.4.5 with current content of states.opml (one summit).
states.opml import.txt

Caveat importing the same OPML multiple times will append to previous imports in Logseq.
Have to delete previous OPML imports before reimporting.

It displays correctly in Logseq as one summit but didn't import all attributes. It saves that Logseq page as states.opml.md .
states.opml.md

the created attributes for headlines/summits were not imported.
the opml head attributes were not imported other than the title attribute.

I then export that into OPML again.
states.opml export.txt

the opml head attribute was not exported, it became "untitled".
extraneous empty initial headline/summit

sabre23t · 2021-11-14T04:25:04Z

Dave wrote:

Is indentation done with tabs?

Yes. Logseq MD file states.opml.md use tab (\t) for each level indentation.

At the end of a line is there a single \n character?

Yes. Logseq MD file states.opml.md use Unix LF (\n) for end of line. Logseq will accept MD file with Windows CRLF (\r\n) for end of line, but on edit will resave as Unix LF format.

sabre23t · 2021-11-14T04:53:49Z

Dave wrote:

Okay -- here's the first result.
https://github.com/scripting/tmp1/blob/main/demo1.md

Noted this is MD file using tab and Unix LF format. Logseq has no import command for MD files. You just copy the MD file into Logseq pages folder and do a "Refresh (import changes from local files)". Doing that gets me this screenshot in Logseq. I also note:

The "created" attributes are displayed and editable in Logseq as custom attributes.
It display as "created:" normally and as "created::" when editing.
Currently Logseq v0.4.5 doesn't use timestamp such as "created" in their MD files. I understand the developers are still exploring options on how to timestamp each headline/blocks in MD files.

Upon edit and resave by Logseq the MD was changed a bit.
drummer tmp1 demo1 x.md.txt

There is additional blank line after the first attribute
There are additional tab & space chars before the created: attributes.

scripting · 2021-12-30T16:57:31Z

@sabre23t -- more than a month later ;-)

I'm working on making Electric Drummer work with LogSeq markdown files, and in doing so hope to gain interop with Obsidian and perhaps other products.

The way it works is that when E/D opens a file, I look at the extension. If it's .opml, i proceed as usual.

But if the extension is .md -- first I convert it to OPML and pass it on up. At that point Drummer doesn't know it came from a markdown file. It works. Below is a screen shot of an outline i created in Logseq and opened in E/D.

When saving it will reverse the process if the file has a .md extension. Not sure how much fidelity will be lost. Just beginning down the path, haven't implemented attributes yet, this is just Day 1 of this adventure. 😄

sabre23t · 2021-12-31T10:10:45Z

That's great @scripting 😄
I guess in web drummer you should be able to use "File > Import local file..." to import MD files in the future. It doesn't yet work in web drummer v2.0.14, I tried.

scripting · 2021-12-31T16:16:15Z

Here's a demo of two-way interop.

I've read in an outline from LogSeq. Edited it. It autosaves, the change is immediately reflected in LS.

https://www.youtube.com/watch?v=RHs6tqookNc

scripting · 2021-12-31T17:02:44Z

So here's what the outline looks like in LS after I added attribute support.

I had to take out head-level atts -- they were too unsightly.

scripting · 2021-12-31T17:04:35Z

How to create attributes in LogSeq?

Suppose I want to add an attribute to Oregon that says what its average temperature is. What is the UI for that within LogSeq?

scripting · 2021-12-31T18:40:51Z

OK -- I think I figured out how this should work.

When I read a .md file into Drummer, any attributes I find are included in the Drummer outline with a leading underscore in its name.
When I save a .md file, we only include the atts with a leading underscore, which is deleted before writing.

With this you can use Drummer to add atts to a LogSeq outline. I no longer need to know how to create an attribute in LS.

What this will be useful for is using Drummer to edit a LS outline.

It might also be possible to incorporate a LS outline in a Drummer project, I have to learn more about editing in LS to figure out how that might work.,

scripting · 2022-01-01T22:54:52Z

A podcast about this project...

http://scripting.com/2022/01/01/thoughtsOnInteropInOutlineLandBasedOnMarkdown.m4a

sabre23t · 2022-01-03T03:27:20Z

An attribute/propery in logseq is just part of the node/block. It's the first few lines of a block [1], no specific UI [2]. It's any name ending with "::" followed by one or more values. Ref https://docs.logseq.com/#/page/term%2Fproperties .

You can hide any of your properties that you don't want to see in Logseq display mode. You'll only see those properties when you edit the block. Set it by editing config.edn to include lines similar to this.

 ;; hide specific properties for blocks
 ;; E.g. #{:created-at :updated-at}
 :block-hidden-properties #{:created}

Edit:
[1] Actually you can intersperse the property lines anywhere in the block. Just tested found that to be true. As long as the property line is of the format property:: value, value2, etc
[2] No specific UI, but user need to remember to use shift-enter to include new lines in a block. The properties are on separate lines from contents.

scripting · 2022-01-03T05:11:18Z

When I saved my drummer outline with all its attributes the were all shown visibly in logseq. I gather from what you’re saying is I can set a pref to disable this.

sabre23t · 2022-01-03T08:48:04Z

Yes. Logseq ellipsis ... > Settings command now allows direct editing of config.edn file via code mirror. No longer have to do it in external editor. The settings for block-hidden-properties is applicable per logseq graph/repository.

Visible block properties is useful for properties like reference number, phone number etc.
Hidden block properties is useful for properties only useable to programs/scripts such as long hash number, UUID etc.

Logseq does not store all its block related properties in the MD file. It stores only permanent properties/information in the MD file. If you enable "Developer mode" in Settings, Logseq provides (Dev) Show block data option when you right click on the block bullets. Similar (Dev) Show page data when you right click on the page title. The example below shows data for a block in my logseq repo.

{:block/uuid #uuid "61d279b6-36bf-465b-93be-be22fa6f8e45",
 :block/properties {:collapsed true},
 :block/left {:db/id 9093},
 :block/format :markdown,
 :block/content "collapsed:: true\n2. Traffic Study",
 :db/id 9095,
 :block/path-refs [{:db/id 1202} {:db/id 9084}],
 :block/parent {:db/id 9093},
 :block/unordered true, 
 :block/page {:db/id 9084}}

scripting · 2022-01-03T11:40:29Z

I’m just trying to make drummer interop with it.

Is any meaningful interop possible by reading and writing the .md file?

I’m not asking anyone wanting to interop with drummer to understand it’s internal workings.

sabre23t · 2022-01-03T14:56:46Z

Reading and writing the MD files from Drummer could surely allow meaningful interop. Providing both reading/writing operations on MD files from Drummer side does allow both way transfers/interop with other system using MD files.

Conversely, Logseq also provides both way operations on OPML that could be basis for interop. You can export page/graph to OPML. You can also import OPML files. Unfortunately as of now Logseq OPML transformation is very lossy. Many attributes are lost, even title becomes "untitled".

Doing the Logseq/Drummer interop on Drummer side via MD, with better and more complete transformation of features in OPML to/from MD would surely show the way forward. To do so I think the following key features should be prioritised:

Transformation of Logseq multiline outline nodes/blocks to/from Drummer singleline (one paragraph) nodes/headlines
Conversion of Logseq block-level "collapsed:: true" attributes to/from Drummer head-level (page-level) "<expansionState>" attributes.
Conversion of other common attributes/properties should also be done: eg: "Title", "Created".
Attributes/properties unique/unknown to Logseq/Drummer should be passed through unchanged, not silently dropped.
Looking at Dave's video on two way interop https://www.youtube.com/watch?v=RHs6tqookNc , I see that outline structure conversion between OPML/MD looks good already.

Anyway, that's just my2cents. Look forward to test such markdown two-way interop when you release the feature to us Drummer users. 😄

scripting · 2022-01-03T15:11:32Z

Okay I suggest we proceed as follows:

I will publish the source for my reading and writing of MD files from Drummer's OPML structures including example code. Right now my plan is to add this to the OPML package for JavaScript.
I will include this functionality at least in Electric Drummer. (Note is that a problem for @sabre23t?)
Then we will have a codebase to work on. You will have a way to direct this work without having to make me understand everything that's going on, because right now I am not a user of LS and you are, and they are not users of Drummer, and you are., You and other users will imho be key to getting this to work in a way that makes sense and give us advice on how to proceed.

Does that make sense?

sabre23t · 2022-01-03T15:50:53Z

Yes, that does make sense. My main machines are Windows based. If you could put some of the functionality to test into Web Drummer, even if it's just in the DW menu, I would appreciate it. 😄

I understand Electric Drummer is Mac only at the moment. If I really needed to use a Mac, I think I could borrow some time on my daughter's Macbook Air. But I'm totally a newbie on macOS.

Currently I'm a daily user of Drummer for public blogging. And daily user of Logseq for private journaling & knowledge base. Look forward to test the interops between them.

scripting · 2022-01-03T17:39:40Z

@sabre23t -- that's why I asked. Something you posted earlier implied you didn't use E/D, that's why I wanted to check. I'll look into how much work it would be to support it in web Drummer too.

scripting · 2022-01-03T23:37:23Z

The first part of the plan is done, v0.4.12, of the OPML package that reads and writes the extended Markdown used by Tools For Thought products such as LogSeq.

Here's the code, as you can see it's not very complicated.

andrewshell · 2022-01-05T23:52:33Z

I've created a small project that takes a sample LogSeq markdown file and converts it to JSON, OPML, and a new markdown file and compares it with the original. https://github.com/andrewshell/test-opml-md

At the top of the markdown file, there can be page-level attributes that should be stored as OPML head attributes
Attributes under a headline have tabs for the indenting, but then two spaces so they align with the text of the headline.
- You are not removing the spaces so an attribute like collapsed:: true turns into _ collapsed="true" in the OPML which is not valid

scripting · 2022-01-06T00:17:43Z

@andrewshell -- thanks for posting these notes here.

@sabre23t -- likely will see them overnight, he's in Malaysia I believe. On the other side of the world.

Andrew has been reviewing the new OPML package as it's been coming together, and now it's time to connect that to Drummer, as you'll see in the next comment.

scripting · 2022-01-06T00:21:05Z

I just made the changes to web Drummer so that it can now read and write .md files in addition to .opml files.

It's version 2.0.15.

What this means --

You can create a .md file. Choose New from the File menu. Instead of entering states.opml, enter states.md.
When the file is saved, we generate the markdown equivalent of the outline, using the new code in the OPML package.
You can open files with the .md extension using the File/Open command.

In theory you should be able to make the files public, and then you can probably figure out how to get LogSeq to read it.

I am finished for the day, unless there's an emergency (like I broke something in these changes, it's been known to happen).

scripting · 2022-01-06T00:22:54Z

I was able to make my outline public.

http://drummer.scripting.com/cluelessnewbie/states.md

I'm now going to try to fill it with the example outline from the OPML package.

scripting · 2022-01-08T16:25:04Z

Here's the blog post.

http://scripting.com/2022/01/08/153904.html?title=whyLogseq

In conclusion -- we are not supporting head-level attributes in the Markdown format at this time.

Too much confusion. I want to try to find meaningful interop with the support we have now. I think we will find there is quite a bit.

scripting · 2022-01-08T16:36:28Z

OK, here's a question for the LogSeq users here.

I understand how the daily journal feature works.

If I quit and relaunch LS it opens up a new page for today.

There's a folder in my LS folder called Journals. All those files end up there.

I can easily set it up so the contents of a Drummer calendar is mapped to their journal structure.

Finally -- here's the question.

What features hang off the journal structure in LS?

I can tell you what hangs off it in Drummer -- the blogging system. Also Daytona. Haha this would be very useful I think.

With this you could use LS to author a Drummer blog.

What about in the other direct, what good things will LogSeq do with the journals? I assume they are in a graph of some kind.

Another thing is I could use Drummer as my LogSeq editor, which is something my mind would like. Drummer's outliner works much better for me.

sabre23t · 2022-01-09T04:40:33Z

Dave wrote:

What features hang off the journal structure in LS?

As I understand it, Logseq journal per day page/file structure serves quite similar purpose to what calendarDay & calendarMonth node types do in Drummer blog.opml. It groups and dates all the nodes within it. Drummer use it for publishing to OldSchool.

Logseq mainly use journal pages as dates for its todo tasks management. Any nodes in journal pages are potential tasks that are made so by prefixing with markers: TODO, NOW, DOING, DONE, LATER etc. Logseq default journal template queries/display NOW/DOING tasks for previous 14days, on current day view. This is alternative to GFM tasklist markers: [ ] & [x].

With this you could use LS to author a Drummer blog.

Yes, I would certainly love to be able to do this. I'd setup a second Logseq repo with all public info (daily blogs and named pages) that I'd want published in a Drummer blog.

I'm currently doing copy & paste sub-outlines from my private Web Logseq repo to my public Web Drummer blog . Unfortunately this does not fully copy in attributes (bold, italic, external links etc), or the sub-outline structure. I tried to copy this Logseq sub-outline:

I used Logseq's 3 "Copy as" options OPML/TextMD/HTML to copy & paste in a Drummer blog post:

OPML - May be Logseq didn't set some clipboard settings so Drummer didn't understand the paste as OPML (instead of text). Also Logseq didn't translate MD into HTML for inline italic, bold and link tags.
Text MD - Copies the structure correctly, but MD remains as text in Drummer. I manually convert them to HTML markup needed in Drummer.
HTML - Seems Drummer strips all the HTML tags for some reason.

I would love to see at least this use case be better.

emk · 2022-01-09T05:29:28Z

@scripting Hello! It's been a while. I've been using LogSeq a bit lately because I wanted a good Linux desktop outliner. That's been one of my holy grails for a long time now.

What features hang off the journal structure in LS?

The big thing that I use the journal for is to "narrate my work" for the day. I write down a list of things to do. Some headings becomes TODO check boxes. Other headings become notes. There's no obvious way to share my work journal with my colleagues, like people used to do with UserLand Frontier. But I can rearrange and update it as I go, keeping a record of my mental context.

Then the next day, I review yesterday and starting writing down a plan for today. I use it as a personal productivity tool, instead of as a blog.

I have two plugins installed to help with the journal:

One plugin allows me to convert any subheading into a top-level Markdown outline. I use this when I want some part of my daily notes to become a permanent document. Just select the heading and run a command. Done!
The other plugin shows me a calendar with all my journal entries.

The journal is a pretty nice feature. It's so simple. Basically, LogSeq treats "narrate your work" as the center of the product, and everything grows around it. It's an interesting bet.

I understand that if I create a second LogSeq folder, I can have a second journal. So blogging might be another possibility.

what i wanted right now, is validation that head-level atts make sense in LS.

I've seen those YAML front matter attributes before. There are a whole bunch of static blogging packages, mostly derived from or inspired by Jekyll. They tend to use front matter like this:

---
title: "My Blog Post"
description: META tag stuff for Google
tags: outlining,drummer
date: ...
published: false
---

Post body is Markdown.

I think GitHub even detects this front-matter and renders it as a table.

I don't have a concrete interop use case for front matter right now, but I wouldn't be surprised to run into one, either. I guess the biggest challenge is that everyone uses front matter for something, but it tends to be application specific.

Oh, and one final thought after using LogSeq a bit: It's amazing how often I want to import data into the system, or export it out to something else. A lot of my LogSeq outlines will eventually turn into regular Markdown files that get committed to a shared git repository. Or they get converted to a post on Slack. Markdown is everywhere these days. This kind of interop is a major selling feature, and one of the things that your outlining tools were always great at, thanks to the built-in scripting.

scripting · 2022-01-09T14:50:00Z

@emk -- wow so good to hear from you, especially in this context.

there are so many ways we can add value to what you're doing in LogSeq, esp in re GitHub.

i have a new "product" almost ready to go, something i've been using in Frontier for a number of years, that allows me to edit a project in an outline and distribute subs to various different places on github, s3, the local file system. it's how i have been doing all my dev work since 2013 or so, and it runs in Drummer. it also does markdown really well, and javascript, and of course outlines.

i know about yaml. i did a project based on a wizzy github editor i did a few years back. so i have working yaml code in my toolkit. but who needs another format for this! oy. will we never just build on what we already have.

anyway this is going to be fun. 😄

scripting · 2022-01-09T14:54:59Z

@sabre23t -- i think we found our first project. mapping a LogSeq journal to a Drummer blog. Yes? I think you said you'd love to get that working. Me too. I think that's a great first project.

here's a question on how we'll do it -- how can we access your logseq files via web drummer without you having to import them one at a time. we want something that automatically updates. if you were using electric drummer it would be easy.

can you mirror a logseq journal anywhere that's publicly accessible on the web??

btw we got really lucky that @emk is in the loop (see above). we've worked together very effectively many times.

scripting · 2022-01-09T15:26:27Z

@emk -- here's a complete list of the verb set in Drummer, to give you a rough idea of what it connects to now.

https://github.com/scripting/docServer/blob/main/docs/readme.md

scripting · 2022-01-09T16:52:44Z

@sabre23t and others -- here's my plan for the first rev of the LogSeq Journal ==> Drummer blog connection.

a drummer command that reads the logseq journal folder,
loads the structure into an opml file,
saves it,
pings drummercms.scripting.com to build the blog

emk · 2022-01-09T19:23:22Z

can you mirror a logseq journal anywhere that's publicly accessible on the web??

There's a menu item in LogSeq that said Make it public for publishing, but all it did was set a flag marked public:: true. I guess this doesn't really work in their desktop app yet? Or there's some other tool this is intended to work with.

I also turned on Settings > Developer Mode, which gave me an experimental **Plugins..." menu item. And I looked through the plugin marketplace. There are plugins to export to Todoist and to some habit tracking tool. There are also plugins to import outlines from various places. The plugins look like simple JavaScript, so there's probably some long-term potential here for LogSeq users to build useful interop tools.

there are so many ways we can add value to what you're doing in LogSeq, esp in re GitHub.

That doesn't surprise me at all!

Out of curiosity, is Electric Drummer an Electron app? And if it is, how hard do you think it might be to get Electric Drummer working on Linux? It's the ultimate niche market, obviously, and I can't imagine it's commercially worth it. But I thought I'd ask, because sometimes an Electron app is easy to turn into a Linux alpha release. Fully polished Linux releases, sadly, are often more work.

Many years ago, I looked at what it would take to get the GPLed Frontier running well on Linux, and it looked pretty overwhelming.

i have a new "product" almost ready to go, something i've been using in Frontier for a number of years, that allows me to edit a project in an outline and distribute subs to various different places on github, s3, the local file system

This is such a good idea. "Mirror heading to S3" or "Mirror heading to pretty Markdown file" or "Mirror heading to repo" are obviously lovely features. Sometimes it's even feasible to support two-way sync, but that's obviously harder if you don't want to lose too much metadata when converting back and forth.

Once I have a nice outliner, there's such a temptation to use it for editing all different sorts of things. I saw a great plugin for org-mode that had two-way sync with the Trello task management tool. Here's a video with poor sound quality and a fairly clunky text UI, but it's neat to watch the outline stay in sync with the shared Trello board.

There's a lot of potential for good, user-friendly outliners with lots of integrations and plugins. But I don't have to tell you that. :-)

scripting · 2022-01-09T19:50:26Z

@emk -- quick response, more later --

yes electric drummer is electron. and the plan is to go to linux, i want to move my act there. apple is not a good place for what i do, and i'm quite comfortable with linux these days, all my servers are linux.

so yes -- it's very high on my list. but my list is long. ;-)

emk · 2022-01-09T20:33:15Z

so yes -- it's very high on my list. but my list is long. ;-)

Oh, I understand completely. There are always a million things to do. But if you do get around it, feel free to ping me if you need some Linux Electron help. At the very least I could alpha test and debug annoying packaging issues.

Integrating with LogSeq

Also, in the interest of figuring how to export data from LogSeq (or pull it back in), I dug through their hidden, experimental plugin marketplace and found some neat things. The most promising example is this:

Exporting tasks to Todoist and importing a Todoist inbox

(all images from the linked pages)

As you can see from the screen recordings, there's apparently a lot you can do using "/" commands. I could easily imagine a /Publish to Drummer command that used the current heading as a title, and subheadings as body text, and uploaded the whole thing to Drummer as OPML (or whatever). It would probably be easy to add slug:: heading-text-as-slug to the original heading, and then allow same heading to be easily re-exported.

I still can't figure out a good way to mark a heading to automatically export, though.

The other really neat thing I discovered were "renderers". These take a heading containing {{renderer RendererName args...}}, and display it as rich HTML or an image. Usually a heading with "renderer" will summarize its child nodes. Some examples:

Rendering a progress bar
Rendering a "kanban" board (like Trello)
Super simple "word count" demo showing how these update as you edit
There are also a bunch that render graphs and tables.

The neat thing about these is that they keep in sync as you edit the subheadings. So I could also imagine "creatively applying" this mechanism to keep an export in sync, using something like:

{{renderer DrummerBlog}}
- Title
  - Body text here.
  - More body text.

Probably you'd have to figure out how to specify the blog ID somehow. This stuff is clearly completely experimental, still.

There's a third type of plugin which adds an icon to the title bar. Here's an example:

Journal calendar navigation

This kind of plugin seems to be best for "full page" operations, or for things which affect an entire LogSeq directory.

Anyway, this is just a super quick summary of possible integration points. I won't have time to build any plugins this month, but maybe another LogSeq user will get some ideas about how to send data to Drummer. With just a little help from the LogSeq end, it would probably be super easy to flow data out into Drummer and do cool things with it.

(EDIT: Of course, it works fine to pull data, too, especially from something like the journal. I'm just looking at the various options, trying to find interesting possibilities.)

scripting · 2022-01-09T23:40:51Z

@emk -- I don't think it needs to automatically export. This is my current plan for how this will work.

#4 (comment)

Also it couldn't hurt to try creating a Drummer blog in Drummer to see what the goal is.

The docs, I am told, are quite good! :-)

http://docserver.scripting.com/drummer/blogging.opml

sabre23t · 2022-01-10T00:01:08Z

Unfortunately, my laptop is starting to have cooling fan problem, so writing this from my note mobile.

Emk wrote:

(EDIT: Of course, it works fine to pull data, too, especially from something like the journal.

I think current Drummer / Logseq interop is going towards the same model as Obsidian / Logseq interop. There Obsidian / Logseq edit the same folder of MD files. Major benefit is all integration can be done from Drummer end and its outline scripting environment.

Dave asked:

here's a question on how we'll do it -- how can we access your logseq files via web drummer without you having to import them one at a time. we want something that automatically updates.

Perhaps you can update Web Drummer to be able to open a local folder. The same local folder that Web Logseq stores mylogseq repo (with journals & pages subfolders). There's some permission issue in current Chrome browser that requires granting access everytime I refresh Web Logseq though. I'm using Web Logseq from my laptop.

here's my plan for the first rev of the LogSeq Journal ==> Drummer blog connection.

FWIW, the plan looks good to me too.

scripting · 2022-01-10T00:18:35Z

@sabre23t -- I noticed that LogSeq is using a browser API I am not familiar with.

We have everything we need on the Drummer side to be able to do this, once we have a way to access the files in the Journal folder. It'll all be very fast imho.

I think this is the API. 😀

However I think it says it's only accessible from HTTPS pages. Oy. That's too big a hoop to jump through for a quick feature addition.

PS: Maybe the fastest way to get to nirvana is to do the Windows and Linux versions of Electric Drummer. I don't have a desktop Linux machine here, just servers -- and I don't have a Windows machine either.

emk · 2022-01-10T03:08:20Z

This is my current plan for how this will work.

I think reading the journal from disk is excellent plan, for whatever my opinion is worth. :-)

I've played around with Drummer a little bit, and it's fun to see many "old but good" ideas be reborn on the web. I may try out a blog at some point!

Just out of curiosity—and I know this is a huge digression from the real work you folks are all doing in this thread—is there a way to create an outline in Drummer using POST? Does web Drummer use any server-side storage at all, or is it all local? (You can see that I'm still pretty clueless here.)

PS: Maybe the fastest way to get to nirvana is to do the Windows and Linux versions of Electric Drummer. I don't have a desktop Linux machine here, just servers -- and I don't have a Windows machine either.

As a long-time desktop Linux user for most of the last quarter century (yikes!), here's a rambling digression about my experiences.

When I was in college, it was great fun to install desktop Linux and fight with it until it worked. I learned a lot of Unix!

But now I'm a couple decades older and I don't want to fight with desktop Linux any more. I want decent hardware, with Linux pre-installed and fully supported. And if it breaks, I want someone to come and fix it.

For my last few machines, I've had good luck with Dell's Unbuntu laptops for business. These come pre-installed, and I just unbox them, create an account, and start working. Right now, I'm running a fully loaded Precision 5560 (or maybe a 5540). It's not as rugged as a Mac, but I make sure I always have the next day on-site support plan. I had a screen failure this past summer, and they actually sent a tech to drive out into the middle of nowhere to replace the screen. The "next business day" promise turned out to be an exaggeration in the countryside in the middle of the pandemic, but they did fix it. There might be a better option somewhere, but this is the best I've personally found in 25 years of desktop Linux.

Anyway, on to Electron stuff.

For me, the hardest part about porting an Electron app to Linux was building and packaging it. The actual code didn't require many changes at all. And once I had it set up, it was pretty much entirely automatic.

scripting · 2022-01-10T03:22:58Z

@emk all storage in web drummer is on the server, that I run.

emk · 2022-01-10T10:38:52Z

all storage in web drummer is on the server, that I run.

Great! Out of curiosity, is there any way to POST an outline to Drummer? I had at look at the docs and doc server, and there are some really useful integration tools there.

scripting · 2022-01-10T11:25:08Z

@emk — yes. Where are you posting from, inside drummer or outside?

emk · 2022-01-10T12:47:14Z

Outside Drummer, if that's possible!

scripting · 2022-01-10T14:18:08Z

@emk -- I started a new thread for this topic.

#17

scripting · 2022-01-10T14:59:06Z

@sabre23t & @emk -- a thought about how to get the LogSeq journal files into the cloud from the desktop.

http://publicfolder.io/

It's an Electron app. Of course now it only runs on Mac. Not sure how hard it would be to port to other platforms. It depends on a Node module called chokidar which claims to be cross-platform. ;-)

scripting · 2022-01-10T16:52:09Z

Update -- I tried Window.showOpenFilePicker to get started on accessing the local file system. I did it from within an HTTPS page. It says the function is not defined. Running in a fully updated version of Chrome.

sabre23t · 2022-01-10T23:36:22Z

how to get the LogSeq journal files into the cloud from the desktop.

Actually, mylogseq repo is already in the cloud (Dropbox). I use Dropbox on Android to download/upload Logseq Android local files. Would that be as workable as publicfolder.io for interop with web drummer?

scripting · 2022-01-11T00:25:19Z

@sabre23t — maybe. Dropbox has an api. It would be easier the files were on the web, or GitHub. I think logseq used to have a way to mirror to GitHub. Is that something you could look into?

emk · 2022-01-11T01:09:28Z

I think logseq used to have a way to mirror to GitHub.

LogSeq 0.5.4 has an option to automatically commit to a git repository every minute as you edit. I don't see any way to push this repository automatically to GitHub. But users could push it manually using any git client, or I'm sure there's a tool that auto-pushes.

For S3, try awscli. This supports aws S3 sync folder s3://bucket to synchronize a local folder to S3.

scripting · 2022-01-11T01:11:55Z

http://publicfolder.io

sabre23t · 2022-01-11T04:08:09Z

Dave wrote:

I think logseq used to have a way to mirror to GitHub. Is that something you could look into?

Yes. I have TortoiseGit (windows shell) and Git Bash that can mirror/sync a local folder to a Github repo. I've used them for some collaborative mapping apps.

Regarding "mirror to Github", what Logseq deprecated is Web Logseq backend link to Github.

However, Logseq continues supporting local folder synchronization with third party cloud services including Github. This works well except for "pages-metadata.edn" file that stores created-at and updated-at date/time for all Logseq files.

scripting · 2022-01-11T22:02:15Z

I have code that turns a Journals folder into an Old-School-ready OPML file.

Here's the writeup.

http://scripting.com/2022/01/11/215549.html?title=whatIDidToday

Tomorrow I'm going get it working with Old School. Knock wood.

scripting · 2022-01-12T18:17:18Z

I have a Drummer blog building from my LogSeq journal.

http://oldschool.scripting.com/mylogseqblog/

You do not need Electric Drummer to do this. It's a simple Node app. You provide the configuration in a config.json file in the same folder as the app.

I'm going to do some cleanup and then publish the app, in source of course.

scripting · 2022-01-13T16:40:04Z

It's ready for testers.

scripting/myLogseqBlog#1

Please report success or problems in that thread.

scripting mentioned this issue Jan 5, 2022

Changing logic for converting markdown to json and back scripting/opmlPackage#4

Closed

Markdown in Drummer? #4

Markdown in Drummer? #4

Comments

scripting commented Nov 2, 2021 • edited Loading

sabre23t commented Nov 4, 2021 • edited Loading

scripting commented Nov 4, 2021

sabre23t commented Nov 4, 2021 • edited Loading

scripting commented Nov 13, 2021

scripting commented Nov 13, 2021

scripting commented Nov 13, 2021

sabre23t commented Nov 14, 2021 • edited Loading

sabre23t commented Nov 14, 2021

sabre23t commented Nov 14, 2021 • edited Loading

scripting commented Dec 30, 2021

sabre23t commented Dec 31, 2021

scripting commented Dec 31, 2021

scripting commented Dec 31, 2021

scripting commented Dec 31, 2021

scripting commented Dec 31, 2021 • edited Loading

scripting commented Jan 1, 2022

sabre23t commented Jan 3, 2022 • edited Loading

scripting commented Jan 3, 2022

sabre23t commented Jan 3, 2022 • edited Loading

scripting commented Jan 3, 2022

sabre23t commented Jan 3, 2022 • edited Loading

scripting commented Jan 3, 2022

sabre23t commented Jan 3, 2022

scripting commented Jan 3, 2022

scripting commented Jan 3, 2022

andrewshell commented Jan 5, 2022

scripting commented Jan 6, 2022

scripting commented Jan 6, 2022 • edited Loading

scripting commented Jan 6, 2022

scripting commented Jan 8, 2022

scripting commented Jan 8, 2022

sabre23t commented Jan 9, 2022

emk commented Jan 9, 2022

scripting commented Jan 9, 2022

scripting commented Jan 9, 2022

scripting commented Jan 9, 2022

scripting commented Jan 9, 2022 • edited Loading

emk commented Jan 9, 2022 • edited Loading

scripting commented Jan 9, 2022

emk commented Jan 9, 2022 • edited Loading

Integrating with LogSeq

scripting commented Jan 9, 2022 • edited Loading

sabre23t commented Jan 10, 2022

scripting commented Jan 10, 2022 • edited Loading

emk commented Jan 10, 2022

scripting commented Jan 10, 2022

emk commented Jan 10, 2022

scripting commented Jan 10, 2022

emk commented Jan 10, 2022

scripting commented Jan 10, 2022

scripting commented Jan 10, 2022

scripting commented Jan 10, 2022

sabre23t commented Jan 10, 2022

scripting commented Jan 11, 2022

emk commented Jan 11, 2022

scripting commented Jan 11, 2022

sabre23t commented Jan 11, 2022

scripting commented Jan 11, 2022

scripting commented Jan 12, 2022

scripting commented Jan 13, 2022

scripting commented Nov 2, 2021 •

edited

Loading

sabre23t commented Nov 4, 2021 •

edited

Loading

sabre23t commented Nov 4, 2021 •

edited

Loading

sabre23t commented Nov 14, 2021 •

edited

Loading

sabre23t commented Nov 14, 2021 •

edited

Loading

scripting commented Dec 31, 2021 •

edited

Loading

sabre23t commented Jan 3, 2022 •

edited

Loading

sabre23t commented Jan 3, 2022 •

edited

Loading

sabre23t commented Jan 3, 2022 •

edited

Loading

scripting commented Jan 6, 2022 •

edited

Loading

scripting commented Jan 9, 2022 •

edited

Loading

emk commented Jan 9, 2022 •

edited

Loading

emk commented Jan 9, 2022 •

edited

Loading

scripting commented Jan 9, 2022 •

edited

Loading

scripting commented Jan 10, 2022 •

edited

Loading