WIP: Submodes for JSON logs

[gazally]

I've been working with some projects that emit logs in JSON and found myself wishing that I could work with those logs efficiently in Emacs, so with a little free time on my hands I decided to dive into logview-mode to see what I could make it do, and added a JSON submode.

The easiest way to check this out is to open test/json/levels.log. On the bottom of the cheat sheet there are several new commands with the prefix j .

Things it does:

• Recognizes JSON-lines content (one JSON object per line).
• Parses JSON log entries.
• Finds the timestamp, level, thread, name and message in the object (where they are located is configurable).
• Implements filtering based on those values.
• Fontifies based on those values.
• During fontification puts values from the JSON in the line-prefix text property for each entry, and provides commands to toggle display of those values on and off.
• Implements timestamp diffing.
• Can show you pretty-printed JSON in a separate buffer.
• Can show you the Lisp object parsed from the JSON pretty-printed in a separate buffer.

Things that aren't done:

• Recognize JSON objects spanning multiple lines.
• Deal with intermixed plain text and JSON.
• Handle entries with missing values.
• Let you select text from the values you are displaying in the line prefix.
• Update the README to explain how to add new JSON submodes.

Things I'm thinking about making it do:

• Save the selection of values to show in the line-prefix with the view.
• Since you can't select line-prefix text, add some commands that copy values to the kill ring. Those commands could work in plain text logging submodes too.
• Define custom paths to extract values from the JSON that you can use in filtering or display in the line prefix.
• Add a configurable hook or fixup function to run after the JSON is parsed to modify the object (for example I'm working with a library that logs strings containing JSON inside the JSON, so a fixup function could parse those).
• Apply json-mode fontification to each entry.

[gazally]

I've added logview-additional-json-submodes as a place to define your own custom JSON submodes. The docstring is accurate but the customization interface isn't done yet. I've also added a hook that is run on the parsed object from each line, and an alist for column widths for the values shown in the line prefix. After these improvements my JSON logs are now so human-readable that I made logview-toggle-details-globally toggle the display of the JSON, and set the default to not show the JSON.

The approach of stuffing possibly very long log message text into the line-prefix text property is clearly abusing what that property is meant for and has some usability issues. The alternative seems to be actually inserting that text into the buffer (and keeping the original buffer contents somewhere else and swapping them back when the user exits read-only mode or runs write-file or any number of other things I haven't thought of yet).

[doublep]

Sorry, have no time to look into this now, especially since this appears to be a huge change.

Please look into the test failures. Also note that you can always run tests locally ($ eldev test).

Some general points:

• For my personal use, performance of the mode is very important. While I don't care much about JSON logs, at least existing functionality must not suffer performance-wise. Is that the case?
• Can this be broken out into a separate file, or are interdependencies too heavy?

[gazally]

This is still a WIP and all I'm looking for at the moment is suggestions and maybe being able to help someone else who likes logview's interface and has JSON logs they need to look at. I also use logview on plain text files and hope to maintain performance. If adding checks of logview--submode-type at certain inner loop locations turns out to be too much, then this might be better as a fork. But I was going to wait and see. I'll explore breaking it out into a separate file.

I am unable to reproduce the eldev test failure on my machine. Since I use NixOS, which won't run executables that aren't built for it, and eldev is not yet packaged for NixOS, I've been running eldev in Docker, on a Debian base image. I'll look into what the CI is doing that might be different than my Dockerfile.

[doublep]

I see (void-function json-available-p) on Emacs 25.3, which probably indicates that the dependency you introduced is misbehaving on ancient Emacs versions (maybe it doesn't declare required Emacs version at all, don't know, but at least its installation succeeded). You could probably just disable the new JSON extensions on older Emacses.

I use NixOS, which won't run executables that aren't built for it, and eldev is not yet packaged for NixOS, I've been running eldev in Docker

I don't know what NixOS is, but if it can run UNIX shell scripts and Emacs, it should be able to run Eldev too, as nothing else is needed.