Skip to content

Simple "server" implementation for catalog. #80

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
TripleDogDare wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Simple "server" implementation for catalog. #80

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions app/_docs/warpforge-catalog.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,9 @@ Generates HTML output for the root workspace catalog containing information on m
### mirror
Mirror the contents of a catalog to remote warehouses

### serve
Generates html and serves
Copy link
Copy Markdown
Collaborator

@warpfork warpfork Oct 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Propose:

Serve a local http site that contains a navigable copy of the Catalog, and hosts a warehouse at _wares/. (Meant for rapid sharing in LAN environments.)


### help, h
Shows a list of commands or help for one command

Expand Down
119 changes: 99 additions & 20 deletions app/catalog/catalog_cli.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import (
"bytes"
"context"
"fmt"
"net/http"
"os"
"os/exec"
"path/filepath"
Expand Down Expand Up @@ -156,6 +157,37 @@ var catalogCmdDef = &cli.Command{
util.CmdMiddlewareTracingSpan,
),
},
{
Name: "serve",
Usage: "Generates html and serves",
Action: util.ChainCmdMiddleware(cmdServe,
util.CmdMiddlewareLogging,
util.CmdMiddlewareTracingConfig,
util.CmdMiddlewareTracingSpan,
),
Flags: []cli.Flag{
&cli.StringFlag{
Name: "output",
Aliases: []string{"o"},
Usage: "Output path for HTML generation",
},
Comment on lines +169 to +173
Copy link
Copy Markdown
Collaborator

@warpfork warpfork Oct 2, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might prefer to get rid of this, per other comments about making this as works-by-default as possible. We can just pick a path under {rootWorkspaceControlDir}/tmp/ (or $WARPFORGE_RUNPATH?)

&cli.StringFlag{
Name: "url-prefix",
Usage: "URL prefix for links within generated HTML",
Value: "/",
},
Comment on lines +174 to +178
Copy link
Copy Markdown
Collaborator

@warpfork warpfork Oct 2, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's drop this too, per other comments about making this as works-by-default as possible. We control the whole mux that's processing these URLs and not supporting configuring it, so this option doesn't even make sense in that story.

&cli.StringFlag{
Name: "download-url",
Usage: "URL for warehouse to use for download links",
},
Comment on lines +179 to +182
Copy link
Copy Markdown
Collaborator

@warpfork warpfork Oct 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's get rid of this... and make the server put this workspace's .warpforge/warehouse/ directory online at _wares and just commit to that.

IMO, the overall behavior of this feature should be: user wants to immediately share; is on their LAN or otherwise not currently doing any other configuration; and just wants a result that works. So we should pick defaults that get stuff done and just go with them.

For this feature in particular, I don't see any reason to leave it configurable at all. We use this in the full static site-gen tool because it lets us account for stories like "my storage bucket and my website are on two totally different domains", etc... but that's never going to be even possible for this kind of "user on a LAN wants results fast" story.

&cli.UintFlag{
Name: "port",
Aliases: []string{"p"},
Usage: "Port number for the server address",
Value: 8080,
},
Comment on lines +183 to +188
Copy link
Copy Markdown
Collaborator

@warpfork warpfork Oct 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe take a whole --addr flag and default it to ":7070" (with the colon)?

(I'd also just as druther pick a different default port number than 8080. We're not doing a website tool demo (where that number is a norm) so much as we're doing a specific piece of infrastructure here... so I think we're justified in picking a number less likely to collide.)

},
},
},
}

Expand Down Expand Up @@ -715,69 +747,116 @@ func cmdCatalogShow(c *cli.Context) error {
return nil
}

func cmdGenerateHtml(c *cli.Context) error {
catalogName := c.String("name")
type protoSiteConfig struct {
catalogName string
outputPathOverride string
urlPrefix string
downloadUrl string
}
Comment on lines +750 to +755
Copy link
Copy Markdown
Collaborator

@warpfork warpfork Oct 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I understand why this struct is needed, which is almost exactly a repeat of the existing SiteConfig struct?


func (cfg protoSiteConfig) SiteConfig(ctx context.Context) (cataloghtml.SiteConfig, error) {
// open the workspace set
wsSet, err := util.OpenWorkspaceSet()
if err != nil {
return err
return cataloghtml.SiteConfig{}, err
}

// create the catalog if it does not exist
exists, err := wsSet.Root().HasCatalog(catalogName)
exists, err := wsSet.Root().HasCatalog(cfg.catalogName)
if err != nil {
return err
return cataloghtml.SiteConfig{}, err
}
if !exists {
return fmt.Errorf("catalog %q not found", catalogName)
return cataloghtml.SiteConfig{}, fmt.Errorf("catalog %q not found", cfg.catalogName)
}

cat, err := wsSet.Root().OpenCatalog(catalogName)
cat, err := wsSet.Root().OpenCatalog(cfg.catalogName)
if err != nil {
return fmt.Errorf("failed to open catalog %q: %s", catalogName, err)
return cataloghtml.SiteConfig{}, fmt.Errorf("failed to open catalog %q: %s", cfg.catalogName, err)
}

// by default, output to a subdir of the catalog named `_html`
// this can be overriden by a cli flag that provides a path
outputPath, err := wsSet.Root().CatalogPath(catalogName)
outputPath, err := wsSet.Root().CatalogPath(cfg.catalogName)
if err != nil {
return err
return cataloghtml.SiteConfig{}, err
}
outputPath = filepath.Join("/", outputPath, "_html")
if c.String("output") != "" {
outputPath = c.String("output")
if cfg.outputPathOverride != "" {
outputPath = cfg.outputPathOverride
}

// by default, the URL prefix is the same as the output path,
// this works if the HTML is accessed using `file:///` URLs.
// however, to allow for generating a hosted site, this can be
// overridden by the CLI
urlPrefix := outputPath
if c.String("url-prefix") != "" {
urlPrefix = c.String("url-prefix")
if cfg.urlPrefix != "" {
urlPrefix = cfg.urlPrefix
}

var warehouseUrl *string = nil
if c.String("download-url") != "" {
dlUrl := c.String("download-url")
if cfg.downloadUrl != "" {
dlUrl := cfg.downloadUrl
warehouseUrl = &dlUrl
}

cfg := cataloghtml.SiteConfig{
Ctx: context.Background(),
return cataloghtml.SiteConfig{
Ctx: ctx,
Cat_dab: cat,
OutputPath: outputPath,
URLPrefix: urlPrefix,
DownloadURL: warehouseUrl,
}, nil
}

func cmdGenerateHtml(c *cli.Context) error {
pcfg := protoSiteConfig{
catalogName: c.String("name"),
outputPathOverride: c.String("output"),
urlPrefix: c.String("url-prefix"),
downloadUrl: c.String("download-url"),
}
cfg, err := pcfg.SiteConfig(c.Context)
if err != nil {
return err
}

os.RemoveAll(cfg.OutputPath)
if err := cfg.CatalogAndChildrenToHtml(); err != nil {
return fmt.Errorf("failed to generate html: %s", err)
}

fmt.Printf("published HTML for catalog %q to %s\n", catalogName, outputPath)
fmt.Printf("published HTML for catalog %q to %s\n", pcfg.catalogName, cfg.OutputPath)

return nil
}

func cmdServe(c *cli.Context) error {
pcfg := protoSiteConfig{
catalogName: c.String("name"),
outputPathOverride: c.String("output"),
urlPrefix: c.String("url-prefix"),
downloadUrl: c.String("download-url"),
}
port := c.Uint("port")
addr := fmt.Sprintf("%s:%d", "localhost", port)

cfg, err := pcfg.SiteConfig(c.Context)
if err != nil {
return err
}
os.RemoveAll(cfg.OutputPath)
if err := cfg.CatalogAndChildrenToHtml(); err != nil {
return fmt.Errorf("failed to generate html: %s", err)
}

fmt.Printf("published HTML for catalog %q to %s\n", pcfg.catalogName, cfg.OutputPath)

dirHandler := http.Dir(cfg.OutputPath)
fmt.Printf("Serving at http://%s\n", addr)
Copy link
Copy Markdown
Collaborator

@warpfork warpfork Oct 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add some more hints here like "You can now use ca+http://%s/_wares/ as a warehouse URL to fetch content available from this workspace"?

Copy link
Copy Markdown
Collaborator

@warpfork warpfork Oct 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding the actual feature behind that should be just a mux, a few lines below here, roughly:

mux := NewServeMux()
mux.Handle("/", http.Dir(cfg.OutputPath))
mux.Handle("/_wares/", http.Dir(workspaceWareDir)

(well, and then get the workspace info down here, so we actually have that path.)

if err := http.ListenAndServe(addr, http.FileServer(dirHandler)); err != nil {
Copy link
Copy Markdown
Collaborator

@warpfork warpfork Oct 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know I shouldn't pick at a first-pass, but it drives me nuts that the message above gets printed before we actually try to bind :)

I'd like it a lot better if we did listener, _ := net.Listen("tcp", addr), then make an srvr := https.Server{http.FileServer(dirHandler)}, then srvr.Serve(listener).

Bonus one is separate error handling for the bind, which is the main thing that can go wrong.

Bonus two is we can also ask listener.(*TCPListener).Addr(), and then get the real address (e.g. complete with IP address if someone an address spec like ":7070" that's just port) back out, and give the user a URL with that in it. (I think. Haven't tested this.)

return err
}
return nil
}

Expand Down