I've pushed another round of bug fixes and also changed the database
schema to store the GOOS and GOARCH of the documentation so that we can
allow the user to choose the GOOS in the future. The database now stores
a list of the module's versions as well as the module's v1 path so that
we can easily retrieve all the versions present in the database.

On Sat Apr 3, 2021 at 9:00 PM EDT, Adnan Maolood wrote:
> I believe that these changes can be accepted upstream now. I would like
> to send some patches to clean up various gddo-server code before I work
> on implementing the ability to choose which version of a package to
> display.

To clarify: all that needs to be done is to present available versions
to the user in the frontend. The backend support is complete. We might
also want to store a list of versions for each module in the database
independent of which versions have documentation stored in the database.
Otherwise, only the versions which have documentation stored will be
shown.

I've pushed an update which stores go-source meta tags in the database
upon crawling. I've also added a message that is displayed when crawling
a package times out, and fixed a few issues related to import counts.
I've also removed the Go Subrepo index endpoint as there is no easy way
to implement it.

I believe that these changes can be accepted upstream now. I would like
to send some patches to clean up various gddo-server code before I work
on implementing the ability to choose which version of a package to
display.

Later on I would like to implement the ability to choose which GOOS to
display documentation for. Perhaps I should ensure that the database is
set up to support this before these changes are accepted, otherwise we

Another question: since there is no clean way to keep track of all
golang.org/x modules without maintaining a list of them, shall we remove
the /-/subrepo endpoint? That endpoint is currently not advertised
anywhere at the moment.

On Sat Apr 3, 2021 at 12:38 PM EDT, Drew DeVault wrote:
> Yeah, something like that. Or maybe we should just return quickly and
> display a "This module is being fetched in the background. Feel free to
> refresh while we're working on it.", with a meta tag which refreshes
> automatically every 10 seconds or something.

Perhaps we should separate the action of searching for a package and
requesting to add it to the database? We can show a message on the
bottom of the search results like "Can't find what you're looking for?
You might need to [add a module to the database]". Then we can have a
dedicated page for adding modules which shows the status of fetching or
something.

On Thu Apr 1, 2021 at 11:19 AM EDT, Drew DeVault wrote:
> Some feedback from testing:
>
> - The first time a package is fetched, it returns an empty search result
> instead of the package page. Attempting the search again shows the
> now-indexed package.

It seems like this happens when fetching the module times out. The logs
will report "Serving package ... as not found after timeout getting
doc". Perhaps the default timeout should be increased? Or perhaps we
should display an error message along the lines of: "This module is
taking a while to fetch, please check back later".

> On Tue Mar 30, 2021 at 3:30 PM EDT, Adnan Maolood wrote:

On Thu Apr 1, 2021 at 1:01 PM EDT, Drew DeVault wrote:
> On Thu Apr 1, 2021 at 1:00 PM EDT, Adnan Maolood wrote:
> > On Thu Apr 1, 2021 at 12:56 PM EDT, Drew DeVault wrote:
> > > Hm, we should probably base our behavior on what Go itself does when you
> > > try to import each version.
> >
> > That would mean treating each version as a separate module, so the
> > current behavior is correct. Though in the future we may want to fetch
> > later versions and allow the user to switch between them.
>
> Well, there may be some happy medium. Is it possible to enumerate the
> versions, given any of them as an input?

Unfortunately, not really. The Go module proxy will only list versions

On Thu Apr 1, 2021 at 12:56 PM EDT, Drew DeVault wrote:
> Hm, we should probably base our behavior on what Go itself does when you
> try to import each version.

That would mean treating each version as a separate module, so the
current behavior is correct. Though in the future we may want to fetch
later versions and allow the user to switch between them.

On Thu Apr 1, 2021 at 11:19 AM EDT, Drew DeVault wrote:
> Some feedback from testing:
>
> - The first time a package is fetched, it returns an empty search result
> instead of the package page. Attempting the search again shows the
> now-indexed package.
> - I tested it with github.com/go-git/go-git and it did not show the
> latest module version. We should also definitely get mulitple versions
> supported before we ship this.

In this case we probably need to try retrieving every major version of a
module until we run into an error. This brings up a good question:
should we consider github.com/go-git/go-git the same as
github.com/go-git/go-git/v5 or whatever the latest version is? It seems

I've finally got support for Go modules working. In the meantime I've
also removed a dependency on Redis and moved the entire database to
PostgreSQL. I would appreciate any assistance with testing the
implementation.

To test the changes, initialize the PostgreSQL database with the schema
found in schema.sql. Then pass the database URL to gddo-server with the
--pg-server flag.

Nearly all funtionality has been kept intact with the exception of the
following TODOs.

Remaining TODOs:
- Remove all matching packages from the database upon blocking an import