From Adnan Maolood to ~sircmpwn/godocs.io
On Sun Jun 20, 2021 at 7:50 PM EDT, Drew DeVault wrote: > Can we estimate this based on the number of packages in the production > db today? Come up with some kind of model for predicting its behavior? As of right now, there are 15022 unique packages that are imported by packages in the database. Of these 15022 packages, 1693 (about 11.3%) are not already in the database. Now, we can't tell how many packages these 1693 import without adding them to the database or fetching them from the module proxy. --- To retrieve the number of unique imported packages in the database, this command was used:
From Adnan Maolood to ~sircmpwn/godocs.io
Using the version "latest" caused problems when fetching the standard library. Use the resolved latest version instead. --- gddo-server/fetch.go | 3 +++ 1 file changed, 3 insertions(+) diff --git a/gddo-server/fetch.go b/gddo-server/fetch.go index ba795b7..39a0218 100644 --- a/gddo-server/fetch.go +++ b/gddo-server/fetch.go @@ -93,6 +93,9 @@ func (s *Server) fetch(ctx context.Context, modulePath, version string) error { if err := s.db.PutModule(ctx, mod); err != nil { return err } [message trimmed]
From Adnan Maolood to ~sircmpwn/godocs.io
In a separate goroutine, continuously fetch the oldest module in the
database from the module proxy and update it if necessary. Note that the
--crawl-interval flag must be specified to enable background fetching.
---
A few remaining questions:
- Should we crawl the imports of packages (to discover new packages not
in the database)? If we do, perhaps this should be opt-in as this
could cause the database to grow exponentially, which wouldn't be
desirable for small installations.
- Is PostgreSQL's ORDER BY fast enough for large databases? Or is speed
not really a concern?
- Should we log FETCH messages, or should we keep silent except when
[message trimmed]
From Adnan Maolood to ~sircmpwn/godocs.io
---
gddo-server/{crawl.go => fetch.go} | 156 ++++++++++++++++-------------
gddo-server/gemini.go | 5 +-
gddo-server/http.go | 5 +-
gddo-server/server.go | 16 ++-
4 files changed, 98 insertions(+), 84 deletions(-)
rename gddo-server/{crawl.go => fetch.go} (50%)
diff --git a/gddo-server/crawl.go b/gddo-server/fetch.go
similarity index 50%
rename from gddo-server/crawl.go
rename to gddo-server/fetch.go
index 766ce65..a425d24 100644
--- a/gddo-server/crawl.go
[message trimmed]
From Adnan Maolood to ~sircmpwn/godocs.io
Consolidate the parsing of request paths into a single function. Also take the opportunity to reject invalid versions like "master" or "main" which cause issues. --- gddo-server/crawl.go | 2 +- gddo-server/gemini.go | 18 +++++------------- gddo-server/http.go | 20 ++++++-------------- gddo-server/server.go | 24 ++++++++++++++++++++++++ 4 files changed, 36 insertions(+), 28 deletions(-) diff --git a/gddo-server/crawl.go b/gddo-server/crawl.go index 4e527e8..766ce65 100644 --- a/gddo-server/crawl.go [message trimmed]
From Adnan Maolood to ~sircmpwn/godocs.io
--- These functions are not used by gddo-server, so they can be removed. internal/version/version.go | 207 +------------------------------ internal/version/version_test.go | 198 +---------------------------- 2 files changed, 6 insertions(+), 399 deletions(-) diff --git a/internal/version/version.go b/internal/version/version.go index 984c76f..a9ab050 100644 --- a/internal/version/version.go +++ b/internal/version/version.go @@ -14,25 +14,20 @@ import ( ) [message trimmed]
From Adnan Maolood to ~sircmpwn/godocs.io
On Sat Jun 19, 2021 at 8:44 AM EDT, Teddy Wing wrote: > I'm unclear on what the `importerCount` limit at 8 is for, but figured > I'd reuse the code from just before > be62deeb42eeb7a2a467b9751e34108d8e498047. I'm not sure what that is for either. Perhaps it can be removed?
From Adnan Maolood to ~sircmpwn/godocs.io
--- Don't allow specifying pseudo-versions as they can be abused to fill up the database (and they don't show up in the list of versions anyways). Later on, we should also automatically remove pseudo-versions from the database but keep the latest pseudo-version for packages with no release versions available. gddo-server/crawl.go | 1 + gddo-server/gemini.go | 7 +++++++ gddo-server/http.go | 13 ++++++++++++- 3 files changed, 20 insertions(+), 1 deletion(-) diff --git a/gddo-server/crawl.go b/gddo-server/crawl.go [message trimmed]
From Adnan Maolood to ~sircmpwn/godocs.io
--- This patch implements support for specifying which version of a package to display documentation for. The documentation for that version will be fetched on demand if it is not already available. gddo-server/assets/templates/doc.gmi | 2 +- gddo-server/assets/templates/doc.html | 2 +- gddo-server/assets/templates/versions.html | 17 +++++++++++++++++ gddo-server/crawl.go | 12 ++++++++---- gddo-server/gemini.go | 17 ++++++++++++----- gddo-server/http.go | 20 +++++++++++++++----- gddo-server/server.go | 8 ++++---- gddo-server/template.go | 3 ++- 8 files changed, 60 insertions(+), 21 deletions(-) [message trimmed]
From Adnan Maolood to ~sircmpwn/godocs.io
--- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 731f032..dbc7b9b 100644 --- a/README.md +++ b/README.md @@ -18,6 +18,7 @@ First install the dependencies: - Go 1.13 or above - PostgreSQL 13 - Graphviz (required for import graphs) [message trimmed]