Commit Graph

43 Commits

Author SHA1 Message Date
Bryan Richter
6bf160f210
Refactor decideOnSnapshotUpdate for understanding
Putting this in a separate commit since I'm actually refactoring code
rather than just changing names.
2025-02-27 15:05:44 +02:00
Bryan Richter
935a5012fe
Use names and Haddocks to improve understanding 2025-02-27 15:05:43 +02:00
Bryan Richter
652b78ab6b
Hush hlint 2024-04-03 15:48:10 +03:00
Bryan Richter
9420272b55
Log unexpected HTTP response fetching Hoogle DB 2024-04-03 15:45:21 +03:00
Bryan Richter
b56aaf33fc
Add comments, change names for understanding 2024-04-03 15:44:06 +03:00
Bryan Richter
eebde8b817
Add a bunch of docs around Hoogle DBs
so I remember how it all works.
2024-02-16 13:14:45 +02:00
Bryan Richter
5cb5668295
Revert to previous pinned version of pantry
The new pantry version in lts-22.6 was not compatible with the database
and/or config on the stackage server.
2024-02-13 09:20:41 +02:00
Bryan Richter
33e5cb2589
Upgrade all the way to lts-22.6
I stopped at 22.6 because I'm using NixOS and ghc-9.6.3 is the last
version available on the stable channel right now. Later snapshots use
9.6.4.
2024-02-12 15:15:22 +02:00
Bryan Richter
c1c7d14e15
Upgrade some hoogle messages to warnings 2024-02-12 15:12:57 +02:00
Bryan Richter
6b4232b1c6
Add download-bucket-url option 2024-01-08 16:45:50 +02:00
Bryan Richter
a2f77219b6
Remove unused TRUNCATE
The LatestVersion cache table was removed in
f8a82ec511.
2023-12-27 15:18:47 +02:00
Bryan Richter
a4cacd6991
Enable overriding S3 endpoint with AWS_S3_ENDPOINT 2023-12-23 12:27:21 +02:00
Bryan Richter
6331131b68
Enable running stackage-server-cron on an empty DB
It did run migrations, but ran them in the wrong spot.
2023-12-22 18:48:30 +02:00
Jens Petersen
2f8e8ba95a update to lts-18
- Cabal 3.2 PackageDescription uses ShortText
- esqueleto module changes
- silence a few warnings
2023-06-25 15:40:17 +08:00
Jens Petersen
7bc469c5f3 homepage layout changes and more [wip draft]
- enable navbar for homepage
- more navbar entries
- lts-16
- drop /?page
- [q] can hoogle be updated to 5.0.18?
- improve /snapshots "buttons"
- improve snapshot top matter
- [q] better to use haskellstack.org ?
2020-10-31 20:04:08 +08:00
Michael Snoyman
bfb01a7a92
yesod devel leverages SQLite for simplicity 2020-10-19 13:58:48 +03:00
Michael Snoyman
88f951a0b8
Don't vacuum 2020-09-07 09:00:52 +03:00
Michael Snoyman
0680b420e9
Revert "Try deleting Hoogle DBs to save disk space"
This reverts commit ebc27e0746.
2020-09-03 18:52:32 +03:00
Michael Snoyman
68fa14a4bb
Force using only latest Hoogle DBs 2020-09-02 17:39:52 +03:00
Michael Snoyman
4e7e62c3dc
Revert "Recover from Hoogle creation errors"
This reverts commit 42f4f7e586.
2020-08-27 13:33:09 +03:00
Michael Snoyman
42f4f7e586
Recover from Hoogle creation errors 2020-08-27 10:41:02 +03:00
Michael Snoyman
26d4a2312e
Cache latest version 2020-08-27 10:25:06 +03:00
Michael Snoyman
ebc27e0746
Try deleting Hoogle DBs to save disk space 2020-08-27 09:44:59 +03:00
Michael Snoyman
bd40aeddd9
Retry downloading the deprecated.json file 2020-03-13 12:21:58 +02:00
Alexey Kuleshevich
fe25b2fa2f
Record available hoogle db files per snapshot + hoogle version combination:
* Make sure hoogle db is marked as available, when there is a copy on S3

* Create db even with `--do-no-upload` flag (useful for testing)

* Make sure home page uses latest lts with hoogle db available
2020-02-14 22:19:16 +03:00
Michael Snoyman
96973cac11
More snapshots for Hoogle 2020-02-13 18:02:43 +02:00
Michael Snoyman
ef0247d3bb
Newest Hoogle first 2020-02-13 10:41:19 +02:00
Alexey Kuleshevich
bdcdd1887a
Store fallback cabal files into pantry. And few follow up improvements:
* Fix atomic durable writing, since issue in RIO was fixed

* Log information about falling back onto the core-cabal-files repo

* Convert conduit pipe to Maybe fishes.

* Make sure module names, package name and version are added for fallback
  cabal files
2020-02-12 19:44:34 +03:00
Alexey Kuleshevich
8e247dde03
Update to ghc-8.8, pantry-0.2 and Cabal-3.0 2020-02-12 02:10:14 +03:00
Alexey Kuleshevich
722260e1d4
Addition of fallback repository with core cabal files 2020-02-12 00:40:31 +03:00
Michael Snoyman
2701f186ca
Try to avoid chunkedFile #275 2019-09-25 14:43:31 +03:00
Michael Snoyman
dfe0122edf
Cron job: more Hoogle debug info 2019-09-25 12:02:18 +03:00
Alexey Kuleshevich
cbfb68bdc8
Implemented automatic undeprecation of previously deprecated packages, also:
* Made sure update of deprecated is done each run, independently of Hackage update
2019-07-30 13:28:17 +03:00
Alexey Kuleshevich
05307bded8
Fix priority of core packages:
* `global-hints.yaml` is now used as a fallback for packages that
  are not included in the snapshot
* Fix ordering of dependencies on the package page
2019-07-06 20:52:52 +03:00
Alexey Kuleshevich
d9a285a87f
Fix memory leak during stackage cron job when caching is enabled 2019-06-26 12:15:23 +03:00
Alexey Kuleshevich
385620e185
Made status reporting and cabal file caching optional for the cron job 2019-05-25 20:00:28 +03:00
Alexey Kuleshevich
f5e147ab97
Integration with Pantry and usage of new stackage-snapshots:
* Moved all extensions into modules that are using them, rather than globally,
  since they mess up ghci session and introduce conflicts among
  packages. Removed those from `.ghci` file as well
* Redesigned the schema to use Pantry and moved it into it's own module
* Switched all of the db and cron related stuff to RIO. Yesod part is
  still on classy-prelude
* Got pantry to update stackage-server database from hackage
* Got import of stackage-snapshots implemented
* Moved some logic from all-cabal-tool
* Switched everything to `PackageNameP`, `VersionP`, etc. from a la Text.
* Fixed haddock, so it now does proper redirects and pipes the docs
  correctly. Also implemented piping of json files from S3 bucket,
  so index-doc.json is also served by stackage-server thus making
  Ctrl+S feature work properly on haddock. Fix for commercialhaskell/stackage#4301
* Import of modules is done through cabal file parsing, which slows
  down the initial import process drastically, but incremental update
  is not a problem.
* Just as with modules, dependencies are also imported from cabal file.
* In general improved type safety by introducing a few data types:
  eg. `ModuleNameP`, `HackageCabalInfo`, and many more.
* Implemented pulling of deprecation map from hackages and storing it in db
* Implementation of forward/backward dependencies within a snapshot only.
* Drastically improved performance of cron import job, by checking which
  snapshots are not up to date
* Implemented pulling haddock list from S3 bucket. Modules that have
  documentation are marked from the availability of actual haddock. This
  process happens concurrently with snapshots loading.
* Rearranged modules a bit:
  * github related functions went into it's own module
  * cron related functions where moved from Database to Cron module
  * Split up some functions to reduce individual complexity
* Parallelized package loading in cron job
* Implemented parsed cabal file caching.
* All queries where reqritten with esqueleto
* Syntactic improvements:
  * Added stylish-haskell config
  * Formatted all imports and extensions with stylish-haskell.
  * Fixed inconsistent indentation across all modules
* Many improvements to the package page as well as few others.
* Reimplemented hoogledb creation.
* Dropped dependency on tar in favor of tar-conduit
* Added cli for stackage-server-cron
* Add cabal sha and size to the package page
* Fixed links in hoogle searches. Improved type safety for a hoogle handler
* satckage-server-cron is customizable with cli arguments

Final adjustments for the new stackage server release:

* Upgrade to lts-13.16.
* Stackage server related code has been merged to pantry. Made the code
  compatible with the newer version pantry
* Added cli '--snapshots-repo'
* Add readme to package page
* Adjust snapshots expected format:
  * Added `publish-time`
  * Removed name `field`
  * `compiler` field is now in the `resolver` field with fallback to
    the root
2019-04-30 17:10:33 +03:00
Sibi Prabakaran
9ea91909c2
Add CPP flags 2018-08-25 22:13:09 +05:30
Michael Snoyman
96e9a53a17
Remove system-file(path/io) 2018-06-21 19:30:48 +03:00
Michael Snoyman
856ac728b4
Fix warnings 2018-06-21 17:51:47 +03:00
Michael Snoyman
cf14304ee3
Upgrade snapshots 2018-06-21 17:26:58 +03:00
Michael Snoyman
f8aa5bc4de Error out on no packages 2018-06-05 13:40:23 -06:00
Michael Snoyman
e05cc3a20d
Move to src dir 2017-12-11 18:52:11 +02:00