mirror of
https://github.com/commercialhaskell/stackage.git
synced 2026-01-11 23:08:30 +01:00
More cleanup
This commit is contained in:
Michael Snoyman
parent
0e1a160477
commit
e892e6effd
19
.gitignore
vendored
19
.gitignore
vendored
@ -1,26 +1,11 @@
|
||||
dist
|
||||
cabal-dev
|
||||
*.o
|
||||
*.hi
|
||||
*.chi
|
||||
*.chs.h
|
||||
.virthualenv
|
||||
*.swp
|
||||
/runtests/
|
||||
/build-plan.log
|
||||
/build.log
|
||||
/logs/
|
||||
/sandbox/
|
||||
/build-tools.log
|
||||
/logs-tools/
|
||||
build-plan.txt
|
||||
build-plan.csv
|
||||
hackage-map.txt
|
||||
module-name-conflicts.txt
|
||||
/exclusive
|
||||
/inclusive
|
||||
*.stackage
|
||||
/haddock/
|
||||
/build-summary/
|
||||
/.cabal-sandbox/
|
||||
cabal.sandbox.config
|
||||
nightly-*.yaml
|
||||
lts-*.yaml
|
||||
|
||||
43
README.md
43
README.md
@ -42,41 +42,14 @@ agreement](https://github.com/fpco/stackage/wiki/Maintainers-Agreement).
|
||||
Build the package set
|
||||
---------------------
|
||||
|
||||
As this project is just starting, we don't really have a solid set of steps. In
|
||||
general, the following set of commands should be good for getting started:
|
||||
Generally, building the package set should be done only by the Jenkins machine
|
||||
or by the official maintainers, as the process does require quite a bit of
|
||||
setup on the local machine. That said, you'll likely be able to get a stable
|
||||
build by running:
|
||||
|
||||
cabal update
|
||||
git clone https://github.com/fpco/stackage
|
||||
cd stackage
|
||||
cabal sandbox init # requires cabal-install 1.18
|
||||
cabal install --only-dependencies
|
||||
cabal configure
|
||||
cabal build
|
||||
./patching/scripts/create-tarballs.sh
|
||||
./dist/build/stackage/stackage select
|
||||
./dist/build/stackage/stackage check
|
||||
./dist/build/stackage/stackage build # takes a *long* time
|
||||
./dist/build/stackage/stackage test # also takes a *long* time
|
||||
cabal install stackage
|
||||
stackage nightly
|
||||
|
||||
Notes
|
||||
-----
|
||||
|
||||
Make sure to have Cabal-1.16 installed in either your global or user database,
|
||||
regardless of any sandboxing, as custom build types require it to be present.
|
||||
You must build with cabal-install 1.16, due to several important bug fixes.
|
||||
|
||||
Using a non-Haskell Platform versions of GHC
|
||||
--------------------------------------------
|
||||
|
||||
By default, Stackage bases itself off of the Haskell Platform for determining
|
||||
which packages are core packages, and locks down package versions to match the
|
||||
Haskell Platform selections. This works fine when you are compiling with the
|
||||
same version of GHC as the Haskell Platform was built on. If you're using a
|
||||
different version of GHC, you'll probably need to use the following options for
|
||||
the `select` call:
|
||||
|
||||
--no-platform --use-global-db
|
||||
|
||||
The former says to disregard Haskell Platform package versions, and the latter
|
||||
says to determine which packages are core packages based on their presence in
|
||||
the global package database.
|
||||
The code itself
|
||||
---------------
|
||||
|
||||
61
ROADMAP.md
61
ROADMAP.md
@ -1,61 +0,0 @@
|
||||
## Processing
|
||||
|
||||
High level series of steps for processing
|
||||
|
||||
### Nightlies
|
||||
|
||||
1. Get list of core packages
|
||||
2. Get build constraints from list of maintained packages
|
||||
3. Load up package index
|
||||
4. Calculate build plan using newest versions of packages
|
||||
5. Write out a YAML file with complete build plan
|
||||
6. Verify that the build plan can be compiled
|
||||
7. Perform the build
|
||||
|
||||
### LTS
|
||||
|
||||
1. Load up most recent build plan
|
||||
2. Convert build plan into constraints for next build
|
||||
3. Continue from step (3) above
|
||||
|
||||
## Code explanation
|
||||
|
||||
We start off with *constraints*. Constraints state things like "package X has a
|
||||
given version range," who the maintainer is for a package, the description of
|
||||
the system/compiler being used, etc. `BuildConstraints` describes the build as
|
||||
a whole, whereas `PackageConstraints` describes the constraints on an
|
||||
individual package.
|
||||
|
||||
There are two primary ways of getting a `BuildConstraints`.
|
||||
`defaultBuildConstraints` inspects the first GHC in the PATH environment variable to
|
||||
determine GHC version, core packages, core tools, etc. It then uses the
|
||||
`Stackage.Config` module to extract information on additional packages to be
|
||||
installed. The secondary approach is in `Stackage2.UpdateBuildPlan`, which will be
|
||||
discussed later.
|
||||
|
||||
`BuildConstraints` does not specify a build completely. That is given by a
|
||||
`BuildPlan`, which is similarly broken down into `BuildPlan` and `PackagePlan`.
|
||||
In order to get a `BuildPlan`, we need two pieces of information: the
|
||||
`BuildConstraints`, and a package index. The package index (usually downloaded
|
||||
from Hackage) is a collection of all of the cabal files available.
|
||||
|
||||
By applying a `BuildConstraints` to a package index (via `newBuildPlan`), we
|
||||
get a proposed `BuildPlan`. There is no guarantee that this `BuildPlan` is
|
||||
valid. To validate it, we use `checkBuildPlan`. A `BuildPlan` is an instance of
|
||||
both `ToJSON` and `FromJSON`, and therefore can be serialized to a file for
|
||||
later use.
|
||||
|
||||
When dealing with LTS Haskell, we want to be able to take a `BuildPlan`, and
|
||||
update to a newer `BuildPlan` that keeps all packages at the same major
|
||||
version. `updateBuildConstraints` turns a `BuildPlan` into a new
|
||||
`BuildConstraints` with that restriction, and `updateBuildPlan` applies
|
||||
`newBuildPlan` to that result. As mentioned previously: this is *not* a
|
||||
validated result, and therefore `checkBuildPlan` must be used.
|
||||
|
||||
A `BuildPlan` can be acted on. This is done to check that all packages compile
|
||||
together, run relevant test suites, test Haddock documentation is correct, and
|
||||
produce as artifacts both a self-contained GHC binary package database and a
|
||||
set of Haddock documentation. (Not yet implemented.)
|
||||
|
||||
A `BuildPlan` may be converted into a bundle to be uploaded to Stackage Server.
|
||||
(Not yet implemented.)
|
||||
5557
allowed.txt
5557
allowed.txt
File diff suppressed because it is too large
Load Diff
@ -27,13 +27,3 @@ wget http://hackage.haskell.org/packages/archive/cabal-install/1.18.0.5/cabal-in
|
||||
tar zxfv cabal-install-1.18.0.5.tar.gz
|
||||
cd cabal-install-1.18.0.5/
|
||||
bash bootstrap.sh
|
||||
cd ..
|
||||
git clone --recursive https://github.com/fpco/stackage
|
||||
cd stackage
|
||||
cabal update
|
||||
cabal install
|
||||
|
||||
stackage select && \
|
||||
stackage check && \
|
||||
stackage build && \
|
||||
stackage test
|
||||
|
||||
@ -1,6 +0,0 @@
|
||||
#!/bin/bash -ex
|
||||
|
||||
runghc app/stackage.hs select $*
|
||||
runghc app/stackage.hs check
|
||||
runghc app/stackage.hs build
|
||||
runghc app/stackage.hs test
|
||||
@ -1,22 +0,0 @@
|
||||
#!/bin/bash -ex
|
||||
|
||||
cabal update
|
||||
rm -rf patching/tarballs
|
||||
(cd patching/ && ./scripts/create-tarballs.sh)
|
||||
cabal install
|
||||
cabal install Cabal-$(cabal --version | sed -n 's@using version \(.*\) of the Cabal library@\1@p')
|
||||
./dist/build/stackage/stackage select $*
|
||||
./dist/build/stackage/stackage check
|
||||
./dist/build/stackage/stackage build
|
||||
./dist/build/stackage/stackage test
|
||||
|
||||
cabal install http-client --force-reinstalls
|
||||
|
||||
for f in inclusive exclusive
|
||||
do
|
||||
cd $f
|
||||
|
||||
bash create-snapshot.sh
|
||||
|
||||
cd ..
|
||||
done
|
||||
2
patching/.gitignore
vendored
2
patching/.gitignore
vendored
@ -1,2 +0,0 @@
|
||||
/tarballs
|
||||
/work
|
||||
@ -1,25 +0,0 @@
|
||||
#!/bin/bash -ex
|
||||
|
||||
shopt -s nullglob
|
||||
|
||||
mkdir -p patches work
|
||||
|
||||
for f in work/*
|
||||
do
|
||||
(
|
||||
cd $f
|
||||
PKG=$(basename $(pwd))
|
||||
cabal sdist
|
||||
cd ../..
|
||||
rm -rf tmp
|
||||
mkdir tmp
|
||||
cd tmp
|
||||
tar zxfv ../$f/dist/$PKG.tar.gz
|
||||
mv $PKG new
|
||||
cabal unpack $PKG
|
||||
mv $PKG orig
|
||||
diff -ruN orig new > ../patches/$PKG.patch || true
|
||||
cd ..
|
||||
rm -rf tmp
|
||||
)
|
||||
done
|
||||
@ -1,31 +0,0 @@
|
||||
#!/bin/bash -ex
|
||||
|
||||
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
|
||||
|
||||
(
|
||||
cd "$DIR/.."
|
||||
|
||||
shopt -s nullglob
|
||||
|
||||
mkdir -p tarballs
|
||||
|
||||
for f in patches/*
|
||||
do
|
||||
(
|
||||
PKG1=$(basename $f)
|
||||
PKG=${PKG1%.patch}
|
||||
rm -rf tmp
|
||||
mkdir tmp
|
||||
(
|
||||
cd tmp
|
||||
cabal unpack $PKG
|
||||
cd $PKG
|
||||
patch -p1 < ../../$f
|
||||
cabal sdist
|
||||
mv dist/$PKG.tar.gz ../../tarballs
|
||||
)
|
||||
rm -rf tmp
|
||||
)
|
||||
done
|
||||
|
||||
)
|
||||
@ -1,6 +0,0 @@
|
||||
#!/bin/bash -ex
|
||||
|
||||
mkdir -p work
|
||||
cd work
|
||||
rm -rf $1
|
||||
cabal unpack $1
|
||||
@ -1,176 +0,0 @@
|
||||
{-# LANGUAGE OverloadedStrings #-}
|
||||
import Control.Monad (filterM, when)
|
||||
import qualified Data.ByteString.Char8 as S8
|
||||
import qualified Data.ByteString.Lazy as L
|
||||
import qualified Data.ByteString.Lazy.Char8 as L8
|
||||
import Data.List (isInfixOf, isPrefixOf,
|
||||
sort)
|
||||
import Network.HTTP.Client
|
||||
import Network.HTTP.Client.MultipartFormData
|
||||
import System.Directory (doesDirectoryExist,
|
||||
getDirectoryContents)
|
||||
import System.Environment (getArgs, getEnv,
|
||||
getProgName)
|
||||
import System.Exit (exitFailure)
|
||||
import System.Exit (ExitCode (ExitSuccess))
|
||||
import System.FilePath (takeDirectory, (</>))
|
||||
import System.Process (createProcess, cwd,
|
||||
proc, waitForProcess)
|
||||
|
||||
main :: IO ()
|
||||
main = withManager defaultManagerSettings $ \m -> do
|
||||
args <- getArgs
|
||||
token <- readFile "/auth-token"
|
||||
(filepath, alias) <-
|
||||
case args of
|
||||
[x, y] -> return (x, y)
|
||||
_ -> do
|
||||
pn <- getProgName
|
||||
putStrLn $ concat
|
||||
[ "Usage: "
|
||||
, pn
|
||||
, " <filepath> <alias name>"
|
||||
]
|
||||
exitFailure
|
||||
|
||||
let uploadDocs = "exclusive" `isInfixOf` alias
|
||||
uploadHackageDistro = alias == "unstable-ghc78-exclusive"
|
||||
|
||||
putStrLn $ concat
|
||||
[ "Uploading "
|
||||
, filepath
|
||||
, " as "
|
||||
, alias
|
||||
]
|
||||
|
||||
req1 <- parseUrl "http://www.stackage.org/upload"
|
||||
let formData =
|
||||
[ partBS "alias" $ S8.pack alias
|
||||
, partFileSource "stackage" filepath
|
||||
]
|
||||
req2 <- formDataBody formData req1
|
||||
let req3 = req2
|
||||
{ method = "PUT"
|
||||
, requestHeaders =
|
||||
[ ("Authorization", S8.pack token)
|
||||
, ("Accept", "application/json")
|
||||
] ++ requestHeaders req2
|
||||
, redirectCount = 0
|
||||
, checkStatus = \_ _ _ -> Nothing
|
||||
}
|
||||
res <- httpLbs req3 m
|
||||
|
||||
snapid <-
|
||||
case lookup "x-stackage-ident" $ responseHeaders res of
|
||||
Just snapid -> do
|
||||
putStrLn $ "New ident: " ++ S8.unpack snapid
|
||||
return snapid
|
||||
Nothing -> error $ "An error occurred: " ++ show res
|
||||
|
||||
when uploadDocs $ do
|
||||
putStrLn "Generating index file"
|
||||
let root = takeDirectory filepath </> "haddock"
|
||||
contents <- getDirectoryContents root
|
||||
dirs <- filterM (\n -> doesDirectoryExist $ root </> n)
|
||||
$ filter (not . ("." `isPrefixOf`))
|
||||
$ sort contents
|
||||
writeFile (root </> "index.html") $ mkIndex (S8.unpack snapid) dirs
|
||||
writeFile (root </> "style.css") styleCss
|
||||
|
||||
putStrLn "Creating tarball"
|
||||
(Nothing, Nothing, Nothing, ph) <- createProcess
|
||||
(proc "tar" $ "cJf" : "haddock.tar.xz" : "index.html" : "style.css" : dirs)
|
||||
{ cwd = Just root
|
||||
}
|
||||
ec <- waitForProcess ph
|
||||
if ec == ExitSuccess
|
||||
then putStrLn "Haddock tarball generated"
|
||||
else error "Error generating Haddock tarball"
|
||||
|
||||
putStrLn "Uploading Haddocks"
|
||||
|
||||
req1 <- parseUrl $ "http://www.stackage.org/upload-haddock/"
|
||||
++ S8.unpack snapid
|
||||
let formData =
|
||||
[ partFileSource "tarball" $ root </> "haddock.tar.xz"
|
||||
]
|
||||
req2 <- formDataBody formData req1
|
||||
let req3 = req2
|
||||
{ method = "PUT"
|
||||
, requestHeaders =
|
||||
[ ("Authorization", S8.pack token)
|
||||
, ("Accept", "application/json")
|
||||
] ++ requestHeaders req2
|
||||
, redirectCount = 0
|
||||
, checkStatus = \_ _ _ -> Nothing
|
||||
}
|
||||
httpLbs req3 m >>= print
|
||||
|
||||
when uploadHackageDistro $ do
|
||||
lbs <- L.readFile $ takeDirectory filepath </> "build-plan.csv"
|
||||
let req = "http://hackage.haskell.org/distro/Stackage/packages.csv"
|
||||
{ requestHeaders = [("Content-Type", "text/csv")]
|
||||
, requestBody = RequestBodyLBS $ L.intercalate "\n" $ L8.lines lbs
|
||||
, checkStatus = \_ _ _ -> Nothing
|
||||
, method = "PUT"
|
||||
}
|
||||
httpLbs req m >>= print
|
||||
|
||||
mkIndex :: String -> [String] -> String
|
||||
mkIndex snapid dirs = concat
|
||||
[ "<!DOCTYPE html>\n<html lang='en'><head><title>Haddocks index</title>"
|
||||
, "<link rel='stylesheet' href='https://maxcdn.bootstrapcdn.com/bootstrap/3.2.0/css/bootstrap.min.css'>"
|
||||
, "<link rel='stylesheet' href='style.css'>"
|
||||
, "<link rel='shortcut icon' href='http://www.stackage.org/static/img/favicon.ico' />"
|
||||
, "</head>"
|
||||
, "<body><div class='container'>"
|
||||
, "<div class='row'><div class='span12 col-md-12'>"
|
||||
, "<h1>Haddock documentation index</h1>"
|
||||
, "<p class='return'><a href=\"http://www.stackage.org/stackage/"
|
||||
, snapid
|
||||
, "\">Return to snapshot</a></p><ul>"
|
||||
, concatMap toLI dirs
|
||||
, "</ul></div></div></div></body></html>"
|
||||
]
|
||||
where
|
||||
toLI name = concat
|
||||
[ "<li><a href='"
|
||||
, name
|
||||
, "/index.html'>"
|
||||
, name
|
||||
, "</a></li>"
|
||||
]
|
||||
|
||||
styleCss :: String
|
||||
styleCss = concat
|
||||
[ "@media (min-width: 530px) {"
|
||||
, "ul { -webkit-column-count: 2; -moz-column-count: 2; column-count: 2 }"
|
||||
, "}"
|
||||
, "@media (min-width: 760px) {"
|
||||
, "ul { -webkit-column-count: 3; -moz-column-count: 3; column-count: 3 }"
|
||||
, "}"
|
||||
, "ul {"
|
||||
, " margin-left: 0;"
|
||||
, " padding-left: 0;"
|
||||
, " list-style-type: none;"
|
||||
, "}"
|
||||
, "body {"
|
||||
, " background: #f0f0f0;"
|
||||
, " font-family: 'Lato', sans-serif;"
|
||||
, " text-shadow: 1px 1px 1px #ffffff;"
|
||||
, " font-size: 20px;"
|
||||
, " line-height: 30px;"
|
||||
, " padding-bottom: 5em;"
|
||||
, "}"
|
||||
, "h1 {"
|
||||
, " font-weight: normal;"
|
||||
, " color: #06537d;"
|
||||
, " font-size: 45px;"
|
||||
, "}"
|
||||
, ".return a {"
|
||||
, " color: #06537d;"
|
||||
, " font-style: italic;"
|
||||
, "}"
|
||||
, ".return {"
|
||||
, " margin-bottom: 1em;"
|
||||
, "}"]
|
||||
Loading…
Reference in New Issue
Block a user