diff --git a/docs/API.md b/docs/API.md index 1deaf19..08c91c1 100644 --- a/docs/API.md +++ b/docs/API.md @@ -20,15 +20,15 @@ awsCI { connectAccesskey = "your-access-key" ``` -|Bucket operations|Object Operations| -|:---|:---| -|[`listBuckets`](#listBuckets) |[`getObject`](#getObject)| -|[`makeBucket`](#makeBucket)|[`putObject`](#putObject)| -|[`removeBucket`](#removeBucket)|[`fGetObject`](#fGetObject)| -|[`listObjects`](#listObjects)|[`fPutObject`](#fPutObject)| -|[`listObjectsV1`](#listObjectsV1)|[`copyObject`](#copyObject)| -|[`listIncompleteUploads`](#listIncompleteUploads)|[`removeObject`](#removeObject)| -|[`bucketExists`](#bucketExists)|| +|Bucket operations|Object Operations|Presigned Operations| +|:---|:---|:---| +|[`listBuckets`](#listBuckets) |[`getObject`](#getObject)|[`presignedGetObjectUrl`](#presignedGetObjectUrl)| +|[`makeBucket`](#makeBucket)|[`putObject`](#putObject)|[`presignedPutObjectUrl`](#presignedPutObjectUrl)| +|[`removeBucket`](#removeBucket)|[`fGetObject`](#fGetObject)|[`presignedPostPolicy`](#presignedPostPolicy)| +|[`listObjects`](#listObjects)|[`fPutObject`](#fPutObject)|| +|[`listObjectsV1`](#listObjectsV1)|[`copyObject`](#copyObject)|| +|[`listIncompleteUploads`](#listIncompleteUploads)|[`removeObject`](#removeObject)|| +|[`bucketExists`](#bucketExists)||| ## 1. Connecting and running operations on the storage service @@ -685,9 +685,216 @@ In the expression `bucketExists bucketName` the parameters are: | `bucketName` | _Bucket_ (alias for `Text`) | Name of the bucket | - +## 4. Presigned operations - + +### presignedGetObjectUrl :: Bucket -> Object -> UrlExpiry -> Query -> RequestHeaders -> Minio ByteString + +Generate a URL with authentication signature to GET (download) an +object. All extra query parameters and headers passed here will be +signed and are required when the generated URL is used. Query +parameters could be used to change the response headers sent by the +server. Headers can be used to set Etag match conditions among others. + +For a list of possible request parameters and headers, please refer +to the GET object REST API AWS S3 documentation. + +__Parameters__ + +In the expression `presignedGetObjectUrl bucketName objectName expiry queryParams headers` +the parameters are: + +|Param |Type |Description | +|:---|:---| :---| +| `bucketName` | _Bucket_ (alias for `Text`) | Name of the bucket | +| `objectName` | _Object_ (alias for `Text`) | Name of the object | +| `expiry` | _UrlExpiry_ (alias for `Int`) | Url expiry time in seconds | +| `queryParams` | _Query_ (from package `http-types:Network.HTTP.Types`) | Query parameters to add to the URL | +| `headers` | _RequestHeaders_ (from package `http-types:Network.HTTP.Types` | Request headers that would be used with the URL | + +__Return Value__ + +Returns the generated URL - it will include authentication +information. + +|Return type |Description | +|:---|:---| +| _ByteString_ | Generated presigned URL | + +__Example__ + +```haskell +{-# Language OverloadedStrings #-} + +import Network.Minio +import qualified Data.ByteString.Char8 as B + +main :: IO () +main = do + let + bucket = "mybucket" + object = "myobject" + + res <- runMinio minioPlayCI $ do + -- Set a 7 day expiry for the URL + presignedGetObjectUrl bucket object (7*24*3600) [] [] + + -- Print the URL on success. + putStrLn $ either + (("Failed to generate URL: " ++) . show) + B.unpack + res +``` + + +### presignedPutObjectUrl :: Bucket -> Object -> UrlExpiry -> RequestHeaders -> Minio ByteString + +Generate a URL with authentication signature to PUT (upload) an +object. Any extra headers if passed, are signed, and so they are +required when the URL is used to upload data. This could be used, for +example, to set user-metadata on the object. + +For a list of possible headers to pass, please refer to the PUT object +REST API AWS S3 documentation. + +__Parameters__ + +In the expression `presignedPutObjectUrl bucketName objectName expiry headers` +the parameters are: + +|Param |Type |Description | +|:---|:---| :---| +| `bucketName` | _Bucket_ (alias for `Text`) | Name of the bucket | +| `objectName` | _Object_ (alias for `Text`) | Name of the object | +| `expiry` | _UrlExpiry_ (alias for `Int`) | Url expiry time in seconds | +| `headers` | _RequestHeaders_ (from package `http-types:Network.HTTP.Types` | Request headers that would be used with the URL | + +__Return Value__ + +Returns the generated URL - it will include authentication +information. + +|Return type |Description | +|:---|:---| +| _ByteString_ | Generated presigned URL | + +__Example__ + +```haskell +{-# Language OverloadedStrings #-} + +import Network.Minio +import qualified Data.ByteString.Char8 as B + +main :: IO () +main = do + let + bucket = "mybucket" + object = "myobject" + + res <- runMinio minioPlayCI $ do + -- Set a 7 day expiry for the URL + presignedPutObjectUrl bucket object (7*24*3600) [] [] + + -- Print the URL on success. + putStrLn $ either + (("Failed to generate URL: " ++) . show) + B.unpack + res +``` + + +### presignedPostPolicy :: PostPolicy -> Minio (ByteString, Map.Map Text ByteString) + +Generate a presigned URL and POST policy to upload files via a POST +request. This is intended for browser uploads and generates form data +that should be submitted in the request. + +The `PostPolicy` argument is created using the `newPostPolicy` function: + +#### newPostPolicy :: UTCTime -> [PostPolicyCondition] -> Either PostPolicyError PostPolicy + +In the expression `newPostPolicy expirationTime conditions` the parameters are: + +|Param | Type| Description | +|:---|:---|:---| +| `expirationTime` | _UTCTime_ (from package `time:Data.Time.UTCTime`) | The expiration time for the policy | +| `conditions` | _[PostPolicyConditions]_ | List of conditions to be added to the policy | + +The policy conditions are created using various helper functions - +please refer to the Haddocks for details. + +Since conditions are validated by `newPostPolicy` it returns an +`Either` value. + +__Return Value__ + +`presignedPostPolicy` returns a 2-tuple - the generated URL and a map +containing the form-data that should be submitted with the request. + +__Example__ + +```haskell +{-# Language OverloadedStrings #-} + +import Network.Minio + +import qualified Data.ByteString as B +import qualified Data.ByteString.Char8 as Char8 +import qualified Data.Map.Strict as Map +import qualified Data.Text.Encoding as Enc +import qualified Data.Time as Time + +main :: IO () +main = do + now <- Time.getCurrentTime + let + bucket = "mybucket" + object = "myobject" + + -- set an expiration time of 10 days + expireTime = Time.addUTCTime (3600 * 24 * 10) now + + -- create a policy with expiration time and conditions - since the + -- conditions are validated, newPostPolicy returns an Either value + policyE = newPostPolicy expireTime + [ -- set the object name condition + ppCondKey "photos/my-object" + -- set the bucket name condition + , ppCondBucket "my-bucket" + -- set the size range of object as 1B to 10MiB + , ppCondContentLengthRange 1 (10*1024*1024) + -- set content type as jpg image + , ppCondContentType "image/jpeg" + -- on success set the server response code to 200 + , ppCondSuccessActionStatus 200 + ] + + case policyE of + Left err -> putStrLn $ show err + Right policy -> do + res <- runMinio minioPlayCI $ do + (url, formData) <- presignedPostPolicy policy + + -- a curl command is output to demonstrate using the generated + -- URL and form-data + let + formFn (k, v) = B.concat ["-F ", Enc.encodeUtf8 k, "=", + "'", v, "'"] + formOptions = B.intercalate " " $ map formFn $ Map.toList formData + + + return $ B.intercalate " " $ + ["curl", formOptions, "-F file=@/tmp/photo.jpg", url] + + case res of + Left e -> putStrLn $ "post-policy error: " ++ (show e) + Right cmd -> do + putStrLn $ "Put a photo at /tmp/photo.jpg and run command:\n" + + -- print the generated curl command + Char8.putStrLn cmd +``` diff --git a/examples/PresignedGetObject.hs b/examples/PresignedGetObject.hs new file mode 100755 index 0000000..2bc49cd --- /dev/null +++ b/examples/PresignedGetObject.hs @@ -0,0 +1,82 @@ +#!/usr/bin/env stack +-- stack --resolver lts-9.1 runghc --package minio-hs + +-- +-- Minio Haskell SDK, (C) 2017 Minio, Inc. +-- +-- Licensed under the Apache License, Version 2.0 (the "License"); +-- you may not use this file except in compliance with the License. +-- You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, software +-- distributed under the License is distributed on an "AS IS" BASIS, +-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +-- See the License for the specific language governing permissions and +-- limitations under the License. +-- + +{-# LANGUAGE OverloadedStrings #-} +import Network.Minio + +import Control.Monad.IO.Class (liftIO) +import qualified Data.ByteString.Char8 as B +import Data.CaseInsensitive (original) +import qualified Data.Conduit.Combinators as CC +import qualified Data.Text.Encoding as E + +-- | The following example uses minio's play server at +-- https://play.minio.io:9000. The endpoint and associated +-- credentials are provided via the libary constant, +-- +-- > minioPlayCI :: ConnectInfo +-- + +main :: IO () +main = do + let + bucket = "my-bucket" + object = "my-object" + kb15 = 15*1024 + + -- Set query parameter to modify content disposition response + -- header + queryParam = [("response-content-disposition", + Just "attachment; filename=\"your-filename.txt\"")] + + res <- runMinio minioPlayCI $ do + liftIO $ B.putStrLn "Upload a file that we will fetch with a presigned URL..." + putObject bucket object (CC.repeat "a") (Just kb15) + liftIO $ putStrLn $ "Done. Object created at: my-bucket/my-object" + + -- Extract Etag of uploaded object + (ObjectInfo _ _ etag _) <- statObject bucket object + + -- Set header to add an if-match constraint - this makes sure + -- the fetching fails if the object is changed on the server + let headers = [("If-Match", E.encodeUtf8 etag)] + + -- Generate a URL with 7 days expiry time - note that the headers + -- used above must be added to the request with the signed URL + -- generated. + url <- presignedGetObjectUrl "my-bucket" "my-object" (7*24*3600) + queryParam headers + + return (headers, etag, url) + + case res of + Left e -> putStrLn $ "presignedPutObject URL failed." ++ show e + Right (headers, etag, url) -> do + + -- We generate a curl command to demonstrate usage of the signed + -- URL. + let + hdrOpt (k, v) = B.concat ["-H '", original k, ": ", v, "'"] + curlCmd = B.intercalate " " $ + ["curl --fail"] ++ map hdrOpt headers ++ + ["-o /tmp/myfile", B.concat ["'", url, "'"]] + + putStrLn $ "The following curl command would use the presigned " ++ + "URL to fetch the object and write it to \"/tmp/myfile\":" + B.putStrLn curlCmd diff --git a/examples/PresignedPutObject.hs b/examples/PresignedPutObject.hs new file mode 100755 index 0000000..ed8612d --- /dev/null +++ b/examples/PresignedPutObject.hs @@ -0,0 +1,59 @@ +#!/usr/bin/env stack +-- stack --resolver lts-9.1 runghc --package minio-hs + +-- +-- Minio Haskell SDK, (C) 2017 Minio, Inc. +-- +-- Licensed under the Apache License, Version 2.0 (the "License"); +-- you may not use this file except in compliance with the License. +-- You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, software +-- distributed under the License is distributed on an "AS IS" BASIS, +-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +-- See the License for the specific language governing permissions and +-- limitations under the License. +-- + +{-# LANGUAGE OverloadedStrings #-} +import Network.Minio + +import qualified Data.ByteString.Char8 as B +import Data.CaseInsensitive (original) + +-- | The following example uses minio's play server at +-- https://play.minio.io:9000. The endpoint and associated +-- credentials are provided via the libary constant, +-- +-- > minioPlayCI :: ConnectInfo +-- + +main :: IO () +main = do + let + -- Use headers to set user-metadata - note that this header will + -- need to be set when the URL is used to make an upload. + headers = [("x-amz-meta-url-creator", + "minio-hs-presigned-put-example")] + res <- runMinio minioPlayCI $ do + + -- generate a URL with 7 days expiry time + presignedPutObjectURL "my-bucket" "my-object" (7*24*3600) headers + + case res of + Left e -> putStrLn $ "presignedPutObject URL failed." ++ show e + Right url -> do + + -- We generate a curl command to demonstrate usage of the signed + -- URL. + let + hdrOpt (k, v) = B.concat ["-H '", original k, ": ", v, "'"] + curlCmd = B.intercalate " " $ + ["curl "] ++ map hdrOpt headers ++ + ["-T /tmp/myfile", B.concat ["'", url, "'"]] + + B.putStrLn $ "The following curl command would use the presigned " ++ + "URL to upload the file at \"/tmp/myfile\":" + B.putStrLn curlCmd diff --git a/examples/PutObject.hs b/examples/PutObject.hs index 4b086f4..78bd3b5 100755 --- a/examples/PutObject.hs +++ b/examples/PutObject.hs @@ -17,11 +17,10 @@ -- limitations under the License. -- -{-# Language OverloadedStrings #-} +{-# LANGUAGE OverloadedStrings #-} import Network.Minio import qualified Data.Conduit.Combinators as CC -import Prelude -- | The following example uses minio's play server at -- https://play.minio.io:9000. The endpoint and associated @@ -42,7 +41,7 @@ main = do res1 <- runMinio minioPlayCI $ putObject bucket object (CC.repeat "a") (Just kb15) case res1 of - Left e -> putStrLn $ "putObject failed." ++ show e + Left e -> putStrLn $ "putObject failed." ++ show e Right () -> putStrLn "putObject succeeded." @@ -50,5 +49,5 @@ main = do res2 <- runMinio minioPlayCI $ fPutObject bucket object localFile case res2 of - Left e -> putStrLn $ "fPutObject failed." ++ show e + Left e -> putStrLn $ "fPutObject failed." ++ show e Right () -> putStrLn "fPutObject succeeded." diff --git a/src/Network/Minio.hs b/src/Network/Minio.hs index dfb8374..38dabdd 100644 --- a/src/Network/Minio.hs +++ b/src/Network/Minio.hs @@ -79,9 +79,9 @@ module Network.Minio -- * Presigned Operations ------------------------- , UrlExpiry - , presignedPutObjectURL - , presignedGetObjectURL - , presignedHeadObjectURL + , presignedPutObjectUrl + , presignedGetObjectUrl + , presignedHeadObjectUrl , PostPolicyCondition , ppCondBucket @@ -102,11 +102,11 @@ module Network.Minio This module exports the high-level Minio API for object storage. -} -import qualified Data.Conduit as C -import qualified Data.Conduit.Binary as CB +import qualified Data.Conduit as C +import qualified Data.Conduit.Binary as CB import qualified Data.Conduit.Combinators as CC -import Data.Default (def) -import qualified Data.Map as Map +import Data.Default (def) +import qualified Data.Map as Map import Lib.Prelude diff --git a/src/Network/Minio/PresignedOperations.hs b/src/Network/Minio/PresignedOperations.hs index 698790a..e3568b6 100644 --- a/src/Network/Minio/PresignedOperations.hs +++ b/src/Network/Minio/PresignedOperations.hs @@ -16,10 +16,10 @@ module Network.Minio.PresignedOperations ( UrlExpiry - , makePresignedURL - , presignedPutObjectURL - , presignedGetObjectURL - , presignedHeadObjectURL + , makePresignedUrl + , presignedPutObjectUrl + , presignedGetObjectUrl + , presignedHeadObjectUrl , PostPolicyCondition(..) , ppCondBucket @@ -54,17 +54,17 @@ import Network.Minio.Errors import Network.Minio.Sign.V4 -- | Generate a presigned URL. This function allows for advanced usage --- - for simple cases prefer the `presigned*URL` functions. +-- - for simple cases prefer the `presigned*Url` functions. -- -- If region is Nothing, it is picked up from the connection -- information (no check of bucket existence is performed). -- -- All extra query parameters or headers are signed, and therefore are -- required to be sent when the generated URL is actually used. -makePresignedURL :: UrlExpiry -> HT.Method -> Maybe Bucket -> Maybe Object +makePresignedUrl :: UrlExpiry -> HT.Method -> Maybe Bucket -> Maybe Object -> Maybe Region -> HT.Query -> HT.RequestHeaders -> Minio ByteString -makePresignedURL expiry method bucket object region extraQuery extraHeaders = do +makePresignedUrl expiry method bucket object region extraQuery extraHeaders = do when (expiry > 7*24*3600 || expiry < 0) $ throwM $ MErrVInvalidUrlExpiry expiry @@ -98,10 +98,10 @@ makePresignedURL expiry method bucket object region extraQuery extraHeaders = do -- -- For a list of possible headers to pass, please refer to the PUT -- object REST API AWS S3 documentation. -presignedPutObjectURL :: Bucket -> Object -> UrlExpiry -> HT.RequestHeaders +presignedPutObjectUrl :: Bucket -> Object -> UrlExpiry -> HT.RequestHeaders -> Minio ByteString -presignedPutObjectURL bucket object expirySeconds extraHeaders = - makePresignedURL expirySeconds HT.methodPut +presignedPutObjectUrl bucket object expirySeconds extraHeaders = + makePresignedUrl expirySeconds HT.methodPut (Just bucket) (Just object) Nothing [] extraHeaders -- | Generate a URL with authentication signature to GET (download) an @@ -113,10 +113,10 @@ presignedPutObjectURL bucket object expirySeconds extraHeaders = -- -- For a list of possible request parameters and headers, please refer -- to the GET object REST API AWS S3 documentation. -presignedGetObjectURL :: Bucket -> Object -> UrlExpiry -> HT.Query +presignedGetObjectUrl :: Bucket -> Object -> UrlExpiry -> HT.Query -> HT.RequestHeaders -> Minio ByteString -presignedGetObjectURL bucket object expirySeconds extraQuery extraHeaders = - makePresignedURL expirySeconds HT.methodGet +presignedGetObjectUrl bucket object expirySeconds extraQuery extraHeaders = + makePresignedUrl expirySeconds HT.methodGet (Just bucket) (Just object) Nothing extraQuery extraHeaders -- | Generate a URL with authentication signature to make a HEAD @@ -126,10 +126,10 @@ presignedGetObjectURL bucket object expirySeconds extraQuery extraHeaders = -- -- For a list of possible headers to pass, please refer to the HEAD -- object REST API AWS S3 documentation. -presignedHeadObjectURL :: Bucket -> Object -> UrlExpiry +presignedHeadObjectUrl :: Bucket -> Object -> UrlExpiry -> HT.RequestHeaders -> Minio ByteString -presignedHeadObjectURL bucket object expirySeconds extraHeaders = - makePresignedURL expirySeconds HT.methodHead +presignedHeadObjectUrl bucket object expirySeconds extraHeaders = + makePresignedUrl expirySeconds HT.methodHead (Just bucket) (Just object) Nothing [] extraHeaders -- | Represents individual conditions in a Post Policy document. @@ -239,7 +239,7 @@ showPostPolicy :: PostPolicy -> ByteString showPostPolicy = toS . Json.encode -- | Generate a presigned URL and POST policy to upload files via a --- browser. On success, this function returns a URL and a POST +-- browser. On success, this function returns a URL and POST -- form-data. presignedPostPolicy :: PostPolicy -> Minio (ByteString, Map.Map Text ByteString) diff --git a/test/LiveServer.hs b/test/LiveServer.hs index 9426e6d..c810ab1 100644 --- a/test/LiveServer.hs +++ b/test/LiveServer.hs @@ -433,7 +433,7 @@ liveServerUnitTests = testGroup "Unit tests against a live server" forM_ [src, copyObj] (removeObject bucket) - , presignedURLFunTest + , presignedUrlFunTest , presignedPostPolicyFunTest ] @@ -505,8 +505,8 @@ basicTests = funTestWithBucket "Basic tests" $ \step bucket -> do step "delete object" deleteObject bucket object -presignedURLFunTest :: TestTree -presignedURLFunTest = funTestWithBucket "presigned URL tests" $ +presignedUrlFunTest :: TestTree +presignedUrlFunTest = funTestWithBucket "presigned Url tests" $ \step bucket -> do let obj = "mydir/myput" obj2 = "mydir1/myfile1" @@ -514,8 +514,8 @@ presignedURLFunTest = funTestWithBucket "presigned URL tests" $ -- manager for http requests mgr <- liftIO $ NC.newManager NC.tlsManagerSettings - step "PUT object presigned URL - makePresignedURL" - putUrl <- makePresignedURL 3600 HT.methodPut (Just bucket) + step "PUT object presigned URL - makePresignedUrl" + putUrl <- makePresignedUrl 3600 HT.methodPut (Just bucket) (Just obj) (Just "us-east-1") [] [] let size1 = 1000 :: Int64 @@ -526,8 +526,8 @@ presignedURLFunTest = funTestWithBucket "presigned URL tests" $ liftIO $ (NC.responseStatus putResp == HT.status200) @? "presigned PUT failed" - step "GET object presigned URL - makePresignedURL" - getUrl <- makePresignedURL 3600 HT.methodGet (Just bucket) + step "GET object presigned URL - makePresignedUrl" + getUrl <- makePresignedUrl 3600 HT.methodGet (Just bucket) (Just obj) (Just "us-east-1") [] [] getResp <- getR mgr getUrl @@ -540,39 +540,39 @@ presignedURLFunTest = funTestWithBucket "presigned URL tests" $ "presigned put and get got mismatched data" step "PUT object presigned - presignedPutObjectURL" - putUrl2 <- presignedPutObjectURL bucket obj2 3600 [] + putUrl2 <- presignedPutObjectUrl bucket obj2 3600 [] let size2 = 1200 testFile <- mkRandFile size2 putResp2 <- putR size2 testFile mgr putUrl2 liftIO $ (NC.responseStatus putResp2 == HT.status200) @? - "presigned PUT failed (presignedPutObjectURL)" + "presigned PUT failed (presignedPutObjectUrl)" - step "HEAD object presigned URL - presignedHeadObjectURL" - headUrl <- presignedHeadObjectURL bucket obj2 3600 [] + step "HEAD object presigned URL - presignedHeadObjectUrl" + headUrl <- presignedHeadObjectUrl bucket obj2 3600 [] headResp <- do req <- NC.parseRequest $ toS headUrl NC.httpLbs (req {NC.method = HT.methodHead}) mgr liftIO $ (NC.responseStatus headResp == HT.status200) @? - "presigned HEAD failed (presignedHeadObjectURL)" + "presigned HEAD failed (presignedHeadObjectUrl)" -- check that header info is accurate let h = Map.fromList $ NC.responseHeaders headResp cLen = Map.findWithDefault "0" HT.hContentLength h liftIO $ (cLen == show size2) @? "Head req returned bad content length" - step "GET object presigned URL - presignedGetObjectURL" - getUrl2 <- presignedGetObjectURL bucket obj2 3600 [] [] + step "GET object presigned URL - presignedGetObjectUrl" + getUrl2 <- presignedGetObjectUrl bucket obj2 3600 [] [] getResp2 <- getR mgr getUrl2 liftIO $ (NC.responseStatus getResp2 == HT.status200) @? - "presigned GET failed (presignedGetObjectURL)" + "presigned GET failed (presignedGetObjectUrl)" -- read content from file to compare with response above bs2 <- CB.sourceFile testFile $$ CB.sinkLbs liftIO $ (bs2 == NC.responseBody getResp2) @? - "presigned put and get got mismatched data (presigned*URL)" + "presigned put and get got mismatched data (presigned*Url)" mapM_ (removeObject bucket) [obj, obj2] @@ -589,7 +589,7 @@ presignedURLFunTest = funTestWithBucket "presigned URL tests" $ NC.httpLbs req mgr presignedPostPolicyFunTest :: TestTree -presignedPostPolicyFunTest = funTestWithBucket "presigned URL tests" $ +presignedPostPolicyFunTest = funTestWithBucket "Presigned Post Policy tests" $ \step bucket -> do step "presignedPostPolicy basic test"