From 8cb47817f6e9c6e602c284f3a6eac69601349cc0 Mon Sep 17 00:00:00 2001 From: Nicola Murino Date: Sat, 20 Jun 2020 12:38:04 +0200 Subject: [PATCH] Add API endpoint to set current quota Fixes #130 --- docker/README.md | 4 +- docker/sftpgo/alpine/README.md | 7 +- docker/sftpgo/debian/README.md | 2 +- examples/ldapauth/README.md | 8 +- examples/ldapauthserver/README.md | 6 +- examples/rest-api-cli/README.md | 142 +++++++++------ examples/rest-api-cli/sftpgo_api_cli.py | 32 ++++ httpd/api_folder.go | 9 +- httpd/api_quota.go | 114 ++++++++++-- httpd/api_user.go | 24 +-- httpd/api_utils.go | 52 +++++- httpd/httpd.go | 46 ++--- httpd/httpd_test.go | 230 +++++++++++++++++++++--- httpd/internal_test.go | 13 +- httpd/router.go | 2 + httpd/schema/openapi.yaml | 198 +++++++++++++++++++- 16 files changed, 746 insertions(+), 143 deletions(-) diff --git a/docker/README.md b/docker/README.md index 462e9e20..478a1762 100644 --- a/docker/README.md +++ b/docker/README.md @@ -1,5 +1,5 @@ -## Dockerfile examples +# Dockerfile examples Sample Dockerfiles for `sftpgo` daemon and the REST API CLI. -We don't want to add a `Dockerfile` for each single `sftpgo` configuration options or data provider. You can use the docker configurations here as starting point that you can customize to run `sftpgo` with [Docker](http://www.docker.io "Docker"). \ No newline at end of file +We don't want to add a `Dockerfile` for each single `sftpgo` configuration options or data provider. You can use the docker configurations here as starting point that you can customize to run `sftpgo` with [Docker](http://www.docker.io "Docker"). diff --git a/docker/sftpgo/alpine/README.md b/docker/sftpgo/alpine/README.md index c3f66be8..194923a3 100644 --- a/docker/sftpgo/alpine/README.md +++ b/docker/sftpgo/alpine/README.md @@ -2,8 +2,10 @@ This DockerFile is made to build image to host multiple instances of SFTPGo started with different users. -### Example +## Example + > 1003 is a custom uid:gid for this instance of SFTPGo + ```bash # Prereq on docker host sudo groupadd -g 1003 sftpgrp && \ @@ -48,7 +50,8 @@ The script `entrypoint.sh` makes sure to correct the permissions of directories Several images can be run with different parameters. -### Custom systemd script +## Custom systemd script + An example of systemd script is present [here](sftpgo.service), with `Environment` parameter to set `PUID` and `GUID` `WorkingDirectory` parameter must be exist with one file in this directory like `sftpgo-${PUID}.env` corresponding to the variable file for SFTPGo instance. diff --git a/docker/sftpgo/debian/README.md b/docker/sftpgo/debian/README.md index e16e6f8e..567655b5 100644 --- a/docker/sftpgo/debian/README.md +++ b/docker/sftpgo/debian/README.md @@ -1,4 +1,4 @@ -## Dockerfile based on Debian stable +# Dockerfile based on Debian stable Please read the comments inside the `Dockerfile` to learn how to customize things for your setup. diff --git a/examples/ldapauth/README.md b/examples/ldapauth/README.md index 201fa50e..7b29a573 100644 --- a/examples/ldapauth/README.md +++ b/examples/ldapauth/README.md @@ -1,4 +1,4 @@ -## LDAPAuth +# LDAPAuth This is an example for an external authentication program. It performs authentication against an LDAP server. It is tested against [389ds](https://directory.fedoraproject.org/) and can be used as starting point to authenticate using any LDAP server including Active Directory. @@ -6,13 +6,13 @@ It is tested against [389ds](https://directory.fedoraproject.org/) and can be us You need to change the LDAP connection parameters and the user search query to match your environment. You can build this example using the following command: -``` +```console go build -i -ldflags "-s -w" -o ldapauth ``` This program assumes that the 389ds schema was extended to add support for public keys using the following ldif file placed in `/etc/dirsrv/schema/98openssh-ldap.ldif`: -``` +```console dn: cn=schema changetype: modify add: attributetypes @@ -45,4 +45,4 @@ aci: (targetattr = "sshPublicKey") (version 3.0; acl "Allow members of sshpublic - ``` -Please feel free to send pull requests to improve this example authentication program, thanks! \ No newline at end of file +Please feel free to send pull requests to improve this example authentication program, thanks! diff --git a/examples/ldapauthserver/README.md b/examples/ldapauthserver/README.md index f188f378..61f3dad2 100644 --- a/examples/ldapauthserver/README.md +++ b/examples/ldapauthserver/README.md @@ -1,4 +1,4 @@ -## LDAPAuthServer +# LDAPAuthServer This is an example for an HTTP server to use as external authentication HTTP hook. It performs authentication against an LDAP server. It is tested against [389ds](https://directory.fedoraproject.org/) and can be used as starting point to authenticate using any LDAP server including Active Directory. @@ -6,6 +6,6 @@ It is tested against [389ds](https://directory.fedoraproject.org/) and can be us You can configure the server using the [ldapauth.toml](./ldapauth.toml) configuration file. You can build this example using the following command: -``` +```console go build -i -ldflags "-s -w" -o ldapauthserver -``` \ No newline at end of file +``` diff --git a/examples/rest-api-cli/README.md b/examples/rest-api-cli/README.md index cbe5a6dc..4074af65 100644 --- a/examples/rest-api-cli/README.md +++ b/examples/rest-api-cli/README.md @@ -1,4 +1,4 @@ -## REST API CLI client +# REST API CLI client `sftpgo_api_cli.py` is a very simple command line client for `SFTPGo` REST API written in python. @@ -10,26 +10,26 @@ It has the following requirements: You can see the usage with the following command: -``` +```console python sftpgo_api_cli.py --help ``` and -``` +```console python sftpgo_api_cli.py [sub-command] --help ``` Basically there is a sub command for each REST API and the following global arguments: - - `-d`, `--debug`, default disabled, print useful debug info. - - `-b`, `--base-url`, default `http://127.0.0.1:8080`. Base URL for SFTPGo REST API - - `-a`, `--auth-type`, HTTP auth type. Supported HTTP auth type are `basic` and `digest`. Default none - - `-u`, `--auth-user`, user for HTTP authentication - - `-p`, `--auth-password`, password for HTTP authentication - - `-i`, `--insecure`, enable to ignore verifying the SSL certificate. Default disabled - - `-t`, `--no-color`, disable color highligth for JSON responses. You need python pygments module 1.5 or above for this to work. Default disabled if pygments is found and you aren't on Windows, otherwise enabled. - - `-c`, `--color`, enable color highligth for JSON responses. You need python pygments module 1.5 or above for this to work. Default enabled if `pygments` is found and you aren't on Windows, otherwise disabled. Please read the note at the end of this doc for colors in Windows command prompt. +- `-d`, `--debug`, default disabled, print useful debug info. +- `-b`, `--base-url`, default `http://127.0.0.1:8080`. Base URL for SFTPGo REST API +- `-a`, `--auth-type`, HTTP auth type. Supported HTTP auth type are `basic` and `digest`. Default none +- `-u`, `--auth-user`, user for HTTP authentication +- `-p`, `--auth-password`, password for HTTP authentication +- `-i`, `--insecure`, enable to ignore verifying the SSL certificate. Default disabled +- `-t`, `--no-color`, disable color highligth for JSON responses. You need python pygments module 1.5 or above for this to work. Default disabled if pygments is found and you aren't on Windows, otherwise enabled. +- `-c`, `--color`, enable color highligth for JSON responses. You need python pygments module 1.5 or above for this to work. Default enabled if `pygments` is found and you aren't on Windows, otherwise disabled. Please read the note at the end of this doc for colors in Windows command prompt. For each subcommand `--help` shows the available arguments, try for example: @@ -39,11 +39,11 @@ Additionally it can convert users to the SFTPGo format from some supported users Let's see a sample usage for each REST API. -### Add user +## Add user Command: -``` +```console python sftpgo_api_cli.py add-user test_username --password "test_pwd" --home-dir="/tmp/test_home_dir" --uid 33 --gid 1000 --max-sessions 2 --quota-size 0 --quota-files 3 --permissions "list" "download" "upload" "delete" "rename" "create_dirs" "overwrite" --subdirs-permissions "/dir1::list,download" "/dir2::*" --upload-bandwidth 100 --download-bandwidth 60 --status 0 --expiration-date 2019-01-01 --allowed-ip "192.168.1.1/32" --fs S3 --s3-bucket test --s3-region eu-west-1 --s3-access-key accesskey --s3-access-secret secret --s3-endpoint "http://127.0.0.1:9000" --s3-storage-class Standard --s3-key-prefix "vfolder/" --s3-upload-part-size 10 --s3-upload-concurrency 4 --denied-login-methods "password" "keyboard-interactive" --allowed-extensions "/dir1::.jpg,.png" "/dir2::.rar,.png" --denied-extensions "/dir3::.zip,.rar" ``` @@ -135,11 +135,11 @@ Output: } ``` -### Update user +## Update user Command: -``` +```console python sftpgo_api_cli.py update-user 9576 test_username --password "test_pwd" --home-dir="/tmp/test_home_dir" --uid 0 --gid 33 --max-sessions 3 --quota-size 0 --quota-files 4 --permissions "*" --subdirs-permissions "/dir1::list,download,create_symlinks" --upload-bandwidth 90 --download-bandwidth 80 --status 1 --expiration-date "" --allowed-ip "" --denied-ip "192.168.1.0/24" --denied-login-methods "" --fs local --virtual-folders "/vdir1::/tmp/mapped1::-1::-1" "/vdir2::/tmp/mapped2::100::104857600" --allowed-extensions "" --denied-extensions "" ``` @@ -153,11 +153,11 @@ Output: } ``` -### Get user by id +## Get user by id Command: -``` +```console python sftpgo_api_cli.py get-user-by-id 9576 ``` @@ -226,11 +226,11 @@ Output: } ``` -### Get users +## Get users Command: -``` +```console python sftpgo_api_cli.py get-users --limit 1 --offset 0 --username test_username --order DESC ``` @@ -291,11 +291,11 @@ Output: ] ``` -### Get active connections +## Get active connections Command: -``` +```console python sftpgo_api_cli.py get-connections ``` @@ -325,11 +325,11 @@ Output: ] ``` -### Get folders +## Get folders Command: -``` +```console python sftpgo_api_cli.py get-folders --limit 1 --offset 0 --folder-path /tmp/mapped1 --order DESC ``` @@ -350,9 +350,9 @@ Output: ] ``` -### Add folder +## Add folder -``` +```console python sftpgo_api_cli.py add-folder /tmp/mapped_folder ``` @@ -368,11 +368,11 @@ Output: } ``` -### Close connection +## Close connection Command: -``` +```console python sftpgo_api_cli.py close-connection f82cfec6a391ad673edd4ae9a144f32ccb59456139f8e1185b070134fffbab7c ``` @@ -386,19 +386,19 @@ Output: } ``` -### Get quota scans +## Get quota scans Command: -``` +```console python sftpgo_api_cli.py get-quota-scans ``` -### Start quota scan +## Start quota scan Command: -``` +```console python sftpgo_api_cli.py start-quota-scan test_username ``` @@ -412,19 +412,19 @@ Output: } ``` -### Get folder quota scans +## Get folder quota scans Command: -``` +```console python sftpgo_api_cli.py get-folders-quota-scans ``` -### Start folder quota scan +## Start folder quota scan Command: -``` +```console python sftpgo_api_cli.py start-folder-quota-scan /tmp/mapped_folder ``` @@ -438,11 +438,47 @@ Output: } ``` -### Delete user +## Update quota usage Command: +```console +python sftpgo_api_cli.py -d update-quota-usage a -S 123 -F 1 -M reset ``` + +Output: + +```json +{ + "error": "", + "message": "Quota updated", + "status": 200 +} +``` + +## Update folder quota usage + +Command: + +```console +python sftpgo_api_cli.py -d update-quota-usage /tmp/mapped_folder -S 123 -F 1 -M add +``` + +Output: + +```json +{ + "error": "", + "message": "Quota updated", + "status": 200 +} +``` + +## Delete user + +Command: + +```console python sftpgo_api_cli.py delete-user 9576 ``` @@ -456,9 +492,9 @@ Output: } ``` -### Delete folder +## Delete folder -``` +```console python sftpgo_api_cli.py delete-folder /tmp/mapped_folder ``` @@ -472,11 +508,11 @@ Output: } ``` -### Get version +## Get version Command: -``` +```console python sftpgo_api_cli.py get-version ``` @@ -490,11 +526,11 @@ Output: } ``` -### Get provider status +## Get provider status Command: -``` +```console python sftpgo_api_cli.py get-provider-status ``` @@ -508,11 +544,11 @@ Output: } ``` -### Backup data +## Backup data Command: -``` +```console python sftpgo_api_cli.py dumpdata backup.json --indent 1 ``` @@ -526,11 +562,11 @@ Output: } ``` -### Restore data +## Restore data Command: -``` +```console python sftpgo_api_cli.py loaddata /app/data/backups/backup.json --scan-quota 2 --mode 0 ``` @@ -544,7 +580,7 @@ Output: } ``` -### Convert users from other stores +## Convert users from other stores You can convert users to the SFTPGo format from the following users stores: @@ -554,21 +590,21 @@ You can convert users to the SFTPGo format from the following users stores: For details give a look at the `convert-users` subcommand usage: -``` +```console python sftpgo_api_cli.py convert-users --help ``` Let's see some examples: -``` +```console python sftpgo_api_cli.py convert-users "" unix-passwd unix_users.json --min-uid 500 --force-uid 1000 --force-gid 1000 ``` -``` +```console python sftpgo_api_cli.py convert-users pureftpd.passwd pure-ftpd pure_users.json --usernames "user1" "user2" ``` -``` +```console python sftpgo_api_cli.py convert-users proftpd.passwd proftpd pro_users.json ``` @@ -576,7 +612,7 @@ The json file generated using the `convert-users` subcommand can be used as inpu Please note that when importing Linux/Unix users the input file is not required: `/etc/passwd` and `/etc/shadow` are automatically parsed. `/etc/shadow` read permission is is typically granted to the `root` user, so you need to execute the `convert-users` subcommand as `root`. -### Colors highlight for Windows command prompt +## Colors highlight for Windows command prompt If your Windows command prompt does not recognize ANSI/VT100 escape sequences you can download [ANSICON](https://github.com/adoxa/ansicon "ANSICON") extract proper files depending on your Windows OS, and install them using `ansicon -i`. -Thats all. From now on, your Windows command prompt will be aware of ANSI colors. \ No newline at end of file +Thats all. From now on, your Windows command prompt will be aware of ANSI colors. diff --git a/examples/rest-api-cli/sftpgo_api_cli.py b/examples/rest-api-cli/sftpgo_api_cli.py index c6d49aa5..708acd5d 100755 --- a/examples/rest-api-cli/sftpgo_api_cli.py +++ b/examples/rest-api-cli/sftpgo_api_cli.py @@ -40,6 +40,8 @@ class SFTPGoApiRequests: self.providerStatusPath = urlparse.urljoin(baseUrl, '/api/v1/providerstatus') self.dumpDataPath = urlparse.urljoin(baseUrl, '/api/v1/dumpdata') self.loadDataPath = urlparse.urljoin(baseUrl, '/api/v1/loaddata') + self.updateUsedQuotaPath = urlparse.urljoin(baseUrl, "/api/v1/quota_update") + self.updateFolderUsedQuotaPath = urlparse.urljoin(baseUrl, "/api/v1/folder_quota_update") self.debug = debug if authType == 'basic': self.auth = requests.auth.HTTPBasicAuth(authUser, authPassword) @@ -284,6 +286,16 @@ class SFTPGoApiRequests: r = requests.delete(urlparse.urljoin(self.userPath, 'user/' + str(user_id)), auth=self.auth, verify=self.verify) self.printResponse(r) + def updateQuotaUsage(self, username, used_quota_size, used_quota_files, mode): + req = {"username":username, "used_quota_files":used_quota_files, "used_quota_size":used_quota_size} + r = requests.put(self.updateUsedQuotaPath, params={'mode':mode}, json=req, auth=self.auth, verify=self.verify) + self.printResponse(r) + + def updateFolderQuotaUsage(self, mapped_path, used_quota_size, used_quota_files, mode): + req = {"mapped_path":mapped_path, "used_quota_files":used_quota_files, "used_quota_size":used_quota_size} + r = requests.put(self.updateFolderUsedQuotaPath, params={'mode':mode}, json=req, auth=self.auth, verify=self.verify) + self.printResponse(r) + def getConnections(self): r = requests.get(self.activeConnectionsPath, auth=self.auth, verify=self.verify) self.printResponse(r) @@ -688,6 +700,22 @@ if __name__ == '__main__': help='0 means new users are added, existing users are updated. 1 means new users are added,' + ' existing users are not modified. Default: %(default)s') + parserUpdateQuotaUsage = subparsers.add_parser('update-quota-usage', help='Update the user used quota limits') + parserUpdateQuotaUsage.add_argument('username', type=str) + parserUpdateQuotaUsage.add_argument('-M', '--mode', type=str, choices=["add", "reset"], default="reset", + help='the update mode specifies if the given quota usage values should be added or ' + + 'replace the current ones. Default: %(default)s') + parserUpdateQuotaUsage.add_argument('-S', '--used_quota_size', type=int, default=0, help='Default: %(default)s') + parserUpdateQuotaUsage.add_argument('-F', '--used_quota_files', type=int, default=0, help='Default: %(default)s') + + parserUpdateFolderQuotaUsage = subparsers.add_parser('update-folder-quota-usage', help='Update the folder used quota limits') + parserUpdateFolderQuotaUsage.add_argument('folder_path', type=str) + parserUpdateFolderQuotaUsage.add_argument('-M', '--mode', type=str, choices=["add", "reset"], default="reset", + help='the update mode specifies if the given quota usage values should be added or ' + + 'replace the current ones. Default: %(default)s') + parserUpdateFolderQuotaUsage.add_argument('-S', '--used_quota_size', type=int, default=0, help='Default: %(default)s') + parserUpdateFolderQuotaUsage.add_argument('-F', '--used_quota_files', type=int, default=0, help='Default: %(default)s') + parserConvertUsers = subparsers.add_parser('convert-users', help='Convert users to a JSON format suitable to use ' + 'with loadddata') supportedUsersFormats = [] @@ -765,6 +793,10 @@ if __name__ == '__main__': api.dumpData(args.output_file, args.indent) elif args.command == 'loaddata': api.loadData(args.input_file, args.scan_quota, args.mode) + elif args.command == 'update-quota-usage': + api.updateQuotaUsage(args.username, args.used_quota_size, args.used_quota_files, args.mode) + elif args.command == 'update-folder-quota-usage': + api.updateFolderQuotaUsage(args.folder_path, args.used_quota_size, args.used_quota_files, args.mode) elif args.command == 'convert-users': convertUsers = ConvertUsers(args.input_file, args.users_format, args.output_file, args.min_uid, args.max_uid, args.usernames, args.force_uid, args.force_gid) diff --git a/httpd/api_folder.go b/httpd/api_folder.go index 3d4b159a..7c169a29 100644 --- a/httpd/api_folder.go +++ b/httpd/api_folder.go @@ -69,7 +69,7 @@ func addFolder(w http.ResponseWriter, r *http.Request) { if err == nil { render.JSON(w, r, folder) } else { - sendAPIResponse(w, r, err, "", http.StatusInternalServerError) + sendAPIResponse(w, r, err, "", getRespStatus(err)) } } else { sendAPIResponse(w, r, err, "", getRespStatus(err)) @@ -88,11 +88,8 @@ func deleteFolderByPath(w http.ResponseWriter, r *http.Request) { } folder, err := dataprovider.GetFolderByPath(dataProvider, folderPath) - if _, ok := err.(*dataprovider.RecordNotFoundError); ok { - sendAPIResponse(w, r, err, "", http.StatusNotFound) - return - } else if err != nil { - sendAPIResponse(w, r, err, "", http.StatusInternalServerError) + if err != nil { + sendAPIResponse(w, r, err, "", getRespStatus(err)) return } err = dataprovider.DeleteFolder(dataProvider, folder) diff --git a/httpd/api_quota.go b/httpd/api_quota.go index 570a1512..0e2623f1 100644 --- a/httpd/api_quota.go +++ b/httpd/api_quota.go @@ -1,6 +1,7 @@ package httpd import ( + "errors" "net/http" "github.com/go-chi/render" @@ -11,6 +12,11 @@ import ( "github.com/drakkan/sftpgo/vfs" ) +const ( + quotaUpdateModeAdd = "add" + quotaUpdateModeReset = "reset" +) + func getQuotaScans(w http.ResponseWriter, r *http.Request) { render.JSON(w, r, sftpd.GetQuotaScans()) } @@ -19,8 +25,89 @@ func getVFolderQuotaScans(w http.ResponseWriter, r *http.Request) { render.JSON(w, r, sftpd.GetVFoldersQuotaScans()) } +func updateUserQuotaUsage(w http.ResponseWriter, r *http.Request) { + r.Body = http.MaxBytesReader(w, r.Body, maxRequestSize) + var u dataprovider.User + err := render.DecodeJSON(r.Body, &u) + if err != nil { + sendAPIResponse(w, r, err, "", http.StatusBadRequest) + return + } + if u.UsedQuotaFiles < 0 || u.UsedQuotaSize < 0 { + sendAPIResponse(w, r, errors.New("Invalid used quota parameters, negative values are not allowed"), + "", http.StatusBadRequest) + return + } + mode, err := getQuotaUpdateMode(r) + if err != nil { + sendAPIResponse(w, r, err, "", http.StatusBadRequest) + return + } + user, err := dataprovider.UserExists(dataProvider, u.Username) + if err != nil { + sendAPIResponse(w, r, err, "", getRespStatus(err)) + return + } + if mode == quotaUpdateModeAdd && !user.HasQuotaRestrictions() && dataprovider.GetQuotaTracking() == 2 { + sendAPIResponse(w, r, errors.New("this user has no quota restrictions, only reset mode is supported"), + "", http.StatusBadRequest) + return + } + if !sftpd.AddQuotaScan(user.Username) { + sendAPIResponse(w, r, err, "A quota scan is in progress for this user", http.StatusConflict) + return + } + defer sftpd.RemoveQuotaScan(user.Username) //nolint:errcheck + err = dataprovider.UpdateUserQuota(dataProvider, user, u.UsedQuotaFiles, u.UsedQuotaSize, mode == quotaUpdateModeReset) + if err != nil { + sendAPIResponse(w, r, err, "", getRespStatus(err)) + } else { + sendAPIResponse(w, r, err, "Quota updated", http.StatusOK) + } +} + +func updateVFolderQuotaUsage(w http.ResponseWriter, r *http.Request) { + r.Body = http.MaxBytesReader(w, r.Body, maxRequestSize) + var f vfs.BaseVirtualFolder + err := render.DecodeJSON(r.Body, &f) + if err != nil { + sendAPIResponse(w, r, err, "", http.StatusBadRequest) + return + } + if f.UsedQuotaFiles < 0 || f.UsedQuotaSize < 0 { + sendAPIResponse(w, r, errors.New("Invalid used quota parameters, negative values are not allowed"), + "", http.StatusBadRequest) + return + } + mode, err := getQuotaUpdateMode(r) + if err != nil { + sendAPIResponse(w, r, err, "", http.StatusBadRequest) + return + } + folder, err := dataprovider.GetFolderByPath(dataProvider, f.MappedPath) + if err != nil { + sendAPIResponse(w, r, err, "", getRespStatus(err)) + return + } + if !sftpd.AddVFolderQuotaScan(folder.MappedPath) { + sendAPIResponse(w, r, err, "A quota scan is in progress for this folder", http.StatusConflict) + return + } + defer sftpd.RemoveVFolderQuotaScan(folder.MappedPath) //nolint:errcheck + err = dataprovider.UpdateVirtualFolderQuota(dataProvider, folder, f.UsedQuotaFiles, f.UsedQuotaSize, mode == quotaUpdateModeReset) + if err != nil { + sendAPIResponse(w, r, err, "", getRespStatus(err)) + } else { + sendAPIResponse(w, r, err, "Quota updated", http.StatusOK) + } +} + func startQuotaScan(w http.ResponseWriter, r *http.Request) { r.Body = http.MaxBytesReader(w, r.Body, maxRequestSize) + if dataprovider.GetQuotaTracking() == 0 { + sendAPIResponse(w, r, nil, "Quota tracking is disabled!", http.StatusForbidden) + return + } var u dataprovider.User err := render.DecodeJSON(r.Body, &u) if err != nil { @@ -29,11 +116,7 @@ func startQuotaScan(w http.ResponseWriter, r *http.Request) { } user, err := dataprovider.UserExists(dataProvider, u.Username) if err != nil { - sendAPIResponse(w, r, err, "", http.StatusNotFound) - return - } - if dataprovider.GetQuotaTracking() == 0 { - sendAPIResponse(w, r, nil, "Quota tracking is disabled!", http.StatusForbidden) + sendAPIResponse(w, r, err, "", getRespStatus(err)) return } if sftpd.AddQuotaScan(user.Username) { @@ -46,6 +129,10 @@ func startQuotaScan(w http.ResponseWriter, r *http.Request) { func startVFolderQuotaScan(w http.ResponseWriter, r *http.Request) { r.Body = http.MaxBytesReader(w, r.Body, maxRequestSize) + if dataprovider.GetQuotaTracking() == 0 { + sendAPIResponse(w, r, nil, "Quota tracking is disabled!", http.StatusForbidden) + return + } var f vfs.BaseVirtualFolder err := render.DecodeJSON(r.Body, &f) if err != nil { @@ -54,11 +141,7 @@ func startVFolderQuotaScan(w http.ResponseWriter, r *http.Request) { } folder, err := dataprovider.GetFolderByPath(dataProvider, f.MappedPath) if err != nil { - sendAPIResponse(w, r, err, "", http.StatusNotFound) - return - } - if dataprovider.GetQuotaTracking() == 0 { - sendAPIResponse(w, r, nil, "Quota tracking is disabled!", http.StatusForbidden) + sendAPIResponse(w, r, err, "", getRespStatus(err)) return } if sftpd.AddVFolderQuotaScan(folder.MappedPath) { @@ -98,3 +181,14 @@ func doFolderQuotaScan(folder vfs.BaseVirtualFolder) error { logger.Debug(logSender, "", "virtual folder %#v scanned, error: %v", folder.MappedPath, err) return err } + +func getQuotaUpdateMode(r *http.Request) (string, error) { + mode := quotaUpdateModeReset + if _, ok := r.URL.Query()["mode"]; ok { + mode = r.URL.Query().Get("mode") + if mode != quotaUpdateModeReset && mode != quotaUpdateModeAdd { + return "", errors.New("Invalid mode") + } + } + return mode, nil +} diff --git a/httpd/api_user.go b/httpd/api_user.go index 254c4224..fa7dd3ae 100644 --- a/httpd/api_user.go +++ b/httpd/api_user.go @@ -66,10 +66,8 @@ func getUserByID(w http.ResponseWriter, r *http.Request) { user, err := dataprovider.GetUserByID(dataProvider, userID) if err == nil { render.JSON(w, r, dataprovider.HideUserSensitiveData(&user)) - } else if _, ok := err.(*dataprovider.RecordNotFoundError); ok { - sendAPIResponse(w, r, err, "", http.StatusNotFound) } else { - sendAPIResponse(w, r, err, "", http.StatusInternalServerError) + sendAPIResponse(w, r, err, "", getRespStatus(err)) } } @@ -87,7 +85,7 @@ func addUser(w http.ResponseWriter, r *http.Request) { if err == nil { render.JSON(w, r, dataprovider.HideUserSensitiveData(&user)) } else { - sendAPIResponse(w, r, err, "", http.StatusInternalServerError) + sendAPIResponse(w, r, err, "", getRespStatus(err)) } } else { sendAPIResponse(w, r, err, "", getRespStatus(err)) @@ -103,6 +101,10 @@ func updateUser(w http.ResponseWriter, r *http.Request) { return } user, err := dataprovider.GetUserByID(dataProvider, userID) + if err != nil { + sendAPIResponse(w, r, err, "", getRespStatus(err)) + return + } currentPermissions := user.Permissions currentFileExtensions := user.Filters.FileExtensions currentS3AccessSecret := "" @@ -111,13 +113,6 @@ func updateUser(w http.ResponseWriter, r *http.Request) { } user.Permissions = make(map[string][]string) user.Filters.FileExtensions = []dataprovider.ExtensionsFilter{} - if _, ok := err.(*dataprovider.RecordNotFoundError); ok { - sendAPIResponse(w, r, err, "", http.StatusNotFound) - return - } else if err != nil { - sendAPIResponse(w, r, err, "", http.StatusInternalServerError) - return - } err = render.DecodeJSON(r.Body, &user) if err != nil { sendAPIResponse(w, r, err, "", http.StatusBadRequest) @@ -158,11 +153,8 @@ func deleteUser(w http.ResponseWriter, r *http.Request) { return } user, err := dataprovider.GetUserByID(dataProvider, userID) - if _, ok := err.(*dataprovider.RecordNotFoundError); ok { - sendAPIResponse(w, r, err, "", http.StatusNotFound) - return - } else if err != nil { - sendAPIResponse(w, r, err, "", http.StatusInternalServerError) + if err != nil { + sendAPIResponse(w, r, err, "", getRespStatus(err)) return } err = dataprovider.DeleteUser(dataProvider, user) diff --git a/httpd/api_utils.go b/httpd/api_utils.go index b063e2dc..cfa340f8 100644 --- a/httpd/api_utils.go +++ b/httpd/api_utils.go @@ -82,6 +82,9 @@ func getRespStatus(err error) int { if _, ok := err.(*dataprovider.MethodDisabledError); ok { return http.StatusForbidden } + if _, ok := err.(*dataprovider.RecordNotFoundError); ok { + return http.StatusNotFound + } if os.IsNotExist(err) { return http.StatusBadRequest } @@ -218,7 +221,7 @@ func GetQuotaScans(expectedStatusCode int) ([]sftpd.ActiveQuotaScan, []byte, err return quotaScans, body, err } -// StartQuotaScan start a new quota scan for the given user and checks the received HTTP Status code against expectedStatusCode. +// StartQuotaScan starts a new quota scan for the given user and checks the received HTTP Status code against expectedStatusCode. func StartQuotaScan(user dataprovider.User, expectedStatusCode int) ([]byte, error) { var body []byte userAsJSON, _ := json.Marshal(user) @@ -231,6 +234,23 @@ func StartQuotaScan(user dataprovider.User, expectedStatusCode int) ([]byte, err return body, checkResponse(resp.StatusCode, expectedStatusCode) } +// UpdateQuotaUsage updates the user used quota limits and checks the received HTTP Status code against expectedStatusCode. +func UpdateQuotaUsage(user dataprovider.User, mode string, expectedStatusCode int) ([]byte, error) { + var body []byte + userAsJSON, _ := json.Marshal(user) + url, err := addModeQueryParam(buildURLRelativeToBase(updateUsedQuotaPath), mode) + if err != nil { + return body, err + } + resp, err := sendHTTPRequest(http.MethodPut, url.String(), bytes.NewBuffer(userAsJSON), "") + if err != nil { + return body, err + } + defer resp.Body.Close() + body, _ = getResponseBody(resp) + return body, checkResponse(resp.StatusCode, expectedStatusCode) +} + // GetConnections returns status and stats for active SFTP/SCP connections func GetConnections(expectedStatusCode int) ([]sftpd.ConnectionStatus, []byte, error) { var connections []sftpd.ConnectionStatus @@ -370,6 +390,23 @@ func StartFolderQuotaScan(folder vfs.BaseVirtualFolder, expectedStatusCode int) return body, checkResponse(resp.StatusCode, expectedStatusCode) } +// UpdateFolderQuotaUsage updates the folder used quota limits and checks the received HTTP Status code against expectedStatusCode. +func UpdateFolderQuotaUsage(folder vfs.BaseVirtualFolder, mode string, expectedStatusCode int) ([]byte, error) { + var body []byte + folderAsJSON, _ := json.Marshal(folder) + url, err := addModeQueryParam(buildURLRelativeToBase(updateFolderUsedQuotaPath), mode) + if err != nil { + return body, err + } + resp, err := sendHTTPRequest(http.MethodPut, url.String(), bytes.NewBuffer(folderAsJSON), "") + if err != nil { + return body, err + } + defer resp.Body.Close() + body, _ = getResponseBody(resp) + return body, checkResponse(resp.StatusCode, expectedStatusCode) +} + // GetVersion returns version details func GetVersion(expectedStatusCode int) (version.Info, []byte, error) { var appVersion version.Info @@ -778,3 +815,16 @@ func addLimitAndOffsetQueryParams(rawurl string, limit, offset int64) (*url.URL, url.RawQuery = q.Encode() return url, err } + +func addModeQueryParam(rawurl, mode string) (*url.URL, error) { + url, err := url.Parse(rawurl) + if err != nil { + return nil, err + } + q := url.Query() + if len(mode) > 0 { + q.Add("mode", mode) + } + url.RawQuery = q.Encode() + return url, err +} diff --git a/httpd/httpd.go b/httpd/httpd.go index 7ef86dbf..6a6b313d 100644 --- a/httpd/httpd.go +++ b/httpd/httpd.go @@ -21,28 +21,30 @@ import ( ) const ( - logSender = "httpd" - apiPrefix = "/api/v1" - activeConnectionsPath = "/api/v1/connection" - quotaScanPath = "/api/v1/quota_scan" - quotaScanVFolderPath = "/api/v1/folder_quota_scan" - userPath = "/api/v1/user" - versionPath = "/api/v1/version" - folderPath = "/api/v1/folder" - providerStatusPath = "/api/v1/providerstatus" - dumpDataPath = "/api/v1/dumpdata" - loadDataPath = "/api/v1/loaddata" - metricsPath = "/metrics" - pprofBasePath = "/debug" - webBasePath = "/web" - webUsersPath = "/web/users" - webUserPath = "/web/user" - webConnectionsPath = "/web/connections" - webFoldersPath = "/web/folders" - webFolderPath = "/web/folder" - webStaticFilesPath = "/static" - maxRestoreSize = 10485760 // 10 MB - maxRequestSize = 1048576 // 1MB + logSender = "httpd" + apiPrefix = "/api/v1" + activeConnectionsPath = "/api/v1/connection" + quotaScanPath = "/api/v1/quota_scan" + quotaScanVFolderPath = "/api/v1/folder_quota_scan" + userPath = "/api/v1/user" + versionPath = "/api/v1/version" + folderPath = "/api/v1/folder" + providerStatusPath = "/api/v1/providerstatus" + dumpDataPath = "/api/v1/dumpdata" + loadDataPath = "/api/v1/loaddata" + updateUsedQuotaPath = "/api/v1/quota_update" + updateFolderUsedQuotaPath = "/api/v1/folder_quota_update" + metricsPath = "/metrics" + pprofBasePath = "/debug" + webBasePath = "/web" + webUsersPath = "/web/users" + webUserPath = "/web/user" + webConnectionsPath = "/web/connections" + webFoldersPath = "/web/folders" + webFolderPath = "/web/folder" + webStaticFilesPath = "/static" + maxRestoreSize = 10485760 // 10 MB + maxRequestSize = 1048576 // 1MB ) var ( diff --git a/httpd/httpd_test.go b/httpd/httpd_test.go index edadd309..edfe5cd9 100644 --- a/httpd/httpd_test.go +++ b/httpd/httpd_test.go @@ -38,26 +38,28 @@ import ( ) const ( - defaultUsername = "test_user" - defaultPassword = "test_password" - testPubKey = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC03jj0D+djk7pxIf/0OhrxrchJTRZklofJ1NoIu4752Sq02mdXmarMVsqJ1cAjV5LBVy3D1F5U6XW4rppkXeVtd04Pxb09ehtH0pRRPaoHHlALiJt8CoMpbKYMA8b3KXPPriGxgGomvtU2T2RMURSwOZbMtpsugfjYSWenyYX+VORYhylWnSXL961LTyC21ehd6d6QnW9G7E5hYMITMY9TuQZz3bROYzXiTsgN0+g6Hn7exFQp50p45StUMfV/SftCMdCxlxuyGny2CrN/vfjO7xxOo2uv7q1qm10Q46KPWJQv+pgZ/OfL+EDjy07n5QVSKHlbx+2nT4Q0EgOSQaCTYwn3YjtABfIxWwgAFdyj6YlPulCL22qU4MYhDcA6PSBwDdf8hvxBfvsiHdM+JcSHvv8/VeJhk6CmnZxGY0fxBupov27z3yEO8nAg8k+6PaUiW1MSUfuGMF/ktB8LOstXsEPXSszuyXiOv4DaryOXUiSn7bmRqKcEFlJusO6aZP0= nicola@p1" - logSender = "APITesting" - userPath = "/api/v1/user" - folderPath = "/api/v1/folder" - activeConnectionsPath = "/api/v1/connection" - quotaScanPath = "/api/v1/quota_scan" - quotaScanVFolderPath = "/api/v1/folder_quota_scan" - versionPath = "/api/v1/version" - metricsPath = "/metrics" - pprofPath = "/debug/pprof/" - webBasePath = "/web" - webUsersPath = "/web/users" - webUserPath = "/web/user" - webFoldersPath = "/web/folders" - webFolderPath = "/web/folder" - webConnectionsPath = "/web/connections" - configDir = ".." - httpsCert = `-----BEGIN CERTIFICATE----- + defaultUsername = "test_user" + defaultPassword = "test_password" + testPubKey = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC03jj0D+djk7pxIf/0OhrxrchJTRZklofJ1NoIu4752Sq02mdXmarMVsqJ1cAjV5LBVy3D1F5U6XW4rppkXeVtd04Pxb09ehtH0pRRPaoHHlALiJt8CoMpbKYMA8b3KXPPriGxgGomvtU2T2RMURSwOZbMtpsugfjYSWenyYX+VORYhylWnSXL961LTyC21ehd6d6QnW9G7E5hYMITMY9TuQZz3bROYzXiTsgN0+g6Hn7exFQp50p45StUMfV/SftCMdCxlxuyGny2CrN/vfjO7xxOo2uv7q1qm10Q46KPWJQv+pgZ/OfL+EDjy07n5QVSKHlbx+2nT4Q0EgOSQaCTYwn3YjtABfIxWwgAFdyj6YlPulCL22qU4MYhDcA6PSBwDdf8hvxBfvsiHdM+JcSHvv8/VeJhk6CmnZxGY0fxBupov27z3yEO8nAg8k+6PaUiW1MSUfuGMF/ktB8LOstXsEPXSszuyXiOv4DaryOXUiSn7bmRqKcEFlJusO6aZP0= nicola@p1" + logSender = "APITesting" + userPath = "/api/v1/user" + folderPath = "/api/v1/folder" + activeConnectionsPath = "/api/v1/connection" + quotaScanPath = "/api/v1/quota_scan" + quotaScanVFolderPath = "/api/v1/folder_quota_scan" + updateUsedQuotaPath = "/api/v1/quota_update" + updateFolderUsedQuotaPath = "/api/v1/folder_quota_update" + versionPath = "/api/v1/version" + metricsPath = "/metrics" + pprofPath = "/debug/pprof/" + webBasePath = "/web" + webUsersPath = "/web/users" + webUserPath = "/web/user" + webFoldersPath = "/web/folders" + webFolderPath = "/web/folder" + webConnectionsPath = "/web/connections" + configDir = ".." + httpsCert = `-----BEGIN CERTIFICATE----- MIICHTCCAaKgAwIBAgIUHnqw7QnB1Bj9oUsNpdb+ZkFPOxMwCgYIKoZIzj0EAwIw RTELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoMGElu dGVybmV0IFdpZGdpdHMgUHR5IEx0ZDAeFw0yMDAyMDQwOTUzMDRaFw0zMDAyMDEw @@ -603,8 +605,13 @@ func TestUserPublicKey(t *testing.T) { } func TestUpdateUser(t *testing.T) { - user, _, err := httpd.AddUser(getTestUser(), http.StatusOK) + u := getTestUser() + u.UsedQuotaFiles = 1 + u.UsedQuotaSize = 2 + user, _, err := httpd.AddUser(u, http.StatusOK) assert.NoError(t, err) + assert.Equal(t, 0, user.UsedQuotaFiles) + assert.Equal(t, int64(0), user.UsedQuotaSize) user.HomeDir = filepath.Join(homeBasePath, "testmod") user.UID = 33 user.GID = 101 @@ -683,6 +690,48 @@ func TestUpdateUser(t *testing.T) { } } +func TestUpdateUserQuotaUsage(t *testing.T) { + u := getTestUser() + usedQuotaFiles := 1 + usedQuotaSize := int64(65535) + u.UsedQuotaFiles = usedQuotaFiles + u.UsedQuotaSize = usedQuotaSize + user, _, err := httpd.AddUser(u, http.StatusOK) + assert.NoError(t, err) + _, err = httpd.UpdateQuotaUsage(u, "invalid_mode", http.StatusBadRequest) + assert.NoError(t, err) + _, err = httpd.UpdateQuotaUsage(u, "", http.StatusOK) + assert.NoError(t, err) + user, _, err = httpd.GetUserByID(user.ID, http.StatusOK) + assert.NoError(t, err) + assert.Equal(t, usedQuotaFiles, user.UsedQuotaFiles) + assert.Equal(t, usedQuotaSize, user.UsedQuotaSize) + _, err = httpd.UpdateQuotaUsage(u, "add", http.StatusBadRequest) + assert.NoError(t, err, "user has no quota restrictions add mode should fail") + user, _, err = httpd.GetUserByID(user.ID, http.StatusOK) + assert.NoError(t, err) + assert.Equal(t, usedQuotaFiles, user.UsedQuotaFiles) + assert.Equal(t, usedQuotaSize, user.UsedQuotaSize) + user.QuotaFiles = 100 + user, _, err = httpd.UpdateUser(user, http.StatusOK) + assert.NoError(t, err) + _, err = httpd.UpdateQuotaUsage(u, "add", http.StatusOK) + assert.NoError(t, err) + user, _, err = httpd.GetUserByID(user.ID, http.StatusOK) + assert.NoError(t, err) + assert.Equal(t, 2*usedQuotaFiles, user.UsedQuotaFiles) + assert.Equal(t, 2*usedQuotaSize, user.UsedQuotaSize) + u.UsedQuotaFiles = -1 + _, err = httpd.UpdateQuotaUsage(u, "", http.StatusBadRequest) + assert.NoError(t, err) + u.UsedQuotaFiles = usedQuotaFiles + u.Username = u.Username + "1" + _, err = httpd.UpdateQuotaUsage(u, "", http.StatusNotFound) + assert.NoError(t, err) + _, err = httpd.RemoveUser(user, http.StatusOK) + assert.NoError(t, err) +} + func TestUserFolderMapping(t *testing.T) { mappedPath1 := filepath.Join(os.TempDir(), "mapped_dir1") mappedPath2 := filepath.Join(os.TempDir(), "mapped_dir2") @@ -1038,6 +1087,47 @@ func TestStartQuotaScan(t *testing.T) { assert.NoError(t, err) } +func TestUpdateFolderQuotaUsage(t *testing.T) { + f := vfs.BaseVirtualFolder{ + MappedPath: filepath.Join(os.TempDir(), "folder"), + } + usedQuotaFiles := 1 + usedQuotaSize := int64(65535) + f.UsedQuotaFiles = usedQuotaFiles + f.UsedQuotaSize = usedQuotaSize + folder, _, err := httpd.AddFolder(f, http.StatusOK) + assert.NoError(t, err) + _, err = httpd.UpdateFolderQuotaUsage(folder, "invalid mode", http.StatusBadRequest) + assert.NoError(t, err) + _, err = httpd.UpdateFolderQuotaUsage(f, "reset", http.StatusOK) + assert.NoError(t, err) + folders, _, err := httpd.GetFolders(0, 0, f.MappedPath, http.StatusOK) + assert.NoError(t, err) + if assert.Len(t, folders, 1) { + folder = folders[0] + assert.Equal(t, usedQuotaFiles, folder.UsedQuotaFiles) + assert.Equal(t, usedQuotaSize, folder.UsedQuotaSize) + } + _, err = httpd.UpdateFolderQuotaUsage(f, "add", http.StatusOK) + assert.NoError(t, err) + folders, _, err = httpd.GetFolders(0, 0, f.MappedPath, http.StatusOK) + assert.NoError(t, err) + if assert.Len(t, folders, 1) { + folder = folders[0] + assert.Equal(t, 2*usedQuotaFiles, folder.UsedQuotaFiles) + assert.Equal(t, 2*usedQuotaSize, folder.UsedQuotaSize) + } + f.UsedQuotaSize = -1 + _, err = httpd.UpdateFolderQuotaUsage(f, "", http.StatusBadRequest) + assert.NoError(t, err) + f.UsedQuotaSize = usedQuotaSize + f.MappedPath = f.MappedPath + "1" + _, err = httpd.UpdateFolderQuotaUsage(f, "", http.StatusNotFound) + assert.NoError(t, err) + _, err = httpd.RemoveFolder(folder, http.StatusOK) + assert.NoError(t, err) +} + func TestGetVersion(t *testing.T) { _, _, err := httpd.GetVersion(http.StatusOK) assert.NoError(t, err) @@ -1115,6 +1205,8 @@ func TestQuotaTrackingDisabled(t *testing.T) { assert.NoError(t, err) _, err = httpd.StartQuotaScan(user, http.StatusForbidden) assert.NoError(t, err) + _, err = httpd.UpdateQuotaUsage(user, "", http.StatusForbidden) + assert.NoError(t, err) _, err = httpd.RemoveUser(user, http.StatusOK) assert.NoError(t, err) // folder quota scan must fail @@ -1125,6 +1217,8 @@ func TestQuotaTrackingDisabled(t *testing.T) { assert.NoError(t, err) _, err = httpd.StartFolderQuotaScan(folder, http.StatusForbidden) assert.NoError(t, err) + _, err = httpd.UpdateFolderQuotaUsage(folder, "", http.StatusForbidden) + assert.NoError(t, err) _, err = httpd.RemoveFolder(folder, http.StatusOK) assert.NoError(t, err) @@ -1554,6 +1648,42 @@ func TestUpdateUserMock(t *testing.T) { checkResponseCode(t, http.StatusOK, rr.Code) } +func TestUpdateUserQuotaUsageMock(t *testing.T) { + var user dataprovider.User + u := getTestUser() + usedQuotaFiles := 1 + usedQuotaSize := int64(65535) + u.UsedQuotaFiles = usedQuotaFiles + u.UsedQuotaSize = usedQuotaSize + userAsJSON := getUserAsJSON(t, u) + req, _ := http.NewRequest(http.MethodPost, userPath, bytes.NewBuffer(userAsJSON)) + rr := executeRequest(req) + checkResponseCode(t, http.StatusOK, rr.Code) + err := render.DecodeJSON(rr.Body, &user) + assert.NoError(t, err) + req, _ = http.NewRequest(http.MethodPut, updateUsedQuotaPath, bytes.NewBuffer(userAsJSON)) + rr = executeRequest(req) + checkResponseCode(t, http.StatusOK, rr.Code) + req, _ = http.NewRequest(http.MethodGet, userPath+"/"+strconv.FormatInt(user.ID, 10), nil) + rr = executeRequest(req) + checkResponseCode(t, http.StatusOK, rr.Code) + err = render.DecodeJSON(rr.Body, &user) + assert.NoError(t, err) + assert.Equal(t, usedQuotaFiles, user.UsedQuotaFiles) + assert.Equal(t, usedQuotaSize, user.UsedQuotaSize) + req, _ = http.NewRequest(http.MethodPut, updateUsedQuotaPath, bytes.NewBuffer([]byte("string"))) + rr = executeRequest(req) + checkResponseCode(t, http.StatusBadRequest, rr.Code) + assert.True(t, sftpd.AddQuotaScan(user.Username)) + req, _ = http.NewRequest(http.MethodPut, updateUsedQuotaPath, bytes.NewBuffer(userAsJSON)) + rr = executeRequest(req) + checkResponseCode(t, http.StatusConflict, rr.Code) + assert.NoError(t, sftpd.RemoveQuotaScan(user.Username)) + req, _ = http.NewRequest(http.MethodDelete, userPath+"/"+strconv.FormatInt(user.ID, 10), nil) + rr = executeRequest(req) + checkResponseCode(t, http.StatusOK, rr.Code) +} + func TestUserPermissionsMock(t *testing.T) { user := getTestUser() user.Permissions = make(map[string][]string) @@ -1784,6 +1914,64 @@ func TestStartQuotaScanMock(t *testing.T) { assert.NoError(t, err) } +func TestUpdateFolderQuotaUsageMock(t *testing.T) { + mappedPath := filepath.Join(os.TempDir(), "vfolder") + f := vfs.BaseVirtualFolder{ + MappedPath: mappedPath, + } + usedQuotaFiles := 1 + usedQuotaSize := int64(65535) + f.UsedQuotaFiles = usedQuotaFiles + f.UsedQuotaSize = usedQuotaSize + var folder vfs.BaseVirtualFolder + folderAsJSON, err := json.Marshal(f) + assert.NoError(t, err) + req, _ := http.NewRequest(http.MethodPost, folderPath, bytes.NewBuffer(folderAsJSON)) + rr := executeRequest(req) + checkResponseCode(t, http.StatusOK, rr.Code) + err = render.DecodeJSON(rr.Body, &folder) + assert.NoError(t, err) + req, _ = http.NewRequest(http.MethodPut, updateFolderUsedQuotaPath, bytes.NewBuffer(folderAsJSON)) + rr = executeRequest(req) + checkResponseCode(t, http.StatusOK, rr.Code) + + var folders []vfs.BaseVirtualFolder + url, err := url.Parse(folderPath) + assert.NoError(t, err) + q := url.Query() + q.Add("folder_path", mappedPath) + url.RawQuery = q.Encode() + req, _ = http.NewRequest(http.MethodGet, url.String(), nil) + rr = executeRequest(req) + checkResponseCode(t, http.StatusOK, rr.Code) + err = render.DecodeJSON(rr.Body, &folders) + assert.NoError(t, err) + if assert.Len(t, folders, 1) { + folder = folders[0] + assert.Equal(t, usedQuotaFiles, folder.UsedQuotaFiles) + assert.Equal(t, usedQuotaSize, folder.UsedQuotaSize) + } + + req, _ = http.NewRequest(http.MethodPut, updateFolderUsedQuotaPath, bytes.NewBuffer([]byte("string"))) + rr = executeRequest(req) + checkResponseCode(t, http.StatusBadRequest, rr.Code) + + assert.True(t, sftpd.AddVFolderQuotaScan(mappedPath)) + req, _ = http.NewRequest(http.MethodPut, updateFolderUsedQuotaPath, bytes.NewBuffer(folderAsJSON)) + rr = executeRequest(req) + checkResponseCode(t, http.StatusConflict, rr.Code) + assert.NoError(t, sftpd.RemoveVFolderQuotaScan(mappedPath)) + + url, err = url.Parse(folderPath) + assert.NoError(t, err) + q = url.Query() + q.Add("folder_path", mappedPath) + url.RawQuery = q.Encode() + req, _ = http.NewRequest(http.MethodDelete, url.String(), nil) + rr = executeRequest(req) + checkResponseCode(t, http.StatusOK, rr.Code) +} + func TestStartFolderQuotaScanMock(t *testing.T) { mappedPath := filepath.Join(os.TempDir(), "vfolder") folder := vfs.BaseVirtualFolder{ diff --git a/httpd/internal_test.go b/httpd/internal_test.go index 9b428037..9a71298f 100644 --- a/httpd/internal_test.go +++ b/httpd/internal_test.go @@ -357,17 +357,24 @@ func TestApiCallsWithBadURL(t *testing.T) { oldAuthUsername := authUsername oldAuthPassword := authPassword SetBaseURLAndCredentials(invalidURL, oldAuthUsername, oldAuthPassword) + folder := vfs.BaseVirtualFolder{ + MappedPath: os.TempDir(), + } u := dataprovider.User{} _, _, err := UpdateUser(u, http.StatusBadRequest) assert.Error(t, err) _, err = RemoveUser(u, http.StatusNotFound) assert.Error(t, err) - _, err = RemoveFolder(vfs.BaseVirtualFolder{}, http.StatusNotFound) + _, err = RemoveFolder(folder, http.StatusNotFound) assert.Error(t, err) _, _, err = GetUsers(1, 0, "", http.StatusBadRequest) assert.Error(t, err) _, _, err = GetFolders(1, 0, "", http.StatusBadRequest) assert.Error(t, err) + _, err = UpdateQuotaUsage(u, "", http.StatusNotFound) + assert.Error(t, err) + _, err = UpdateFolderQuotaUsage(folder, "", http.StatusNotFound) + assert.Error(t, err) _, err = CloseConnection("non_existent_id", http.StatusNotFound) assert.Error(t, err) _, _, err = Dumpdata("backup.json", "", http.StatusBadRequest) @@ -393,6 +400,8 @@ func TestApiCallToNotListeningServer(t *testing.T) { assert.Error(t, err) _, _, err = GetUsers(100, 0, "", http.StatusOK) assert.Error(t, err) + _, err = UpdateQuotaUsage(u, "", http.StatusNotFound) + assert.Error(t, err) _, _, err = GetQuotaScans(http.StatusOK) assert.Error(t, err) _, err = StartQuotaScan(u, http.StatusNotFound) @@ -408,6 +417,8 @@ func TestApiCallToNotListeningServer(t *testing.T) { assert.Error(t, err) _, _, err = GetFolders(0, 0, "", http.StatusOK) assert.Error(t, err) + _, err = UpdateFolderQuotaUsage(folder, "", http.StatusNotFound) + assert.Error(t, err) _, _, err = GetFoldersQuotaScans(http.StatusOK) assert.Error(t, err) _, _, err = GetConnections(http.StatusOK) diff --git a/httpd/router.go b/httpd/router.go index c3c14238..b94525da 100644 --- a/httpd/router.go +++ b/httpd/router.go @@ -86,6 +86,8 @@ func initializeRouter(staticFilesPath string, enableProfiler, enableWebAdmin boo router.Delete(folderPath, deleteFolderByPath) router.Get(dumpDataPath, dumpData) router.Get(loadDataPath, loadData) + router.Put(updateUsedQuotaPath, updateUserQuotaUsage) + router.Put(updateFolderUsedQuotaPath, updateVFolderQuotaUsage) if enableWebAdmin { router.Get(webUsersPath, handleGetWebUsers) router.Get(webUserPath, handleWebAddUserGet) diff --git a/httpd/schema/openapi.yaml b/httpd/schema/openapi.yaml index 6b5b3bab..e1a6ad94 100644 --- a/httpd/schema/openapi.yaml +++ b/httpd/schema/openapi.yaml @@ -2,7 +2,7 @@ openapi: 3.0.1 info: title: SFTPGo description: 'SFTPGo REST API' - version: 1.9.0 + version: 1.9.1 servers: - url: /api/v1 @@ -348,6 +348,202 @@ paths: status: 500 message: "" error: "Error description if any" + /quota_update: + put: + tags: + - quota + summary: update the user used quota limits + description: Set the current used quota limits for the given user + operationId: quota_update + parameters: + - in: query + name: mode + required: false + description: the update mode specifies if the given quota usage values should be added or replace the current ones + schema: + type: string + enum: [add, reset] + description: > + Update type: + * `add` - add the specified quota limits to the current used ones + * `reset` - reset the values to the specified ones. This is the default + example: reset + requestBody: + required: true + description: The only user mandatory fields are username,used_quota_size and used_quota_files. Please note that if the quota fields are missing they will default to 0 + content: + application/json: + schema: + $ref : '#/components/schemas/User' + responses: + 200: + description: successful operation + content: + application/json: + schema: + $ref: '#/components/schemas/ApiResponse' + example: + status: 200 + message: "Quota updated" + error: "" + 400: + description: Bad request + content: + application/json: + schema: + $ref: '#/components/schemas/ApiResponse' + example: + status: 400 + message: "" + error: "Error description if any" + 401: + description: Unauthorized + content: + application/json: + schema: + $ref: '#/components/schemas/ApiResponse' + example: + status: 401 + message: "" + error: "Error description if any" + 403: + description: Forbidden + content: + application/json: + schema: + $ref: '#/components/schemas/ApiResponse' + example: + status: 403 + message: "" + error: "Error description if any" + 404: + description: Not Found + content: + application/json: + schema: + $ref: '#/components/schemas/ApiResponse' + example: + status: 404 + message: "" + error: "Error description if any" + 409: + description: A quota scan is in progress for this user + content: + application/json: + schema: + $ref: '#/components/schemas/ApiResponse' + example: + status: 409 + message: "A quota scan is in progress" + error: "Error description if any" + 500: + description: Internal Server Error + content: + application/json: + schema: + $ref: '#/components/schemas/ApiResponse' + example: + status: 500 + message: "" + error: "Error description if any" + /folder_quota_update: + put: + tags: + - quota + summary: update the folder used quota limits + description: Set the current used quota limits for the given folder + operationId: folder_quota_update + parameters: + - in: query + name: mode + required: false + description: the update mode specifies if the given quota usage values should be added or replace the current ones + schema: + type: string + enum: [add, reset] + description: > + Update type: + * `add` - add the specified quota limits to the current used ones + * `reset` - reset the values to the specified ones. This is the default + example: reset + requestBody: + required: true + description: The only folder mandatory fields are mapped_path,used_quota_size and used_quota_files. Please note that if the used quota fields are missing they will default to 0 + content: + application/json: + schema: + $ref : '#/components/schemas/BaseVirtualFolder' + responses: + 200: + description: successful operation + content: + application/json: + schema: + $ref: '#/components/schemas/ApiResponse' + example: + status: 200 + message: "Quota updated" + error: "" + 400: + description: Bad request + content: + application/json: + schema: + $ref: '#/components/schemas/ApiResponse' + example: + status: 400 + message: "" + error: "Error description if any" + 401: + description: Unauthorized + content: + application/json: + schema: + $ref: '#/components/schemas/ApiResponse' + example: + status: 401 + message: "" + error: "Error description if any" + 403: + description: Forbidden + content: + application/json: + schema: + $ref: '#/components/schemas/ApiResponse' + example: + status: 403 + message: "" + error: "Error description if any" + 404: + description: Not Found + content: + application/json: + schema: + $ref: '#/components/schemas/ApiResponse' + example: + status: 404 + message: "" + error: "Error description if any" + 409: + description: A quota scan is in progress for this folder + content: + application/json: + schema: + $ref: '#/components/schemas/ApiResponse' + example: + status: 409 + message: "A quota scan is in progress" + error: "Error description if any" + 500: + description: Internal Server Error + content: + application/json: + schema: + $ref: '#/components/schemas/ApiResponse' + example: + status: 500 + message: "" + error: "Error description if any" /folder_quota_scan: get: tags: