Add API endpoint to set current quota

Fixes #130
This commit is contained in:
Nicola Murino
2020-06-20 12:38:04 +02:00
parent 23a80b01b6
commit 8cb47817f6
16 changed files with 746 additions and 143 deletions

View File

@@ -1,4 +1,4 @@
## Dockerfile examples
# Dockerfile examples
Sample Dockerfiles for `sftpgo` daemon and the REST API CLI.

View File

@@ -2,8 +2,10 @@
This DockerFile is made to build image to host multiple instances of SFTPGo started with different users.
### Example
## Example
> 1003 is a custom uid:gid for this instance of SFTPGo
```bash
# Prereq on docker host
sudo groupadd -g 1003 sftpgrp && \
@@ -48,7 +50,8 @@ The script `entrypoint.sh` makes sure to correct the permissions of directories
Several images can be run with different parameters.
### Custom systemd script
## Custom systemd script
An example of systemd script is present [here](sftpgo.service), with `Environment` parameter to set `PUID` and `GUID`
`WorkingDirectory` parameter must be exist with one file in this directory like `sftpgo-${PUID}.env` corresponding to the variable file for SFTPGo instance.

View File

@@ -1,4 +1,4 @@
## Dockerfile based on Debian stable
# Dockerfile based on Debian stable
Please read the comments inside the `Dockerfile` to learn how to customize things for your setup.

View File

@@ -1,4 +1,4 @@
## LDAPAuth
# LDAPAuth
This is an example for an external authentication program. It performs authentication against an LDAP server.
It is tested against [389ds](https://directory.fedoraproject.org/) and can be used as starting point to authenticate using any LDAP server including Active Directory.
@@ -6,13 +6,13 @@ It is tested against [389ds](https://directory.fedoraproject.org/) and can be us
You need to change the LDAP connection parameters and the user search query to match your environment.
You can build this example using the following command:
```
```console
go build -i -ldflags "-s -w" -o ldapauth
```
This program assumes that the 389ds schema was extended to add support for public keys using the following ldif file placed in `/etc/dirsrv/schema/98openssh-ldap.ldif`:
```
```console
dn: cn=schema
changetype: modify
add: attributetypes

View File

@@ -1,4 +1,4 @@
## LDAPAuthServer
# LDAPAuthServer
This is an example for an HTTP server to use as external authentication HTTP hook. It performs authentication against an LDAP server.
It is tested against [389ds](https://directory.fedoraproject.org/) and can be used as starting point to authenticate using any LDAP server including Active Directory.
@@ -6,6 +6,6 @@ It is tested against [389ds](https://directory.fedoraproject.org/) and can be us
You can configure the server using the [ldapauth.toml](./ldapauth.toml) configuration file.
You can build this example using the following command:
```
```console
go build -i -ldflags "-s -w" -o ldapauthserver
```

View File

@@ -1,4 +1,4 @@
## REST API CLI client
# REST API CLI client
`sftpgo_api_cli.py` is a very simple command line client for `SFTPGo` REST API written in python.
@@ -10,13 +10,13 @@ It has the following requirements:
You can see the usage with the following command:
```
```console
python sftpgo_api_cli.py --help
```
and
```
```console
python sftpgo_api_cli.py [sub-command] --help
```
@@ -39,11 +39,11 @@ Additionally it can convert users to the SFTPGo format from some supported users
Let's see a sample usage for each REST API.
### Add user
## Add user
Command:
```
```console
python sftpgo_api_cli.py add-user test_username --password "test_pwd" --home-dir="/tmp/test_home_dir" --uid 33 --gid 1000 --max-sessions 2 --quota-size 0 --quota-files 3 --permissions "list" "download" "upload" "delete" "rename" "create_dirs" "overwrite" --subdirs-permissions "/dir1::list,download" "/dir2::*" --upload-bandwidth 100 --download-bandwidth 60 --status 0 --expiration-date 2019-01-01 --allowed-ip "192.168.1.1/32" --fs S3 --s3-bucket test --s3-region eu-west-1 --s3-access-key accesskey --s3-access-secret secret --s3-endpoint "http://127.0.0.1:9000" --s3-storage-class Standard --s3-key-prefix "vfolder/" --s3-upload-part-size 10 --s3-upload-concurrency 4 --denied-login-methods "password" "keyboard-interactive" --allowed-extensions "/dir1::.jpg,.png" "/dir2::.rar,.png" --denied-extensions "/dir3::.zip,.rar"
```
@@ -135,11 +135,11 @@ Output:
}
```
### Update user
## Update user
Command:
```
```console
python sftpgo_api_cli.py update-user 9576 test_username --password "test_pwd" --home-dir="/tmp/test_home_dir" --uid 0 --gid 33 --max-sessions 3 --quota-size 0 --quota-files 4 --permissions "*" --subdirs-permissions "/dir1::list,download,create_symlinks" --upload-bandwidth 90 --download-bandwidth 80 --status 1 --expiration-date "" --allowed-ip "" --denied-ip "192.168.1.0/24" --denied-login-methods "" --fs local --virtual-folders "/vdir1::/tmp/mapped1::-1::-1" "/vdir2::/tmp/mapped2::100::104857600" --allowed-extensions "" --denied-extensions ""
```
@@ -153,11 +153,11 @@ Output:
}
```
### Get user by id
## Get user by id
Command:
```
```console
python sftpgo_api_cli.py get-user-by-id 9576
```
@@ -226,11 +226,11 @@ Output:
}
```
### Get users
## Get users
Command:
```
```console
python sftpgo_api_cli.py get-users --limit 1 --offset 0 --username test_username --order DESC
```
@@ -291,11 +291,11 @@ Output:
]
```
### Get active connections
## Get active connections
Command:
```
```console
python sftpgo_api_cli.py get-connections
```
@@ -325,11 +325,11 @@ Output:
]
```
### Get folders
## Get folders
Command:
```
```console
python sftpgo_api_cli.py get-folders --limit 1 --offset 0 --folder-path /tmp/mapped1 --order DESC
```
@@ -350,9 +350,9 @@ Output:
]
```
### Add folder
## Add folder
```
```console
python sftpgo_api_cli.py add-folder /tmp/mapped_folder
```
@@ -368,11 +368,11 @@ Output:
}
```
### Close connection
## Close connection
Command:
```
```console
python sftpgo_api_cli.py close-connection f82cfec6a391ad673edd4ae9a144f32ccb59456139f8e1185b070134fffbab7c
```
@@ -386,19 +386,19 @@ Output:
}
```
### Get quota scans
## Get quota scans
Command:
```
```console
python sftpgo_api_cli.py get-quota-scans
```
### Start quota scan
## Start quota scan
Command:
```
```console
python sftpgo_api_cli.py start-quota-scan test_username
```
@@ -412,19 +412,19 @@ Output:
}
```
### Get folder quota scans
## Get folder quota scans
Command:
```
```console
python sftpgo_api_cli.py get-folders-quota-scans
```
### Start folder quota scan
## Start folder quota scan
Command:
```
```console
python sftpgo_api_cli.py start-folder-quota-scan /tmp/mapped_folder
```
@@ -438,11 +438,47 @@ Output:
}
```
### Delete user
## Update quota usage
Command:
```console
python sftpgo_api_cli.py -d update-quota-usage a -S 123 -F 1 -M reset
```
Output:
```json
{
"error": "",
"message": "Quota updated",
"status": 200
}
```
## Update folder quota usage
Command:
```console
python sftpgo_api_cli.py -d update-quota-usage /tmp/mapped_folder -S 123 -F 1 -M add
```
Output:
```json
{
"error": "",
"message": "Quota updated",
"status": 200
}
```
## Delete user
Command:
```console
python sftpgo_api_cli.py delete-user 9576
```
@@ -456,9 +492,9 @@ Output:
}
```
### Delete folder
## Delete folder
```
```console
python sftpgo_api_cli.py delete-folder /tmp/mapped_folder
```
@@ -472,11 +508,11 @@ Output:
}
```
### Get version
## Get version
Command:
```
```console
python sftpgo_api_cli.py get-version
```
@@ -490,11 +526,11 @@ Output:
}
```
### Get provider status
## Get provider status
Command:
```
```console
python sftpgo_api_cli.py get-provider-status
```
@@ -508,11 +544,11 @@ Output:
}
```
### Backup data
## Backup data
Command:
```
```console
python sftpgo_api_cli.py dumpdata backup.json --indent 1
```
@@ -526,11 +562,11 @@ Output:
}
```
### Restore data
## Restore data
Command:
```
```console
python sftpgo_api_cli.py loaddata /app/data/backups/backup.json --scan-quota 2 --mode 0
```
@@ -544,7 +580,7 @@ Output:
}
```
### Convert users from other stores
## Convert users from other stores
You can convert users to the SFTPGo format from the following users stores:
@@ -554,21 +590,21 @@ You can convert users to the SFTPGo format from the following users stores:
For details give a look at the `convert-users` subcommand usage:
```
```console
python sftpgo_api_cli.py convert-users --help
```
Let's see some examples:
```
```console
python sftpgo_api_cli.py convert-users "" unix-passwd unix_users.json --min-uid 500 --force-uid 1000 --force-gid 1000
```
```
```console
python sftpgo_api_cli.py convert-users pureftpd.passwd pure-ftpd pure_users.json --usernames "user1" "user2"
```
```
```console
python sftpgo_api_cli.py convert-users proftpd.passwd proftpd pro_users.json
```
@@ -576,7 +612,7 @@ The json file generated using the `convert-users` subcommand can be used as inpu
Please note that when importing Linux/Unix users the input file is not required: `/etc/passwd` and `/etc/shadow` are automatically parsed. `/etc/shadow` read permission is is typically granted to the `root` user, so you need to execute the `convert-users` subcommand as `root`.
### Colors highlight for Windows command prompt
## Colors highlight for Windows command prompt
If your Windows command prompt does not recognize ANSI/VT100 escape sequences you can download [ANSICON](https://github.com/adoxa/ansicon "ANSICON") extract proper files depending on your Windows OS, and install them using `ansicon -i`.
Thats all. From now on, your Windows command prompt will be aware of ANSI colors.

View File

@@ -40,6 +40,8 @@ class SFTPGoApiRequests:
self.providerStatusPath = urlparse.urljoin(baseUrl, '/api/v1/providerstatus')
self.dumpDataPath = urlparse.urljoin(baseUrl, '/api/v1/dumpdata')
self.loadDataPath = urlparse.urljoin(baseUrl, '/api/v1/loaddata')
self.updateUsedQuotaPath = urlparse.urljoin(baseUrl, "/api/v1/quota_update")
self.updateFolderUsedQuotaPath = urlparse.urljoin(baseUrl, "/api/v1/folder_quota_update")
self.debug = debug
if authType == 'basic':
self.auth = requests.auth.HTTPBasicAuth(authUser, authPassword)
@@ -284,6 +286,16 @@ class SFTPGoApiRequests:
r = requests.delete(urlparse.urljoin(self.userPath, 'user/' + str(user_id)), auth=self.auth, verify=self.verify)
self.printResponse(r)
def updateQuotaUsage(self, username, used_quota_size, used_quota_files, mode):
req = {"username":username, "used_quota_files":used_quota_files, "used_quota_size":used_quota_size}
r = requests.put(self.updateUsedQuotaPath, params={'mode':mode}, json=req, auth=self.auth, verify=self.verify)
self.printResponse(r)
def updateFolderQuotaUsage(self, mapped_path, used_quota_size, used_quota_files, mode):
req = {"mapped_path":mapped_path, "used_quota_files":used_quota_files, "used_quota_size":used_quota_size}
r = requests.put(self.updateFolderUsedQuotaPath, params={'mode':mode}, json=req, auth=self.auth, verify=self.verify)
self.printResponse(r)
def getConnections(self):
r = requests.get(self.activeConnectionsPath, auth=self.auth, verify=self.verify)
self.printResponse(r)
@@ -688,6 +700,22 @@ if __name__ == '__main__':
help='0 means new users are added, existing users are updated. 1 means new users are added,' +
' existing users are not modified. Default: %(default)s')
parserUpdateQuotaUsage = subparsers.add_parser('update-quota-usage', help='Update the user used quota limits')
parserUpdateQuotaUsage.add_argument('username', type=str)
parserUpdateQuotaUsage.add_argument('-M', '--mode', type=str, choices=["add", "reset"], default="reset",
help='the update mode specifies if the given quota usage values should be added or ' +
'replace the current ones. Default: %(default)s')
parserUpdateQuotaUsage.add_argument('-S', '--used_quota_size', type=int, default=0, help='Default: %(default)s')
parserUpdateQuotaUsage.add_argument('-F', '--used_quota_files', type=int, default=0, help='Default: %(default)s')
parserUpdateFolderQuotaUsage = subparsers.add_parser('update-folder-quota-usage', help='Update the folder used quota limits')
parserUpdateFolderQuotaUsage.add_argument('folder_path', type=str)
parserUpdateFolderQuotaUsage.add_argument('-M', '--mode', type=str, choices=["add", "reset"], default="reset",
help='the update mode specifies if the given quota usage values should be added or ' +
'replace the current ones. Default: %(default)s')
parserUpdateFolderQuotaUsage.add_argument('-S', '--used_quota_size', type=int, default=0, help='Default: %(default)s')
parserUpdateFolderQuotaUsage.add_argument('-F', '--used_quota_files', type=int, default=0, help='Default: %(default)s')
parserConvertUsers = subparsers.add_parser('convert-users', help='Convert users to a JSON format suitable to use ' +
'with loadddata')
supportedUsersFormats = []
@@ -765,6 +793,10 @@ if __name__ == '__main__':
api.dumpData(args.output_file, args.indent)
elif args.command == 'loaddata':
api.loadData(args.input_file, args.scan_quota, args.mode)
elif args.command == 'update-quota-usage':
api.updateQuotaUsage(args.username, args.used_quota_size, args.used_quota_files, args.mode)
elif args.command == 'update-folder-quota-usage':
api.updateFolderQuotaUsage(args.folder_path, args.used_quota_size, args.used_quota_files, args.mode)
elif args.command == 'convert-users':
convertUsers = ConvertUsers(args.input_file, args.users_format, args.output_file, args.min_uid, args.max_uid,
args.usernames, args.force_uid, args.force_gid)

View File

@@ -69,7 +69,7 @@ func addFolder(w http.ResponseWriter, r *http.Request) {
if err == nil {
render.JSON(w, r, folder)
} else {
sendAPIResponse(w, r, err, "", http.StatusInternalServerError)
sendAPIResponse(w, r, err, "", getRespStatus(err))
}
} else {
sendAPIResponse(w, r, err, "", getRespStatus(err))
@@ -88,11 +88,8 @@ func deleteFolderByPath(w http.ResponseWriter, r *http.Request) {
}
folder, err := dataprovider.GetFolderByPath(dataProvider, folderPath)
if _, ok := err.(*dataprovider.RecordNotFoundError); ok {
sendAPIResponse(w, r, err, "", http.StatusNotFound)
return
} else if err != nil {
sendAPIResponse(w, r, err, "", http.StatusInternalServerError)
if err != nil {
sendAPIResponse(w, r, err, "", getRespStatus(err))
return
}
err = dataprovider.DeleteFolder(dataProvider, folder)

View File

@@ -1,6 +1,7 @@
package httpd
import (
"errors"
"net/http"
"github.com/go-chi/render"
@@ -11,6 +12,11 @@ import (
"github.com/drakkan/sftpgo/vfs"
)
const (
quotaUpdateModeAdd = "add"
quotaUpdateModeReset = "reset"
)
func getQuotaScans(w http.ResponseWriter, r *http.Request) {
render.JSON(w, r, sftpd.GetQuotaScans())
}
@@ -19,8 +25,89 @@ func getVFolderQuotaScans(w http.ResponseWriter, r *http.Request) {
render.JSON(w, r, sftpd.GetVFoldersQuotaScans())
}
func updateUserQuotaUsage(w http.ResponseWriter, r *http.Request) {
r.Body = http.MaxBytesReader(w, r.Body, maxRequestSize)
var u dataprovider.User
err := render.DecodeJSON(r.Body, &u)
if err != nil {
sendAPIResponse(w, r, err, "", http.StatusBadRequest)
return
}
if u.UsedQuotaFiles < 0 || u.UsedQuotaSize < 0 {
sendAPIResponse(w, r, errors.New("Invalid used quota parameters, negative values are not allowed"),
"", http.StatusBadRequest)
return
}
mode, err := getQuotaUpdateMode(r)
if err != nil {
sendAPIResponse(w, r, err, "", http.StatusBadRequest)
return
}
user, err := dataprovider.UserExists(dataProvider, u.Username)
if err != nil {
sendAPIResponse(w, r, err, "", getRespStatus(err))
return
}
if mode == quotaUpdateModeAdd && !user.HasQuotaRestrictions() && dataprovider.GetQuotaTracking() == 2 {
sendAPIResponse(w, r, errors.New("this user has no quota restrictions, only reset mode is supported"),
"", http.StatusBadRequest)
return
}
if !sftpd.AddQuotaScan(user.Username) {
sendAPIResponse(w, r, err, "A quota scan is in progress for this user", http.StatusConflict)
return
}
defer sftpd.RemoveQuotaScan(user.Username) //nolint:errcheck
err = dataprovider.UpdateUserQuota(dataProvider, user, u.UsedQuotaFiles, u.UsedQuotaSize, mode == quotaUpdateModeReset)
if err != nil {
sendAPIResponse(w, r, err, "", getRespStatus(err))
} else {
sendAPIResponse(w, r, err, "Quota updated", http.StatusOK)
}
}
func updateVFolderQuotaUsage(w http.ResponseWriter, r *http.Request) {
r.Body = http.MaxBytesReader(w, r.Body, maxRequestSize)
var f vfs.BaseVirtualFolder
err := render.DecodeJSON(r.Body, &f)
if err != nil {
sendAPIResponse(w, r, err, "", http.StatusBadRequest)
return
}
if f.UsedQuotaFiles < 0 || f.UsedQuotaSize < 0 {
sendAPIResponse(w, r, errors.New("Invalid used quota parameters, negative values are not allowed"),
"", http.StatusBadRequest)
return
}
mode, err := getQuotaUpdateMode(r)
if err != nil {
sendAPIResponse(w, r, err, "", http.StatusBadRequest)
return
}
folder, err := dataprovider.GetFolderByPath(dataProvider, f.MappedPath)
if err != nil {
sendAPIResponse(w, r, err, "", getRespStatus(err))
return
}
if !sftpd.AddVFolderQuotaScan(folder.MappedPath) {
sendAPIResponse(w, r, err, "A quota scan is in progress for this folder", http.StatusConflict)
return
}
defer sftpd.RemoveVFolderQuotaScan(folder.MappedPath) //nolint:errcheck
err = dataprovider.UpdateVirtualFolderQuota(dataProvider, folder, f.UsedQuotaFiles, f.UsedQuotaSize, mode == quotaUpdateModeReset)
if err != nil {
sendAPIResponse(w, r, err, "", getRespStatus(err))
} else {
sendAPIResponse(w, r, err, "Quota updated", http.StatusOK)
}
}
func startQuotaScan(w http.ResponseWriter, r *http.Request) {
r.Body = http.MaxBytesReader(w, r.Body, maxRequestSize)
if dataprovider.GetQuotaTracking() == 0 {
sendAPIResponse(w, r, nil, "Quota tracking is disabled!", http.StatusForbidden)
return
}
var u dataprovider.User
err := render.DecodeJSON(r.Body, &u)
if err != nil {
@@ -29,11 +116,7 @@ func startQuotaScan(w http.ResponseWriter, r *http.Request) {
}
user, err := dataprovider.UserExists(dataProvider, u.Username)
if err != nil {
sendAPIResponse(w, r, err, "", http.StatusNotFound)
return
}
if dataprovider.GetQuotaTracking() == 0 {
sendAPIResponse(w, r, nil, "Quota tracking is disabled!", http.StatusForbidden)
sendAPIResponse(w, r, err, "", getRespStatus(err))
return
}
if sftpd.AddQuotaScan(user.Username) {
@@ -46,6 +129,10 @@ func startQuotaScan(w http.ResponseWriter, r *http.Request) {
func startVFolderQuotaScan(w http.ResponseWriter, r *http.Request) {
r.Body = http.MaxBytesReader(w, r.Body, maxRequestSize)
if dataprovider.GetQuotaTracking() == 0 {
sendAPIResponse(w, r, nil, "Quota tracking is disabled!", http.StatusForbidden)
return
}
var f vfs.BaseVirtualFolder
err := render.DecodeJSON(r.Body, &f)
if err != nil {
@@ -54,11 +141,7 @@ func startVFolderQuotaScan(w http.ResponseWriter, r *http.Request) {
}
folder, err := dataprovider.GetFolderByPath(dataProvider, f.MappedPath)
if err != nil {
sendAPIResponse(w, r, err, "", http.StatusNotFound)
return
}
if dataprovider.GetQuotaTracking() == 0 {
sendAPIResponse(w, r, nil, "Quota tracking is disabled!", http.StatusForbidden)
sendAPIResponse(w, r, err, "", getRespStatus(err))
return
}
if sftpd.AddVFolderQuotaScan(folder.MappedPath) {
@@ -98,3 +181,14 @@ func doFolderQuotaScan(folder vfs.BaseVirtualFolder) error {
logger.Debug(logSender, "", "virtual folder %#v scanned, error: %v", folder.MappedPath, err)
return err
}
func getQuotaUpdateMode(r *http.Request) (string, error) {
mode := quotaUpdateModeReset
if _, ok := r.URL.Query()["mode"]; ok {
mode = r.URL.Query().Get("mode")
if mode != quotaUpdateModeReset && mode != quotaUpdateModeAdd {
return "", errors.New("Invalid mode")
}
}
return mode, nil
}

View File

@@ -66,10 +66,8 @@ func getUserByID(w http.ResponseWriter, r *http.Request) {
user, err := dataprovider.GetUserByID(dataProvider, userID)
if err == nil {
render.JSON(w, r, dataprovider.HideUserSensitiveData(&user))
} else if _, ok := err.(*dataprovider.RecordNotFoundError); ok {
sendAPIResponse(w, r, err, "", http.StatusNotFound)
} else {
sendAPIResponse(w, r, err, "", http.StatusInternalServerError)
sendAPIResponse(w, r, err, "", getRespStatus(err))
}
}
@@ -87,7 +85,7 @@ func addUser(w http.ResponseWriter, r *http.Request) {
if err == nil {
render.JSON(w, r, dataprovider.HideUserSensitiveData(&user))
} else {
sendAPIResponse(w, r, err, "", http.StatusInternalServerError)
sendAPIResponse(w, r, err, "", getRespStatus(err))
}
} else {
sendAPIResponse(w, r, err, "", getRespStatus(err))
@@ -103,6 +101,10 @@ func updateUser(w http.ResponseWriter, r *http.Request) {
return
}
user, err := dataprovider.GetUserByID(dataProvider, userID)
if err != nil {
sendAPIResponse(w, r, err, "", getRespStatus(err))
return
}
currentPermissions := user.Permissions
currentFileExtensions := user.Filters.FileExtensions
currentS3AccessSecret := ""
@@ -111,13 +113,6 @@ func updateUser(w http.ResponseWriter, r *http.Request) {
}
user.Permissions = make(map[string][]string)
user.Filters.FileExtensions = []dataprovider.ExtensionsFilter{}
if _, ok := err.(*dataprovider.RecordNotFoundError); ok {
sendAPIResponse(w, r, err, "", http.StatusNotFound)
return
} else if err != nil {
sendAPIResponse(w, r, err, "", http.StatusInternalServerError)
return
}
err = render.DecodeJSON(r.Body, &user)
if err != nil {
sendAPIResponse(w, r, err, "", http.StatusBadRequest)
@@ -158,11 +153,8 @@ func deleteUser(w http.ResponseWriter, r *http.Request) {
return
}
user, err := dataprovider.GetUserByID(dataProvider, userID)
if _, ok := err.(*dataprovider.RecordNotFoundError); ok {
sendAPIResponse(w, r, err, "", http.StatusNotFound)
return
} else if err != nil {
sendAPIResponse(w, r, err, "", http.StatusInternalServerError)
if err != nil {
sendAPIResponse(w, r, err, "", getRespStatus(err))
return
}
err = dataprovider.DeleteUser(dataProvider, user)

View File

@@ -82,6 +82,9 @@ func getRespStatus(err error) int {
if _, ok := err.(*dataprovider.MethodDisabledError); ok {
return http.StatusForbidden
}
if _, ok := err.(*dataprovider.RecordNotFoundError); ok {
return http.StatusNotFound
}
if os.IsNotExist(err) {
return http.StatusBadRequest
}
@@ -218,7 +221,7 @@ func GetQuotaScans(expectedStatusCode int) ([]sftpd.ActiveQuotaScan, []byte, err
return quotaScans, body, err
}
// StartQuotaScan start a new quota scan for the given user and checks the received HTTP Status code against expectedStatusCode.
// StartQuotaScan starts a new quota scan for the given user and checks the received HTTP Status code against expectedStatusCode.
func StartQuotaScan(user dataprovider.User, expectedStatusCode int) ([]byte, error) {
var body []byte
userAsJSON, _ := json.Marshal(user)
@@ -231,6 +234,23 @@ func StartQuotaScan(user dataprovider.User, expectedStatusCode int) ([]byte, err
return body, checkResponse(resp.StatusCode, expectedStatusCode)
}
// UpdateQuotaUsage updates the user used quota limits and checks the received HTTP Status code against expectedStatusCode.
func UpdateQuotaUsage(user dataprovider.User, mode string, expectedStatusCode int) ([]byte, error) {
var body []byte
userAsJSON, _ := json.Marshal(user)
url, err := addModeQueryParam(buildURLRelativeToBase(updateUsedQuotaPath), mode)
if err != nil {
return body, err
}
resp, err := sendHTTPRequest(http.MethodPut, url.String(), bytes.NewBuffer(userAsJSON), "")
if err != nil {
return body, err
}
defer resp.Body.Close()
body, _ = getResponseBody(resp)
return body, checkResponse(resp.StatusCode, expectedStatusCode)
}
// GetConnections returns status and stats for active SFTP/SCP connections
func GetConnections(expectedStatusCode int) ([]sftpd.ConnectionStatus, []byte, error) {
var connections []sftpd.ConnectionStatus
@@ -370,6 +390,23 @@ func StartFolderQuotaScan(folder vfs.BaseVirtualFolder, expectedStatusCode int)
return body, checkResponse(resp.StatusCode, expectedStatusCode)
}
// UpdateFolderQuotaUsage updates the folder used quota limits and checks the received HTTP Status code against expectedStatusCode.
func UpdateFolderQuotaUsage(folder vfs.BaseVirtualFolder, mode string, expectedStatusCode int) ([]byte, error) {
var body []byte
folderAsJSON, _ := json.Marshal(folder)
url, err := addModeQueryParam(buildURLRelativeToBase(updateFolderUsedQuotaPath), mode)
if err != nil {
return body, err
}
resp, err := sendHTTPRequest(http.MethodPut, url.String(), bytes.NewBuffer(folderAsJSON), "")
if err != nil {
return body, err
}
defer resp.Body.Close()
body, _ = getResponseBody(resp)
return body, checkResponse(resp.StatusCode, expectedStatusCode)
}
// GetVersion returns version details
func GetVersion(expectedStatusCode int) (version.Info, []byte, error) {
var appVersion version.Info
@@ -778,3 +815,16 @@ func addLimitAndOffsetQueryParams(rawurl string, limit, offset int64) (*url.URL,
url.RawQuery = q.Encode()
return url, err
}
func addModeQueryParam(rawurl, mode string) (*url.URL, error) {
url, err := url.Parse(rawurl)
if err != nil {
return nil, err
}
q := url.Query()
if len(mode) > 0 {
q.Add("mode", mode)
}
url.RawQuery = q.Encode()
return url, err
}

View File

@@ -32,6 +32,8 @@ const (
providerStatusPath = "/api/v1/providerstatus"
dumpDataPath = "/api/v1/dumpdata"
loadDataPath = "/api/v1/loaddata"
updateUsedQuotaPath = "/api/v1/quota_update"
updateFolderUsedQuotaPath = "/api/v1/folder_quota_update"
metricsPath = "/metrics"
pprofBasePath = "/debug"
webBasePath = "/web"

View File

@@ -47,6 +47,8 @@ const (
activeConnectionsPath = "/api/v1/connection"
quotaScanPath = "/api/v1/quota_scan"
quotaScanVFolderPath = "/api/v1/folder_quota_scan"
updateUsedQuotaPath = "/api/v1/quota_update"
updateFolderUsedQuotaPath = "/api/v1/folder_quota_update"
versionPath = "/api/v1/version"
metricsPath = "/metrics"
pprofPath = "/debug/pprof/"
@@ -603,8 +605,13 @@ func TestUserPublicKey(t *testing.T) {
}
func TestUpdateUser(t *testing.T) {
user, _, err := httpd.AddUser(getTestUser(), http.StatusOK)
u := getTestUser()
u.UsedQuotaFiles = 1
u.UsedQuotaSize = 2
user, _, err := httpd.AddUser(u, http.StatusOK)
assert.NoError(t, err)
assert.Equal(t, 0, user.UsedQuotaFiles)
assert.Equal(t, int64(0), user.UsedQuotaSize)
user.HomeDir = filepath.Join(homeBasePath, "testmod")
user.UID = 33
user.GID = 101
@@ -683,6 +690,48 @@ func TestUpdateUser(t *testing.T) {
}
}
func TestUpdateUserQuotaUsage(t *testing.T) {
u := getTestUser()
usedQuotaFiles := 1
usedQuotaSize := int64(65535)
u.UsedQuotaFiles = usedQuotaFiles
u.UsedQuotaSize = usedQuotaSize
user, _, err := httpd.AddUser(u, http.StatusOK)
assert.NoError(t, err)
_, err = httpd.UpdateQuotaUsage(u, "invalid_mode", http.StatusBadRequest)
assert.NoError(t, err)
_, err = httpd.UpdateQuotaUsage(u, "", http.StatusOK)
assert.NoError(t, err)
user, _, err = httpd.GetUserByID(user.ID, http.StatusOK)
assert.NoError(t, err)
assert.Equal(t, usedQuotaFiles, user.UsedQuotaFiles)
assert.Equal(t, usedQuotaSize, user.UsedQuotaSize)
_, err = httpd.UpdateQuotaUsage(u, "add", http.StatusBadRequest)
assert.NoError(t, err, "user has no quota restrictions add mode should fail")
user, _, err = httpd.GetUserByID(user.ID, http.StatusOK)
assert.NoError(t, err)
assert.Equal(t, usedQuotaFiles, user.UsedQuotaFiles)
assert.Equal(t, usedQuotaSize, user.UsedQuotaSize)
user.QuotaFiles = 100
user, _, err = httpd.UpdateUser(user, http.StatusOK)
assert.NoError(t, err)
_, err = httpd.UpdateQuotaUsage(u, "add", http.StatusOK)
assert.NoError(t, err)
user, _, err = httpd.GetUserByID(user.ID, http.StatusOK)
assert.NoError(t, err)
assert.Equal(t, 2*usedQuotaFiles, user.UsedQuotaFiles)
assert.Equal(t, 2*usedQuotaSize, user.UsedQuotaSize)
u.UsedQuotaFiles = -1
_, err = httpd.UpdateQuotaUsage(u, "", http.StatusBadRequest)
assert.NoError(t, err)
u.UsedQuotaFiles = usedQuotaFiles
u.Username = u.Username + "1"
_, err = httpd.UpdateQuotaUsage(u, "", http.StatusNotFound)
assert.NoError(t, err)
_, err = httpd.RemoveUser(user, http.StatusOK)
assert.NoError(t, err)
}
func TestUserFolderMapping(t *testing.T) {
mappedPath1 := filepath.Join(os.TempDir(), "mapped_dir1")
mappedPath2 := filepath.Join(os.TempDir(), "mapped_dir2")
@@ -1038,6 +1087,47 @@ func TestStartQuotaScan(t *testing.T) {
assert.NoError(t, err)
}
func TestUpdateFolderQuotaUsage(t *testing.T) {
f := vfs.BaseVirtualFolder{
MappedPath: filepath.Join(os.TempDir(), "folder"),
}
usedQuotaFiles := 1
usedQuotaSize := int64(65535)
f.UsedQuotaFiles = usedQuotaFiles
f.UsedQuotaSize = usedQuotaSize
folder, _, err := httpd.AddFolder(f, http.StatusOK)
assert.NoError(t, err)
_, err = httpd.UpdateFolderQuotaUsage(folder, "invalid mode", http.StatusBadRequest)
assert.NoError(t, err)
_, err = httpd.UpdateFolderQuotaUsage(f, "reset", http.StatusOK)
assert.NoError(t, err)
folders, _, err := httpd.GetFolders(0, 0, f.MappedPath, http.StatusOK)
assert.NoError(t, err)
if assert.Len(t, folders, 1) {
folder = folders[0]
assert.Equal(t, usedQuotaFiles, folder.UsedQuotaFiles)
assert.Equal(t, usedQuotaSize, folder.UsedQuotaSize)
}
_, err = httpd.UpdateFolderQuotaUsage(f, "add", http.StatusOK)
assert.NoError(t, err)
folders, _, err = httpd.GetFolders(0, 0, f.MappedPath, http.StatusOK)
assert.NoError(t, err)
if assert.Len(t, folders, 1) {
folder = folders[0]
assert.Equal(t, 2*usedQuotaFiles, folder.UsedQuotaFiles)
assert.Equal(t, 2*usedQuotaSize, folder.UsedQuotaSize)
}
f.UsedQuotaSize = -1
_, err = httpd.UpdateFolderQuotaUsage(f, "", http.StatusBadRequest)
assert.NoError(t, err)
f.UsedQuotaSize = usedQuotaSize
f.MappedPath = f.MappedPath + "1"
_, err = httpd.UpdateFolderQuotaUsage(f, "", http.StatusNotFound)
assert.NoError(t, err)
_, err = httpd.RemoveFolder(folder, http.StatusOK)
assert.NoError(t, err)
}
func TestGetVersion(t *testing.T) {
_, _, err := httpd.GetVersion(http.StatusOK)
assert.NoError(t, err)
@@ -1115,6 +1205,8 @@ func TestQuotaTrackingDisabled(t *testing.T) {
assert.NoError(t, err)
_, err = httpd.StartQuotaScan(user, http.StatusForbidden)
assert.NoError(t, err)
_, err = httpd.UpdateQuotaUsage(user, "", http.StatusForbidden)
assert.NoError(t, err)
_, err = httpd.RemoveUser(user, http.StatusOK)
assert.NoError(t, err)
// folder quota scan must fail
@@ -1125,6 +1217,8 @@ func TestQuotaTrackingDisabled(t *testing.T) {
assert.NoError(t, err)
_, err = httpd.StartFolderQuotaScan(folder, http.StatusForbidden)
assert.NoError(t, err)
_, err = httpd.UpdateFolderQuotaUsage(folder, "", http.StatusForbidden)
assert.NoError(t, err)
_, err = httpd.RemoveFolder(folder, http.StatusOK)
assert.NoError(t, err)
@@ -1554,6 +1648,42 @@ func TestUpdateUserMock(t *testing.T) {
checkResponseCode(t, http.StatusOK, rr.Code)
}
func TestUpdateUserQuotaUsageMock(t *testing.T) {
var user dataprovider.User
u := getTestUser()
usedQuotaFiles := 1
usedQuotaSize := int64(65535)
u.UsedQuotaFiles = usedQuotaFiles
u.UsedQuotaSize = usedQuotaSize
userAsJSON := getUserAsJSON(t, u)
req, _ := http.NewRequest(http.MethodPost, userPath, bytes.NewBuffer(userAsJSON))
rr := executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
err := render.DecodeJSON(rr.Body, &user)
assert.NoError(t, err)
req, _ = http.NewRequest(http.MethodPut, updateUsedQuotaPath, bytes.NewBuffer(userAsJSON))
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
req, _ = http.NewRequest(http.MethodGet, userPath+"/"+strconv.FormatInt(user.ID, 10), nil)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
err = render.DecodeJSON(rr.Body, &user)
assert.NoError(t, err)
assert.Equal(t, usedQuotaFiles, user.UsedQuotaFiles)
assert.Equal(t, usedQuotaSize, user.UsedQuotaSize)
req, _ = http.NewRequest(http.MethodPut, updateUsedQuotaPath, bytes.NewBuffer([]byte("string")))
rr = executeRequest(req)
checkResponseCode(t, http.StatusBadRequest, rr.Code)
assert.True(t, sftpd.AddQuotaScan(user.Username))
req, _ = http.NewRequest(http.MethodPut, updateUsedQuotaPath, bytes.NewBuffer(userAsJSON))
rr = executeRequest(req)
checkResponseCode(t, http.StatusConflict, rr.Code)
assert.NoError(t, sftpd.RemoveQuotaScan(user.Username))
req, _ = http.NewRequest(http.MethodDelete, userPath+"/"+strconv.FormatInt(user.ID, 10), nil)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
}
func TestUserPermissionsMock(t *testing.T) {
user := getTestUser()
user.Permissions = make(map[string][]string)
@@ -1784,6 +1914,64 @@ func TestStartQuotaScanMock(t *testing.T) {
assert.NoError(t, err)
}
func TestUpdateFolderQuotaUsageMock(t *testing.T) {
mappedPath := filepath.Join(os.TempDir(), "vfolder")
f := vfs.BaseVirtualFolder{
MappedPath: mappedPath,
}
usedQuotaFiles := 1
usedQuotaSize := int64(65535)
f.UsedQuotaFiles = usedQuotaFiles
f.UsedQuotaSize = usedQuotaSize
var folder vfs.BaseVirtualFolder
folderAsJSON, err := json.Marshal(f)
assert.NoError(t, err)
req, _ := http.NewRequest(http.MethodPost, folderPath, bytes.NewBuffer(folderAsJSON))
rr := executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
err = render.DecodeJSON(rr.Body, &folder)
assert.NoError(t, err)
req, _ = http.NewRequest(http.MethodPut, updateFolderUsedQuotaPath, bytes.NewBuffer(folderAsJSON))
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
var folders []vfs.BaseVirtualFolder
url, err := url.Parse(folderPath)
assert.NoError(t, err)
q := url.Query()
q.Add("folder_path", mappedPath)
url.RawQuery = q.Encode()
req, _ = http.NewRequest(http.MethodGet, url.String(), nil)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
err = render.DecodeJSON(rr.Body, &folders)
assert.NoError(t, err)
if assert.Len(t, folders, 1) {
folder = folders[0]
assert.Equal(t, usedQuotaFiles, folder.UsedQuotaFiles)
assert.Equal(t, usedQuotaSize, folder.UsedQuotaSize)
}
req, _ = http.NewRequest(http.MethodPut, updateFolderUsedQuotaPath, bytes.NewBuffer([]byte("string")))
rr = executeRequest(req)
checkResponseCode(t, http.StatusBadRequest, rr.Code)
assert.True(t, sftpd.AddVFolderQuotaScan(mappedPath))
req, _ = http.NewRequest(http.MethodPut, updateFolderUsedQuotaPath, bytes.NewBuffer(folderAsJSON))
rr = executeRequest(req)
checkResponseCode(t, http.StatusConflict, rr.Code)
assert.NoError(t, sftpd.RemoveVFolderQuotaScan(mappedPath))
url, err = url.Parse(folderPath)
assert.NoError(t, err)
q = url.Query()
q.Add("folder_path", mappedPath)
url.RawQuery = q.Encode()
req, _ = http.NewRequest(http.MethodDelete, url.String(), nil)
rr = executeRequest(req)
checkResponseCode(t, http.StatusOK, rr.Code)
}
func TestStartFolderQuotaScanMock(t *testing.T) {
mappedPath := filepath.Join(os.TempDir(), "vfolder")
folder := vfs.BaseVirtualFolder{

View File

@@ -357,17 +357,24 @@ func TestApiCallsWithBadURL(t *testing.T) {
oldAuthUsername := authUsername
oldAuthPassword := authPassword
SetBaseURLAndCredentials(invalidURL, oldAuthUsername, oldAuthPassword)
folder := vfs.BaseVirtualFolder{
MappedPath: os.TempDir(),
}
u := dataprovider.User{}
_, _, err := UpdateUser(u, http.StatusBadRequest)
assert.Error(t, err)
_, err = RemoveUser(u, http.StatusNotFound)
assert.Error(t, err)
_, err = RemoveFolder(vfs.BaseVirtualFolder{}, http.StatusNotFound)
_, err = RemoveFolder(folder, http.StatusNotFound)
assert.Error(t, err)
_, _, err = GetUsers(1, 0, "", http.StatusBadRequest)
assert.Error(t, err)
_, _, err = GetFolders(1, 0, "", http.StatusBadRequest)
assert.Error(t, err)
_, err = UpdateQuotaUsage(u, "", http.StatusNotFound)
assert.Error(t, err)
_, err = UpdateFolderQuotaUsage(folder, "", http.StatusNotFound)
assert.Error(t, err)
_, err = CloseConnection("non_existent_id", http.StatusNotFound)
assert.Error(t, err)
_, _, err = Dumpdata("backup.json", "", http.StatusBadRequest)
@@ -393,6 +400,8 @@ func TestApiCallToNotListeningServer(t *testing.T) {
assert.Error(t, err)
_, _, err = GetUsers(100, 0, "", http.StatusOK)
assert.Error(t, err)
_, err = UpdateQuotaUsage(u, "", http.StatusNotFound)
assert.Error(t, err)
_, _, err = GetQuotaScans(http.StatusOK)
assert.Error(t, err)
_, err = StartQuotaScan(u, http.StatusNotFound)
@@ -408,6 +417,8 @@ func TestApiCallToNotListeningServer(t *testing.T) {
assert.Error(t, err)
_, _, err = GetFolders(0, 0, "", http.StatusOK)
assert.Error(t, err)
_, err = UpdateFolderQuotaUsage(folder, "", http.StatusNotFound)
assert.Error(t, err)
_, _, err = GetFoldersQuotaScans(http.StatusOK)
assert.Error(t, err)
_, _, err = GetConnections(http.StatusOK)

View File

@@ -86,6 +86,8 @@ func initializeRouter(staticFilesPath string, enableProfiler, enableWebAdmin boo
router.Delete(folderPath, deleteFolderByPath)
router.Get(dumpDataPath, dumpData)
router.Get(loadDataPath, loadData)
router.Put(updateUsedQuotaPath, updateUserQuotaUsage)
router.Put(updateFolderUsedQuotaPath, updateVFolderQuotaUsage)
if enableWebAdmin {
router.Get(webUsersPath, handleGetWebUsers)
router.Get(webUserPath, handleWebAddUserGet)

View File

@@ -2,7 +2,7 @@ openapi: 3.0.1
info:
title: SFTPGo
description: 'SFTPGo REST API'
version: 1.9.0
version: 1.9.1
servers:
- url: /api/v1
@@ -348,6 +348,202 @@ paths:
status: 500
message: ""
error: "Error description if any"
/quota_update:
put:
tags:
- quota
summary: update the user used quota limits
description: Set the current used quota limits for the given user
operationId: quota_update
parameters:
- in: query
name: mode
required: false
description: the update mode specifies if the given quota usage values should be added or replace the current ones
schema:
type: string
enum: [add, reset]
description: >
Update type:
* `add` - add the specified quota limits to the current used ones
* `reset` - reset the values to the specified ones. This is the default
example: reset
requestBody:
required: true
description: The only user mandatory fields are username,used_quota_size and used_quota_files. Please note that if the quota fields are missing they will default to 0
content:
application/json:
schema:
$ref : '#/components/schemas/User'
responses:
200:
description: successful operation
content:
application/json:
schema:
$ref: '#/components/schemas/ApiResponse'
example:
status: 200
message: "Quota updated"
error: ""
400:
description: Bad request
content:
application/json:
schema:
$ref: '#/components/schemas/ApiResponse'
example:
status: 400
message: ""
error: "Error description if any"
401:
description: Unauthorized
content:
application/json:
schema:
$ref: '#/components/schemas/ApiResponse'
example:
status: 401
message: ""
error: "Error description if any"
403:
description: Forbidden
content:
application/json:
schema:
$ref: '#/components/schemas/ApiResponse'
example:
status: 403
message: ""
error: "Error description if any"
404:
description: Not Found
content:
application/json:
schema:
$ref: '#/components/schemas/ApiResponse'
example:
status: 404
message: ""
error: "Error description if any"
409:
description: A quota scan is in progress for this user
content:
application/json:
schema:
$ref: '#/components/schemas/ApiResponse'
example:
status: 409
message: "A quota scan is in progress"
error: "Error description if any"
500:
description: Internal Server Error
content:
application/json:
schema:
$ref: '#/components/schemas/ApiResponse'
example:
status: 500
message: ""
error: "Error description if any"
/folder_quota_update:
put:
tags:
- quota
summary: update the folder used quota limits
description: Set the current used quota limits for the given folder
operationId: folder_quota_update
parameters:
- in: query
name: mode
required: false
description: the update mode specifies if the given quota usage values should be added or replace the current ones
schema:
type: string
enum: [add, reset]
description: >
Update type:
* `add` - add the specified quota limits to the current used ones
* `reset` - reset the values to the specified ones. This is the default
example: reset
requestBody:
required: true
description: The only folder mandatory fields are mapped_path,used_quota_size and used_quota_files. Please note that if the used quota fields are missing they will default to 0
content:
application/json:
schema:
$ref : '#/components/schemas/BaseVirtualFolder'
responses:
200:
description: successful operation
content:
application/json:
schema:
$ref: '#/components/schemas/ApiResponse'
example:
status: 200
message: "Quota updated"
error: ""
400:
description: Bad request
content:
application/json:
schema:
$ref: '#/components/schemas/ApiResponse'
example:
status: 400
message: ""
error: "Error description if any"
401:
description: Unauthorized
content:
application/json:
schema:
$ref: '#/components/schemas/ApiResponse'
example:
status: 401
message: ""
error: "Error description if any"
403:
description: Forbidden
content:
application/json:
schema:
$ref: '#/components/schemas/ApiResponse'
example:
status: 403
message: ""
error: "Error description if any"
404:
description: Not Found
content:
application/json:
schema:
$ref: '#/components/schemas/ApiResponse'
example:
status: 404
message: ""
error: "Error description if any"
409:
description: A quota scan is in progress for this folder
content:
application/json:
schema:
$ref: '#/components/schemas/ApiResponse'
example:
status: 409
message: "A quota scan is in progress"
error: "Error description if any"
500:
description: Internal Server Error
content:
application/json:
schema:
$ref: '#/components/schemas/ApiResponse'
example:
status: 500
message: ""
error: "Error description if any"
/folder_quota_scan:
get:
tags: