Tomasz Muras
Technical blog.

Limit process CPU use with cgroups version 1 and cgroup tools

  • 2022-09-22 
#!/bin/bash
sudo cgdelete cpu:/testgroup
sudo cgcreate -g cpu:/testgroup

# Test with and without the line below
sudo cgset -r cpu.cfs_quota_us=50000 testgroup

# See how much time do we need to complete the operation
echo "Starting CPU intensive operations"
date
stress-ng --qsort 1 --qsort-ops 200 &

sleep 1

PID=$(pgrep stress-ng-qsort)

sudo cgclassify -g cpu:testgroup $PID

wait
echo "Finished"
date

The results with cfs_quota_us=50000 are below. The top utility shows about 50% CPU usage.

➜ ./cpu_stress.sh                                                                       
Starting CPU intensive operations
czw, 22 wrz 2022, 17:41:33 CEST
stress-ng: info:  [3345901] defaulting to a 86400 second (1 day, 0.00 secs) run per stressor
stress-ng: info:  [3345901] dispatching hogs: 1 qsort
stress-ng: info:  [3345901] successful run completed in 33.05s
Finished
czw, 22 wrz 2022, 17:42:06 CEST

If the line is commented out (so no limitations apply), top shows 100% CPU usage and the output is:

➜ ./cpu_stress.sh
Starting CPU intensive operations
czw, 22 wrz 2022, 17:42:53 CEST
stress-ng: info:  [3348203] defaulting to a 86400 second (1 day, 0.00 secs) run per stressor
stress-ng: info:  [3348203] dispatching hogs: 1 qsort
stress-ng: info:  [3348203] successful run completed in 15.98s
Finished
czw, 22 wrz 2022, 17:43:09 CEST

The corresponding file entries in sysfs:

$ cat /sys/fs/cgroup/cpu/testgroup/cpu.cfs_quota_us
50000
$ cat /sys/fs/cgroup/cpu/testgroup/tasks
3353629

Book Linux Service Management Made Easy with systemd by Donald A. Tevault

  • 2022-09-04

systemd_cover

Monitoring files and directories

How to change editor for “systemctl edit command”.

Add to .bashrc / .zshrc:

 export SYSTEMD_EDITOR=vim

Add to /etc/sudoers:

 Defaults	env_keep += "SYSTEMD_EDITOR"

Get / set default target when booting up:

$ systemctl get-default
$ systemctl set-default ...target

Create a new service:

sudo systemctl edit --force --full new.service

Limit CPU usage for user:

sudo systemctl set-property user-1001.slice CPUQuota=10%
sudo systemctl daemon-reload

Limit IO read rate to 1MB/sec for user:

sudo systemctl set-property user-1001.slice BlockIOReadBandwidth="/dev/sda 1M"
sudo systemctl daemon-reload

journald log files:

  • transient in /run/log/journal/
  • persistent in /var/log/journal/ (create directory to enable)

journald log entries since last boot:

journalctl -b

Enforce memory limit in systemd service

  • 2022-09-01

Create new service with MemoryHigh and MemoryMax directives.

$ systemctl edit --force --full memory.service
[Unit]
Description=Simple service to test memory limit.

[Service]
ExecStart=/root/memory.sh
MemoryHigh=1M
MemoryMax=2M

[Install]
WantedBy=multi-user.target

The content of /root/memory.sh:

#!/bin/bash

echo $(date) > /tmp/test.log
a=()

for (( a=1; a<=10; a++ ))
do
    echo Loop $a >> /tmp/test.log
    for (( c=1; c<=600000; c++ ))
    do
        a+=( "abcdefghijklmnopqrstquvxyabcdefghijklmnopqrstquvxyzabcdefghijklmnopqrstquvxyzabcdefghijklmnopqrstquvxyzabcdefghijklmnopqrstquvxyzabcdefghijklmnopqrstquvxyzabcdefghijklmnopqrstquvxyzabcdefghijklmnopqrstquvxyzabcdefghijklmnopqrstquvxyzabcdefghijklmnopqrstquvxyzabcdefghijklmnopqrstquvxyzzabcdefghijklmnopqrstquvxyabcdefghijklmnopqrstquvxyzabcdefghijklmnopqrstquvxyzabcdefghijklmnopqrstquvxyzabcdefghijklmnopqrstquvxyzabcdefghijklmnopqrstquvxyzabcdefghijklmnopqrstquvxyzabcdefghijklmnopqrstquvxyzabcdefghijklmnopqrstquvxyzabcdefghijklmnopqrstquvxyzabcdefghijklmnopqrstquvxyzzabcdefghijklmnopqrstquvxyabcdefghijklmnopqrstquvxyzabcdefghijklmnopqrstquvxyzabcdefghijklmnopqrstquvxyzabcdefghijklmnopqrstquvxyzabcdefghijklmnopqrstquvxyzabcdefghijklmnopqrstquvxyzabcdefghijklmnopqrstquvxyzabcdefghijklmnopqrstquvxyzabcdefghijklmnopqrstquvxyzabcdefghijklmnopqrstquvxyzz" )
    done
done

sleep 10

Start the service:

root@tuxedo:/etc/systemd/system# systemctl daemon-reload
root@tuxedo:/etc/systemd/system# systemctl enable --now memory
root@tuxedo:/etc/systemd/system# systemctl status memory
● memory.service - Simple service to test memory limit.
Loaded: loaded (/etc/systemd/system/memory.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2022-09-01 22:26:27 CEST; 9s ago
Main PID: 14675 (memory.sh)
Tasks: 1 (limit: 76224)
Memory: 1.9M (high: 1.0M max: 2.0M)
CGroup: /system.slice/_memory.service
└─14675 /bin/bash /root/memory.sh

wrz 01 22:26:27 tuxedo systemd[1]: Started Simple service to test memory limit..

After a while

root@tuxedo:/etc/systemd/system# systemctl status memory
● memory.service - Simple service to test memory limit.
Loaded: loaded (/etc/systemd/system/memory.service; enabled; vendor preset: enabled)
Active: failed (Result: signal) since Thu 2022-09-01 22:27:31 CEST; 8s ago
Process: 14675 ExecStart=/root/memory.sh (code=killed, signal=KILL)
Main PID: 14675 (code=killed, signal=KILL)

wrz 01 22:26:27 tuxedo systemd[1]: Started Simple service to test memory limit..
wrz 01 22:27:31 tuxedo systemd[1]: memory.service: Main process exited, code=killed, status=9/KILL
wrz 01 22:27:31 tuxedo systemd[1]: memory.service: Failed with result 'signal'.

And in the dmesg:

$ dmesg
[ 5679.682307] Tasks state (memory values in pages):
[ 5679.682308] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[ 5679.682310] [  14675]     0 14675   202158      862  1646592   199134             0 memory.sh
[ 5679.682316] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/system.slice/_memory.service,task_memcg=/system.slice/_memory.service,task=memory.sh,pid=14675,uid=0
[ 5679.682330] Memory cgroup out of memory: Killed process 14675 (memory.sh) total-vm:808632kB, anon-rss:0kB, file-rss:3448kB, shmem-rss:0kB, UID:0 pgtables:1608kB oom_score_adj:0

Enforce process limit in systemd service

  • 2022-08-30

Simplest possible service that runs /root/process.sh script:

root@tuxedo:/etc/systemd/system# cat process.service

[Unit]
Description=Simple service to test process limit.

[Service]
ExecStart=/root/process.sh

[Install]
WantedBy=multi-user.target

The content of /root/process.sh:

#!/bin/bash

echo $(date) >> /tmp/test.log
for (( c=1; c<=10; c++ ))
do
sleep 10 &
done

sleep 20

Start the service:

root@tuxedo:/etc/systemd/system# systemctl daemon-reload
root@tuxedo:/etc/systemd/system# systemctl enable --now process
root@tuxedo:/etc/systemd/system# systemctl status process
● process.service - Simple service to test process limit.
Loaded: loaded (/etc/systemd/system/process.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2022-08-30 22:59:03 CEST; 2s ago
Main PID: 18596 (process.sh)
Tasks: 12 (limit: 76224)
Memory: 2.3M
CGroup: /system.slice/process.service
├─18596 /bin/bash /root/process.sh
├─18598 sleep 10
├─18599 sleep 10
├─18600 sleep 10
├─18601 sleep 10
├─18602 sleep 10
├─18603 sleep 10
├─18604 sleep 10
├─18605 sleep 10
├─18606 sleep 10
├─18607 sleep 10
└─18608 sleep 20

sie 30 22:59:03 tuxedo systemd[1]: Started Simple service to test process limit..

Now add TasksMax=5 into [Service] section:

[Unit]
Description=Simple service to test process limit.

[Service]
ExecStart=/root/process.sh
TasksMax=5

[Install]
WantedBy=multi-user.target

and restart the service:

root@tuxedo:/etc/systemd/system# systemctl daemon-reload
root@tuxedo:/etc/systemd/system# systemctl restart process
root@tuxedo:/etc/systemd/system# systemctl status process
● process.service - Simple service to test process limit.
Loaded: loaded (/etc/systemd/system/process.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2022-08-30 23:04:20 CEST; 9s ago
Main PID: 18961 (process.sh)
Tasks: 5 (limit: 5)
Memory: 1.0M
CGroup: /system.slice/process.service
├─18961 /bin/bash /root/process.sh
├─18963 sleep 10
├─18964 sleep 10
├─18965 sleep 10
└─18966 sleep 10

sie 30 23:04:20 tuxedo systemd[1]: Started Simple service to test process limit..
sie 30 23:04:20 tuxedo process.sh[18961]: /root/process.sh: fork: retry: Resource temporarily unavailab>
sie 30 23:04:21 tuxedo process.sh[18961]: /root/process.sh: fork: retry: Resource temporarily unavailab>
sie 30 23:04:23 tuxedo process.sh[18961]: /root/process.sh: fork: retry: Resource temporarily unavailab>
sie 30 23:04:27 tuxedo process.sh[18961]: /root/process.sh: fork: retry: Resource temporarily unavailab>
sie 30 23:04:30 tuxedo process.sh[18961]: /root/process.sh: fork: Interrupted system call
sie 30 23:04:30 tuxedo systemd[1]: process.service: Main process exited, code=exited, status=254/n/a
sie 30 23:04:30 tuxedo systemd[1]: process.service: Failed with result 'exit-code'.

Book Keycloak - Identity and Access Management for Modern Applications by Stian Thorgersen and Pedro Igor Silva

  • 2022-08-07

Keycloak_cover

Roles defined in OAuth 2.0:

  • Resource owner
  • Resource server
  • Client
  • Authorization server

Which flow to use:

  • Client Credentials flow if the application is accessing the resource on behalf of itself (the application is the resource owner)
  • Device flow if the application is running on a device without a browser or is input-constrained (i.e. smart TV)
  • Otherwise Authorization Code flow.

Do not use Implicit flow or Resource Owner Password Credentials flow at all.

Strategies that can be used to limit access for a specific access token:

  • Audience
  • Roles
  • Scope

To verify an access token:

  • Retrieve public key from JWKS endpoint
  • Verify the signature
  • Verify that token is not expired
  • Verify the issuer, audience and type of token
  • Verify any other claims that your application cares about

Internal applications (first-party) are those owned byt the enterprise. It may be self-hosted of SaaS. No need for user’s consent.

External applications (third-party) should require user’s consent.

Possible architectures of an application we are securing:

  • Server side
  • SPA (Single Page Application) with dedicated REST API under the same domain.
  • SPA with intermediary API under the same domain (which it turn may call external APIs)
  • SPA with external API

Securing native and mobile applications. How to return authorization code to the application:

  • Claimed HTTPS scheme. Native application claims scheme like https://my.app.org. Such URL then open in the app instead of a browser.
  • Custom URI scheme. Such request is sent (open) by application. Example: org.app.my://oauth2/provider-name .
  • Loopback interface. Application opens temporary web server on random port,i.e. https://127.0.0.1:2345
  • Special redirect URI, i.e. urn:ietf:wg:oauth:2.0:oob - authorization code is displayed by Keycloak for manual copy & paste.

Example authorization strategies:

  • role-based access control (RBAC). Keycloak has realm and client-level roles.
  • group-based access control (GBAC)
  • OAuth2 scopes
  • attribute-based access control (ABAC)

Keycloak cal act as a cenralized authorization service through a functionality called Authorization Services.

For production set up:

  • properties.frontendURL
  • properties.forceBackendUrlToFrontendURL
  • properties.adminURL (make it private URL)
  • TLS
  • production database (encryption in transit and at rest)
  • possibly clustering and load balancing
  • password policy
  • enable refresh token rotation
  • use ECDSA (Elliptic Curve Digital Signature Algorithm) for signatures instead of RSA

Keycloak uses Apache Freemaker for templates.

How to configure OpenID Connect with github

  • 2022-05-29

The example below is based on Moodle but similar will apply to any open OpenID Connect (OIDC) Relying Party. Moodle URL used here as example is https://test.pycert.org. Replace it with your installation URL.

Login to github and register new oauth app.

  • Application name can be anything you want
  • Homepage URL is your Moodle URL, i.e. https://test.pycert.org
  • Authorization callback URL is your Moodle URL + /admin/oauth2callback.php, i.e. https://test.pycert.org/admin/oauth2callback.php

From the next page note:

  • Client ID
  • Client secret (Generate a new client secret)

Go to Moodle, login as admin. Under: Site administration -> Authentication -> Manage authentication (https://test.pycert.org/admin/settings.php?section=manageauths) enable OAuth2.

Go to Site administration -> Server -> OAuth 2 services and click on “Custom”.

  • Name - anything you want, i.e. github.
  • Client ID - copied from github, from your note above.
  • Client secret - like above.
  • Authenticate token requests via HTTP headers - set to true.
  • Service base URL - https://github.com
  • Logo URL - https://github.com/favicon.ico
  • This service will be used - Login page only
  • Name displayed on the login page - anything you want, i.e. github.
  • Scopes included in a login request - user:email
  • Scopes included in a login request for offline access - user:email
  • Additional parameters included in a login request - leave empty.
  • Additional parameters included in a login request for offline access - leave empty.
  • Login domains - leave empty.
  • Require email verification - leave checked (true).

After submit, on the admin/tool/oauth2/issuers.php page click “Configure endpoints” icon. Add following endpoints (Name, URL):

  • authorization_endpoint, https://github.com/login/oauth/authorize
  • token_endpoint, https://github.com/login/oauth/access_token
  • userinfo_endpoint, https://api.github.com/user

Logout (or use another browser i.e. in incognito mode) and try to login to your Moodle site using github account.

If you receive an error:

The user information returned did not contain a username and email address. The OAuth 2 service may be configured incorrectly.

It means that your emails are kept private in github settings (setting “Keep my email addresses private”). The email is not passed from github to Moodle during the OAuth workflow. And since Moodle requires login & email address at the very least when creating the account, it fails. One way to fix it is to set “Keep my email addresses private” to false in github.

I ended up patching core Moodle and adding an extra call to https://api.github.com/user/emails to retrieve an email.

Troubleshooting HTTPS with openssl

  • 2022-05-18

Display certificate information:

$ ➜ openssl s_client -connect muras.eu:443
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = ala.muras.eu
verify return:1
---
Certificate chain
0 s:CN = ala.muras.eu
i:C = US, O = Let's Encrypt, CN = R3
1 s:C = US, O = Let's Encrypt, CN = R3
i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFZjCCBE6gAwIBAgISAytgxCG8Nfa5gbAkMQHXSwOMMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMjA0MDQxODA5MzFaFw0yMjA3MDMxODA5MzBaMBcxFTATBgNVBAMT
DGFsYS5tdXJhcy5ldTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALg4
9WBf1tHJNysqDl6bTKj+8no8+QSV/xqxfpcgr9uIEUTYbJtHHNFHDi1QjaufaDBG
ryZsAUO5VfxHygPH93WQc4qX3ZQoaZ7+xA4QjGwR4zJw3CqdQNXXXfoW456iIHrz
EgzSf6KctnQg8VBGhnTqE0ZZN3QTHtLoRy2J/RcTl0z48SLBS60EpeOmIzjek5X1
mii+ZznEa3R+zat9bXxVxiwhFvxS+bhClEUrFYI5I5zPOs7ByUstc2c6Tws1wW2y
R4CEsuLcwvHSH6W7dN3CPjYZ5TbuYuprGxEgYSDJRN07bipy95R4BrHiKAk6R66a
UHlho4KwK7tnjs9VSdkCAwEAAaOCAo8wggKLMA4GA1UdDwEB/wQEAwIFoDAdBgNV
HSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4E
FgQUObZv+7j4EQHj5orKa2O1i0Yhd7UwHwYDVR0jBBgwFoAUFC6zF7dYVsuuUAlA
5h+vnYsUwsYwVQYIKwYBBQUHAQEESTBHMCEGCCsGAQUFBzABhhVodHRwOi8vcjMu
by5sZW5jci5vcmcwIgYIKwYBBQUHMAKGFmh0dHA6Ly9yMy5pLmxlbmNyLm9yZy8w
YAYDVR0RBFkwV4IMYWxhLm11cmFzLmV1ggxkb2MubXVyYXMuZXWCEG1pa29sYWou
bXVyYXMuZXWCD21vbmljYS5tdXJhcy5ldYIIbXVyYXMuZXWCDHd3dy5tdXJhcy5l
dTBMBgNVHSAERTBDMAgGBmeBDAECATA3BgsrBgEEAYLfEwEBATAoMCYGCCsGAQUF
BwIBFhpodHRwOi8vY3BzLmxldHNlbmNyeXB0Lm9yZzCCAQMGCisGAQQB1nkCBAIE
gfQEgfEA7wB1AEHIyrHfIkZKEMahOglCh15OMYsbA+vrS8do8JBilgb2AAABf/X7
deIAAAQDAEYwRAIgL+/+47ymSnPD786/vSsLAe9DnvdPSDhzB95iDJWRjBECIAYI
AwwP6sQhB852PAq2ImsgJC0UGrmr3BodVWjnRcMFAHYARqVV63X6kSAwtaKJafTz
fREsQXS+/Um4havy/HD+bUcAAAF/9ft2BgAABAMARzBFAiAYmpaYKA4Rklxe7KF2
3faQo5WQzwIQGMG/EBHsj55bWgIhAN/AyVz5PZ5x74R1otpwH+ULFcbyodU2TjrV
tmJMi1QSMA0GCSqGSIb3DQEBCwUAA4IBAQBTRMekA7B8D3EHvHPVFsjCePvWUX1D
sDTX/HJIAZ+L7szjQLZKHvDZRuoCceikZmGV4aFIdyt+jlEQneJVFj5QCEtjjjiI
j1eTEGSnotHXRAQeW1sjtGgSLWXrRJsLJNqzLfXw25/XJgSK/KIwuvh+KI32kaYl
+95nd1FHwZshNgttC8ihTFBQWijJVV6sOeyGE3JZHWBDQfjp7kbUvGxfLIi1ziWM
6ry0+FcICtVMWwLbQi4HMxax2PvTdCCQZCrOaWiM1xQ/p4k1p3iY7fyTdl9Sr6yr
Y+m6RPgVr/JEIKWGQtWCwtqk0TzrOUwIBIw5xU1HyA5hz7vOrzxeROSM
-----END CERTIFICATE-----
subject=CN = ala.muras.eu

issuer=C = US, O = Let's Encrypt, CN = R3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4642 bytes and written 380 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol  : TLSv1.3
Cipher    : TLS_AES_256_GCM_SHA384
Session-ID: 1A3E72C7B0EE867B11C105EFAE3C39CE4FB149B0EBCFD836792AA44161637204
Session-ID-ctx:
Resumption PSK: 5A4A7C751C76D7AED82A657151666E9AA15749C001A641871C2F70295EFCEDC9CFBE6FDDCD9C2EC151086FE17DE8222F
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 - 92 7a dd 24 74 a4 76 ba-76 7a 9f 79 3b 9c 35 bc   .z.$t.v.vz.y;.5.
0010 - ba 5f e8 bb 82 4a 84 47-7f 9d d0 0a f9 fa ab f3   ._...J.G........

    Start Time: 1652895551
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol  : TLSv1.3
Cipher    : TLS_AES_256_GCM_SHA384
Session-ID: B90C92D0D4BF52C38E33C8C28210D43137E0642528C4F7FEEF9F77E06D5484CF
Session-ID-ctx:
Resumption PSK: 27AFDE9BBFFCCC660EC5C73053BCB7837A002ACBD7E7233397A834439B5514CDBDE8B4FDC2002970A61A2764262ABBAA
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 - 59 d0 96 33 41 f2 23 a7-45 87 d3 57 e5 eb 5f ba   Y..3A.#.E..W.._.
0010 - 58 18 2f 31 f6 28 ef 21-5e e6 e7 34 2f f0 43 72   X./1.(.!^..4/.Cr

    Start Time: 1652895551
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK

Send GET request to https://muras.eu and use muras.eu for SNI.

$ echo -e "GET / HTTP/1.1\r\nHost: muras.eu\r\nConnection: close\r\n\r\n" |  openssl s_client -quiet -connect muras.eu:443 -servername muras.eu
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = ala.muras.eu
verify return:1
HTTP/1.1 200 OK
Date: Wed, 18 May 2022 17:29:56 GMT
Server: Apache/2.4.48 (Ubuntu)
Last-Modified: Tue, 08 Feb 2022 17:44:05 GMT
ETag: "11c3c-5d785436c2f8e"
Accept-Ranges: bytes
Content-Length: 72764
Vary: Accept-Encoding
Connection: close
Content-Type: text/html

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>Tomasz Muras</title>
    <meta name="description" content="Technical blog.">
...

Verify certificate chain:

# Store all certificates
$ openssl s_client -connect muras.eu:443  -showcerts > cert.pem
# Extract them into cert1.pem  cert2.pem  cert3.pem
# Verify
$ openssl verify -CAfile cert2.pem cert1.pem
cert1.pem: OK

List of the error codes:

$ man verify