Uploaded image for project: 'Artifactory Binary Repository'
  1. Artifactory Binary Repository
  2. RTFACT-19088

Large scale users <> groups <> permissions ACL matrix can increase API and build requests response time

    XMLWordPrintable

    Details

      Description

      Builds will take longer time to deploy since a single thread building one of the permissions cache in Artifactory will cause other threads to wait for the first thread to finish.

      Steps to reproduce:

      1. Build an Artifactory env with:
      At least 25,000 Artifactory users
      At least 125,000 Artifactory groups
      At least 30,000 permission targets

      2. Use the below script to deploy builds:

      #!/bin/bash
      BUILDS_NUM=$1
      username=$2
      password=$3
      a=1
      mkdir builds
      while [ $a -le $BUILDS_NUM ] ; do
      echo '{
      "version": "1.0.1",
      "name": "'a$a'",
      "number": "1",
      "type": "GENERIC",
      "buildAgent": {
      "name": "Generic",
      "version": "Generic"
      },
      "agent": {
      "name": "hudson",
      "version": "2.7.4"
      },
      "started": "2016-09-23T08:06:37.262+0300",
      "durationMillis": 2176,
      "principal": "admin",
      "artifactoryPrincipal": "admin",
      "artifactoryPluginVersion": "2.7.2",
      "url": "http://192.168.59.182:8080/job/hgj/4/",
      "licenseControl": {
      "runChecks": false,
      "includePublishedArtifacts": false,
      "autoDiscover": false,
      "scopesList": "",
      "licenseViolationsRecipientsList": ""
      },
      "buildRetention": {
      "count": -1,
      "deleteBuildArtifacts": true,
      "buildNumbersNotToBeDiscarded": []
      },
      "modules": [{
      "id": "hgj:4",
      "artifacts": [{
      "type": "gz",
      "sha1": "ab078447ae3d3479b6c1a17ad30c9978dd371995",
      "md5": "578a3b9c44edcadc7e9e32a576910d6f",
      "name": "git-lfs-linux-amd64-1.2.0.tar.gz"
      }],
      "dependencies": []
      }],
      "buildDependencies": [],
      "governance": {
      "blackDuckProperties": {
      "runChecks": false,
      "includePublishedArtifacts": false,
      "autoCreateMissingComponentRequests": false,
      "autoDiscardStaleComponentRequests": false
      }
      }
      }' > builds/buildinfo$a.json
      time curl -i -Lvv -H 'Content-Type: application/json' -u $username:$password -X PUT -T builds/buildinfo$a.json localhost:8081/artifactory/api/build &
      	sleep 1
              a=`expr $a + 1`
      done
      

      Use this command to run it:

      bash -x deployBuilds.sh 100 {someUserWithManyPermissions} password
      
        • Do not use an admin user
      • u can increment the unique builds by adding up the a variable in the script between the runs

      3. While the above runs, issue a permissions change command:

      time curl -i -X PUT -u admin:password "localhost:8081/artifactory/api/security/permissions/cachePermissions" -H "Content-Type: application/json" --data '{"repositories":["generic-local"],"principals":{"groups":{"readers":["r","m","n"]}}}'
      

      4. See how that slows down the builds - you can take a thread dump and see all other user threads are waiting for. Example thread:

      "http-nio-8081-exec-376" #3811 daemon prio=5 os_prio=0 tid=0x00007fac4c15f800 nid=0x6905 runnable [0x00007faaf8094000]
         java.lang.Thread.State: TIMED_WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00007facfea141f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
      	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:934)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1247)
      	at java.util.concurrent.locks.ReentrantLock.tryLock(ReentrantLock.java:442)
      	at org.artifactory.storage.db.security.service.VersioningCache.tryToWaitForLock(VersioningCache.java:116)
      	at org.artifactory.storage.db.security.service.VersioningCache.get(VersioningCache.java:66)
      	at org.artifactory.repo.service.RepositoryServiceImpl.localRepositoryByKey(RepositoryServiceImpl.java:1492)
      	at org.artifactory.repo.service.RepositoryServiceImpl.localOrCachedRepositoryByKey(RepositoryServiceImpl.java:1525)
      	at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source)
      .....
      

      All of the threads will also compete for the lock acquirement (ReentrantLock$NonfairSync)

      The below trend will be witnessed on the request.log:

      20190418171630|3934|REQUEST|0:0:0:0:0:0:0:1|userWithManyPermissions|PUT|/api/build|HTTP/1.1|204|1090
      20190418171631|4025|REQUEST|0:0:0:0:0:0:0:1|userWithManyPermissions|PUT|/api/build|HTTP/1.1|204|1090
      20190418171632|5716|REQUEST|0:0:0:0:0:0:0:1|userWithManyPermissions|PUT|/api/build|HTTP/1.1|204|1090
      20190418171635|9787|REQUEST|0:0:0:0:0:0:0:1|userWithManyPermissions|PUT|/api/build|HTTP/1.1|204|1090
      20190418171711|9376|REQUEST|0:0:0:0:0:0:0:1|userWithManyPermissions|PUT|/api/build|HTTP/1.1|204|1090
      20190418171714|11203|REQUEST|0:0:0:0:0:0:0:1|userWithManyPermissions|PUT|/api/build|HTTP/1.1|204|1090
      20190418171716|11554|REQUEST|0:0:0:0:0:0:0:1|userWithManyPermissions|PUT|/api/build|HTTP/1.1|204|1090
      20190418171716|13992|REQUEST|0:0:0:0:0:0:0:1|admin|PUT|/api/security/permissions/permissionCacheRebuildPermission|HTTP/1.1|201|71
      20190418171721|14674|REQUEST|0:0:0:0:0:0:0:1|userWithManyPermissions|PUT|/api/build|HTTP/1.1|204|1090
      20190418171723|15472|REQUEST|0:0:0:0:0:0:0:1|userWithManyPermissions|PUT|/api/build|HTTP/1.1|204|1090
      20190418171724|15323|REQUEST|0:0:0:0:0:0:0:1|userWithManyPermissions|PUT|/api/build|HTTP/1.1|204|1090
      20190418171725|16142|REQUEST|0:0:0:0:0:0:0:1|userWithManyPermissions|PUT|/api/build|HTTP/1.1|204|1090
      20190418171726|16069|REQUEST|0:0:0:0:0:0:0:1|userWithManyPermissions|PUT|/api/build|HTTP/1.1|204|1090
      20190418171727|21335|REQUEST|0:0:0:0:0:0:0:1|userWithManyPermissions|PUT|/api/build|HTTP/1.1|204|1090
      20190418171727|15723|REQUEST|0:0:0:0:0:0:0:1|userWithManyPermissions|PUT|/api/build|HTTP/1.1|204|1090

      5. Taking thread dumps threads calculating permissions prior to the action (in the case above build deployments) will take longer time due to the amount of permissions

      "http-nio-8081-exec-99" #151 daemon prio=5 os_prio=0 tid=0x00007fd078d23800 nid=0x7ffd runnable [0x00007fbe2cfc9000]
         java.lang.Thread.State: RUNNABLE
      	at org.artifactory.security.SecurityServiceImpl.isGranted(SecurityServiceImpl.java:2505)
      	at org.artifactory.security.SecurityServiceImpl.permissionCheckOnAcl(SecurityServiceImpl.java:2608)
      	at org.artifactory.security.SecurityServiceImpl.getAndCheckAcl(SecurityServiceImpl.java:2579)
      	at org.artifactory.security.SecurityServiceImpl.getAndCheckAllUserAcls(SecurityServiceImpl.java:2538)
      	at org.artifactory.security.SecurityServiceImpl.isGranted(SecurityServiceImpl.java:2521)
      	at org.artifactory.security.SecurityServiceImpl.hasPermission(SecurityServiceImpl.java:2348)
      	at org.artifactory.security.SecurityServiceImpl.canAnnotate(SecurityServiceImpl.java:1856)
      	at sun.reflect.GeneratedMethodAccessor632.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
      	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:201)
      	at com.sun.proxy.$Proxy143.canAnnotate(Unknown Source)
      	at org.artifactory.util.UploadServiceUtils.populateItemPropertiesFromRequest(UploadServiceUtils.java:169)
      	at org.artifactory.engine.UploadServiceImpl.uploadItemWithContent(UploadServiceImpl.java:539)
      	at org.artifactory.engine.UploadServiceImpl.uploadItemWithProvidedContent(UploadServiceImpl.java:527)
      	at org.artifactory.engine.UploadServiceImpl.uploadItem(UploadServiceImpl.java:416)
      	at org.artifactory.engine.UploadServiceImpl.uploadFile(UploadServiceImpl.java:406)
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              shayb Shay Bagants
              Reporter:
              andreik Andrei Komarov
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Sync Status

                  Connection: RTFACT Sync
                  RTMID-19088 -
                  SYNCHRONIZED
                  • Last Sync Date: