[RTFACT-18301] Login backoff will grow indefinitely Created: 17/Jan/19  Updated: 05/Sep/19  Resolved: 21/Jan/19

Status: Resolved
Project: Artifactory Binary Repository
Component/s: None
Affects Version/s: 6.6.5
Fix Version/s: 6.7.0

Type: Bug Priority: Normal
Reporter: Rafael Cunha de Almeida Assignee: Inbar Tal
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Dependency
Regression:
Yes
Sprint: Leap 37

 Description   

The implementation login backoff implementation has a memory leak. From storage/db/src/main/java/org/artifactory/storage/db/security/service/UserLockInMemoryServiceImpl.java:

BigInteger timeToBackoff = SEQUENCE_BASE.pow(attempts).multiply(baseDelay); 

That value will grow indefinitely (well at least up to SEQUENCE_BASE^(2^32)*baseDelay), while the user attempts to login. We have had that happening due to automated scripts. Besides making the logs less readable I'm afraid this could potentially DoS the server, if many users keep on retrying to connect. This is how the log looks like

2019-01-16 16:43:32,605 [http-nio-8081-exec-5] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:89) - loginBlockDelay '8000' has exceeded maximum allowed delay '5000', which will be used instead
2019-01-16 16:43:32,975 [http-nio-8081-exec-10] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:89) - loginBlockDelay '16000' has exceeded maximum allowed delay '5000', which will be used instead
2019-01-16 16:43:33,278 [http-nio-8081-exec-7] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:89) - loginBlockDelay '32000' has exceeded maximum allowed delay '5000', which will be used instead
2019-01-16 16:43:33,699 [http-nio-8081-exec-4] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:89) - loginBlockDelay '64000' has exceeded maximum allowed delay '5000', which will be used instead
2019-01-16 16:43:34,024 [http-nio-8081-exec-9] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:89) - loginBlockDelay '128000' has exceeded maximum allowed delay '5000', which will be used instead
2019-01-16 16:43:34,367 [http-nio-8081-exec-2] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:89) - loginBlockDelay '256000' has exceeded maximum allowed delay '5000', which will be used instead
2019-01-16 16:43:34,761 [http-nio-8081-exec-6] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:89) - loginBlockDelay '512000' has exceeded maximum allowed delay '5000', which will be used instead
2019-01-16 16:43:35,062 [http-nio-8081-exec-1] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:89) - loginBlockDelay '1024000' has exceeded maximum allowed delay '5000', which will be used instead
2019-01-16 16:43:35,365 [http-nio-8081-exec-5] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:89) - loginBlockDelay '2048000' has exceeded maximum allowed delay '5000', which will be used instead
2019-01-16 16:43:35,715 [http-nio-8081-exec-10] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:89) - loginBlockDelay '4096000' has exceeded maximum allowed delay '5000', which will be used instead
2019-01-16 16:43:36,133 [http-nio-8081-exec-7] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:89) - loginBlockDelay '8192000' has exceeded maximum allowed delay '5000', which will be used instead
2019-01-16 16:43:36,474 [http-nio-8081-exec-4] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:89) - loginBlockDelay '16384000' has exceeded maximum allowed delay '5000', which will be used instead
2019-01-16 16:43:36,828 [http-nio-8081-exec-8] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:89) - loginBlockDelay '32768000' has exceeded maximum allowed delay '5000', which will be used instead
2019-01-16 16:43:37,176 [http-nio-8081-exec-9] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:89) - loginBlockDelay '65536000' has exceeded maximum allowed delay '5000', which will be used instead
2019-01-16 16:43:37,511 [http-nio-8081-exec-2] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:89) - loginBlockDelay '131072000' has exceeded maximum allowed delay '5000', which will be used instead
2019-01-16 16:43:37,857 [http-nio-8081-exec-6] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:89) - loginBlockDelay '262144000' has exceeded maximum allowed delay '5000', which will be used instead
2019-01-16 16:43:38,170 [http-nio-8081-exec-5] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:89) - loginBlockDelay '524288000' has exceeded maximum allowed delay '5000', which will be used instead
2019-01-16 16:43:38,495 [http-nio-8081-exec-10] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:89) - loginBlockDelay '1048576000' has exceeded maximum allowed delay '5000', which will be used instead
2019-01-16 16:43:38,835 [http-nio-8081-exec-7] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:89) - loginBlockDelay '2097152000' has exceeded maximum allowed delay '5000', which will be used instead
2019-01-16 16:43:39,179 [http-nio-8081-exec-3] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:89) - loginBlockDelay '4194304000' has exceeded maximum allowed delay '5000', which will be used instead
2019-01-16 16:43:39,479 [http-nio-8081-exec-4] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:89) - loginBlockDelay '8388608000' has exceeded maximum allowed delay '5000', which will be used instead
2019-01-16 16:43:39,787 [http-nio-8081-exec-8] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:89) - loginBlockDelay '16777216000' has exceeded maximum allowed delay '5000', which will be used instead
2019-01-16 16:43:40,114 [http-nio-8081-exec-9] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:89) - loginBlockDelay '33554432000' has exceeded maximum allowed delay '5000', which will be used instead
2019-01-16 16:43:40,453 [http-nio-8081-exec-2] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:89) - loginBlockDelay '67108864000' has exceeded maximum allowed delay '5000', which will be used instead
2019-01-16 16:43:40,799 [http-nio-8081-exec-6] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:89) - loginBlockDelay '134217728000' has exceeded maximum allowed delay '5000', which will be used instead
2019-01-16 16:43:41,159 [http-nio-8081-exec-1] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:89) - loginBlockDelay '268435456000' has exceeded maximum allowed delay '5000', which will be used instead
2019-01-16 16:43:41,545 [http-nio-8081-exec-5] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:89) - loginBlockDelay '536870912000' has exceeded maximum allowed delay '5000', which will be used instead
2019-01-16 16:43:41,974 [http-nio-8081-exec-10] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:89) - loginBlockDelay '1073741824000' has exceeded maximum allowed delay '5000', which will be used instead
...
2019-01-16 16:45:23,912 [http-nio-8081-exec-10] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:89) - loginBlockDelay '1125899906842624000' has exceeded maximum allowed delay '5000', which will be used instead
2019-01-16 16:45:23,927 [http-nio-8081-exec-6] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:89) - loginBlockDelay '1125899906842624000' has exceeded maximum allowed delay '5000', which will be used instead
2019-01-16 16:45:23,964 [http-nio-8081-exec-2] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:89) - loginBlockDelay '2251799813685248000' has exceeded maximum allowed delay '5000', which will be used instead
2019-01-16 16:45:24,165 [http-nio-8081-exec-8] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:89) - loginBlockDelay '9007199254740992000' has exceeded maximum allowed delay '5000', which will be used instead
2019-01-16 16:45:24,735 [http-nio-8081-exec-4] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:83) - Backoff time '18014398509481984000' exceeds long max value
2019-01-16 16:45:24,754 [http-nio-8081-exec-5] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:83) - Backoff time '36028797018963968000' exceeds long max value
2019-01-16 16:45:27,133 [http-nio-8081-exec-7] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:83) - Backoff time '72057594037927936000' exceeds long max value
2019-01-16 16:45:27,150 [http-nio-8081-exec-3] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:83) - Backoff time '144115188075855872000' exceeds long max value
2019-01-16 16:45:27,158 [http-nio-8081-exec-6] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:83) - Backoff time '288230376151711744000' exceeds long max value
2019-01-16 16:47:44,998 [http-nio-8081-exec-4] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:83) - Backoff time '576460752303423488000' exceeds long max value
2019-01-16 16:47:45,035 [http-nio-8081-exec-3] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:83) - Backoff time '576460752303423488000' exceeds long max value
2019-01-16 16:47:45,120 [http-nio-8081-exec-5] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:83) - Backoff time '1152921504606846976000' exceeds long max value
2019-01-16 16:47:45,148 [http-nio-8081-exec-7] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:83) - Backoff time '2305843009213693952000' exceeds long max value
2019-01-16 16:47:45,171 [http-nio-8081-exec-10] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:83) - Backoff time '9223372036854775808000' exceeds long max value
2019-01-16 16:47:45,188 [http-nio-8081-exec-1] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:83) - Backoff time '18446744073709551616000' exceeds long max value
2019-01-16 16:47:45,200 [http-nio-8081-exec-8] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:83) - Backoff time '36893488147419103232000' exceeds long max value
2019-01-16 16:47:45,200 [http-nio-8081-exec-6] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:83) - Backoff time '36893488147419103232000' exceeds long max value
2019-01-16 16:47:45,238 [http-nio-8081-exec-9] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:83) - Backoff time '147573952589676412928000' exceeds long max value
2019-01-16 16:47:45,238 [http-nio-8081-exec-2] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:83) - Backoff time '147573952589676412928000' exceeds long max value
2019-01-16 16:47:45,241 [http-nio-8081-exec-3] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:83) - Backoff time '590295810358705651712000' exceeds long max value
2019-01-16 16:47:45,251 [http-nio-8081-exec-5] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:83) - Backoff time '1180591620717411303424000' exceeds long max value
2019-01-16 16:47:45,254 [http-nio-8081-exec-7] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:83) - Backoff time '2361183241434822606848000' exceeds long max value
2019-01-16 16:47:45,264 [http-nio-8081-exec-10] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:83) - Backoff time '4722366482869645213696000' exceeds long max value
2019-01-16 16:47:45,273 [http-nio-8081-exec-1] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:83) - Backoff time '9444732965739290427392000' exceeds long max value
2019-01-16 16:47:45,273 [http-nio-8081-exec-4] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:83) - Backoff time '9444732965739290427392000' exceeds long max value
2019-01-16 16:47:45,287 [http-nio-8081-exec-8] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:83) - Backoff time '37778931862957161709568000' exceeds long max value
2019-01-16 16:47:45,286 [http-nio-8081-exec-6] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:83) - Backoff time '37778931862957161709568000' exceeds long max value
2019-01-16 16:47:45,304 [http-nio-8081-exec-2] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:83) - Backoff time '151115727451828646838272000' exceeds long max value
2019-01-16 16:47:45,325 [http-nio-8081-exec-9] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:83) - Backoff time '302231454903657293676544000' exceeds long max value
2019-01-16 16:47:45,338 [http-nio-8081-exec-3] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:83) - Backoff time '604462909807314587353088000' exceeds long max value
2019-01-16 16:47:45,391 [http-nio-8081-exec-7] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:83) - Backoff time '1208925819614629174706176000' exceeds long max value
2019-01-16 16:47:45,391 [http-nio-8081-exec-5] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:83) - Backoff time '1208925819614629174706176000' exceeds long max value
2019-01-16 16:47:45,431 [http-nio-8081-exec-10] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:83) - Backoff time '4835703278458516698824704000' exceeds long max value
2019-01-16 16:47:45,464 [http-nio-8081-exec-1] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:83) - Backoff time '9671406556917033397649408000' exceeds long max value

This is how far we've got in our dev instance

2019-01-17 14:18:46,478 [http-nio-8081-exec-21] [WARN ] (o.a.s.d.s.s.UserLockInMemoryServiceImpl:83) - Backoff time '518201151027899309442573310534104524021200103005238567673797203733326845384974181254167158928854938043700859223074
027486691881971398642041333784201068965746370643716713444172629581988707626918873074400342943918673495031214402457657880301635084636262474243610064373940161238514022091179366303866213263304402757313789481550583243746625771946300800
075125678892587083176784689480605945732286512921687439239252368115477701950864296778156943810253518810207068918783040427133356001024308923904073739706489256592310865244695437165103881836634378325627225122117134090326702282170342615
056771496259563813583092222655602541994597505701029186016274641881546793489862208348356192854994150159017173834173070573262512370905638831815039120233126940939551890284636050835769763033352462702159565272037783840426590006067290158
248664332243531220874995848462884830688150442694532198700990197003990738151780207989116323968976199212464139335180914237492405178563930399607934750051565806512985070560194136099343423432721314663721703242728146264932452501585576445
014916807005810666330918838908178323480867113516600043237723985592319675408595121078341302477587911254853470818619473522151550061751660234495942761951949113641115560506293293223718718102532364108058643914497768083806235004791745168
783753568902951012151037859709639428055214632800011452369270823899752197042573844623409226243878495800186345252366802408481205772414290552118401110826430617282566942336605223714028407727164649765237638531976663793436526318317881953
280591542362875456418907500707840612011949272633773730465113671726108399268004843761063447477300632552947253323997869545298047192804120172196054482729789444626330840168615970856811055651210073194202257153512317124780000526713777008
316214858225216711044093543023322319568807030575463585495721829840063211660800107700764438443247763714612947998068983486405246768610191463985112223964442626937750645254703117965802162293204155488413850845706254656453594594264563456
651202656016018626336742780579682900638977327026403459492765197561789122919653489432366368580091717988663813311462584920752322827080411927303513677815714383380753107219896090833385266882900435269760247512959422871001533922840339443
020062709669399852381768529496466814966384872381289183166878694251136059200796233261561049952039204920189534243142786877090640365596271313086318289998680839038637230586541063396102366960884274457334028401120802412041539023735461501
304205023639739456165615379416568813191234557727262401387785710419659957008215288945917300718298271799696474957557519935107571683995032305385050106107026538077562502303290063799654256567889581393588144045642127246073058062193775178
044381255208697261657930353460076462528804062365076967620008706658973754650281480172102520137776816118059887601434165802093806549460458843531003637949439640947056190274179651917749530828404638318926765525169130041373234123728764315
460340288298517839059857357172982797178010675173941318402248999468776347916398201115958952163406933465900403978951753469525197010546962862612969279416614218017490430894106638339991759763323311237734316846123528053780110824553590968
572704186752672026985126480909951003523966782973641699711593475419437362842138447710902389259328410148176788258287520922784578095116690707978358868438579113818943982784266500480278148737309879461726378563232318538265121343297736202
044932924259868804534701795089934849157947471580270502482655334481845174719058911689892961215790981023636579333900678709946076252024282431828036713610815372960254367598600326766517818837104413030612429236596318100337594358953612635
006591422140811973979073157930891887779248364297019659860953519554221349577189945607132644242654976231006428507782158817925896050874203233745482172164476725327720083014105946539991209404310087900175037695438530954367358023888589716
016691484773432135768445645693464875642378859447840012844269731229188749213655900971980243297815229528660345169053174223411094639552708246139947976711429228292440714189644105807937138451166655180826415209172499564989531770646300773
381151631911892950575882916210821252037787704113928355074904442748124878996171414047822340255017844993933653640843745129870655444651624216839197662035909227974755348204391780707391208544409512783590841492398859642890293224081920228
039679648371398033748714217782159433112130186199831444300423546948390158638526587413201025320506714333257087765557011664713795797016909041478225165719599262071787890274473163162717594988897990523242928241890900750977253766618174878
649266572677535121410709352604766802173156206217694900659570368316981942677779403067390904442639930464203461434203265508150188596111688457585706242375610487880294966800973436528529124082868697963860118367673742992921348930326638670
39199409878826329648817547014105244629005259584842058768711515431741141552035478468692883832052380880496266523831051400938385841733608059718008141287050110833770884295521585633856290042514033490958903943678561851583689927989879347282444288000' exceeds long max value

Is there a workaround this issue? I'm happy to disable the feature.



 Comments   
Comment by Trent Dailey [ 21/Jan/19 ]

Can these errors cause any issues, like OOM or DoS as the submitter stated?  Is there a workaround that can be put into place until 6.7.0 is released?

Inbar Tal

Comment by Michael Huebschmann [ 22/Feb/19 ]

we got this workaround from jfrog support:

Thank you for your reply and reporting this.
It seems like mistakenly a logging message had been set in the code to warning instead of debug, we will investigate this and will release a fix this in one of the next releases ( probably the next release).
As for now the workaround is to add the below to the _$ARTIFACTORY_HOME/etc/logback.xml_file after the last logger( to EACH node in case of HA):

    <logger name="org.artifactory.storage.db.security.service.UserLockInMemoryServiceImpl">
        <level value="error"/>
    </logger>
    
This will prevent printing warning messages to the logs for this specific class.

Comment by Gili Tzabari [ 14/Mar/19 ]

Inbar Tal What's the impact of this issue? We're using the SAAS deployment so we can't use the workaround. Thank you.

Generated at Tue Dec 10 03:31:16 UTC 2019 using JIRA 7.6.16#76018-sha1:9ed376192612a49536ac834c64177a0fed6290f5.