mirror/gitea
1
0
Fork 0
mirror of https://github.com/go-gitea/gitea.git synced 2024-05-11 05:55:29 +00:00
Code Issues Projects Releases Wiki Activity
10,533 Commits 17 Branches 210 Tags
5bd05331cef2beadc2ff87402db659ef532cb2e4
Commit Graph
21 Commits
Author SHA1 Message Date
zeripath c8f7a6b774 Slightly simplify the queue settings code to help reduce the risk of problems (#12976) 
Signed-off-by: Andrew Thornton <[email protected]>

Co-authored-by: techknowlogick <[email protected]>
 2020-10-15 17:40:03 -04:00
zeripath 5cfc1f573f Fix the issue reported on #12385 (#12969) 
Missed setting ConnectionString on queuesettings

Signed-off-by: Andrew Thornton <[email protected]>
 2020-09-28 19:00:54 -04:00
zeripath 7f8e3192cd Allow common redis and leveldb connections (#12385) 
* Allow common redis and leveldb connections

Prevents multiple reopening of redis and leveldb connections to the same
place by sharing connections.

Further allows for more configurable redis connection type using the
redisURI and a leveldbURI scheme.

Signed-off-by: Andrew Thornton <[email protected]>

* add unit-test

Signed-off-by: Andrew Thornton <[email protected]>

* as per @lunny

Signed-off-by: Andrew Thornton <[email protected]>

* add test

Signed-off-by: Andrew Thornton <[email protected]>

* Update modules/cache/cache_redis.go

* Update modules/queue/queue_disk.go

* Update modules/cache/cache_redis.go

* Update modules/cache/cache_redis.go

* Update modules/queue/unique_queue_disk.go

* Update modules/queue/queue_disk.go

* Update modules/queue/unique_queue_disk.go

* Update modules/session/redis.go

Co-authored-by: techknowlogick <[email protected]>
Co-authored-by: Lauris BH <[email protected]>
 2020-09-28 00:09:46 +03:00
Lunny Xiao 91e7ad569a Add queue for code indexer (#10332) 
* Add queue for code indexer

* Fix lint

* Fix test

* Fix lint

* Fix bug

* Fix bug

* Fix lint

* Add noqueue

* Fix tests

* Rename noqueue to immediate
 2020-09-07 23:05:08 +08:00
zeripath 69b8d7ba19 use assignment in tests (#12734) 
Signed-off-by: Andrew Thornton <[email protected]>
 2020-09-06 01:50:57 +03:00
zeripath 74bd9691c6 Re-attempt to delete temporary upload if the file is locked by another process (#12447) 
Replace all calls to os.Remove/os.RemoveAll by retrying util.Remove/util.RemoveAll and remove circular dependencies from util.

Fix #12339

Signed-off-by: Andrew Thornton <[email protected]>
Co-authored-by: silverwind <[email protected]>
 2020-08-11 21:05:34 +01:00
zeripath 217647f331 Multiple small admin dashboard fixes (#12153) 
* Remove spurious spacing between Maintenance Operations and its table on dashboard
* Prevent (EXTRA string) comments in Task headers
* Redirect tasks started from monitor page back to monitor
* Fix #12107 - redirects from process cancel should use AppSubUrl
* When wrapping queues set the name correctly

Signed-off-by: Andrew Thornton <[email protected]>
 2020-07-05 22:38:03 +03:00
zeripath c58bc4bf80 Prevent timer leaks in Workerpool and others (#11333) 
There is a potential memory leak in `Workerpool` due to the intricacies of
`time.Timer` stopping.

Whenever a `time.Timer` is `Stop`ped its channel must be cleared using a
`select` if the result of the `Stop()` is `false`.

Unfortunately in `Workerpool` these were checked the wrong way round.

However, there were a few other places that were not being checked.

Signed-off-by: Andrew Thornton <[email protected]>

Co-authored-by: techknowlogick <[email protected]>
Co-authored-by: Lunny Xiao <[email protected]>
 2020-05-08 16:46:05 +01:00
Lunny Xiao fcc8cdd446 Improve config logging when WrappedQueue times out (#11174) 
Before
```sh
Unable to set the internal queue for -wrapper Error: Timedout creating queue redis with cfg []byte{0x7b, 0x22, 0x41, 0x64, 0x64, 0x72, 0x65, 0x73, 0x73, 0x65, 0x73, 0x22, 0x3a, 0x22, 0x31, 0x32, 0x37, 0x2e, 0x30, 0x2e, 0x30, 0x2e, 0x31, 0x3a, 0x36, 0x33, 0x37, 0x39, 0x22, 0x2c, 0x22, 0x42, 0x61, 0x74, 0x63, 0x68, 0x4c, 0x65, 0x6e, 0x67, 0x74, 0x68, 0x22, 0x3a, 0x32, 0x30, 0x2c, 0x22, 0x42, 0x6c, 0x6f, 0x63, 0x6b, 0x54, 0x69, 0x6d, 0x65, 0x6f, 0x75, 0x74, 0x22, 0x3a, 0x31, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x2c, 0x22, 0x42, 0x6f, 0x6f, 0x73, 0x74, 0x54, 0x69, 0x6d, 0x65, 0x6f, 0x75, 0x74, 0x22, 0x3a, 0x33, 0x30, 0x30, 0x30, 0x30, 0x30
......
```

After
```sh
Unable to set the internal queue for -wrapper Error: Timedout creating queue redis with cfg "{\"Addresses\":\"127.0.0.1:6379\",\"BatchLength\":20,\"BlockTimeout\":1000000000,\"BoostTimeout\":300000000000,\"BoostWorkers\":5,\"DBIndex\":0,\"DataDir\":\".../data/queues/mail\",\"MaxWorkers\":10,\"Name\":\"mail\",\"Network\":\"\",\"Password\":\"\",\"QueueLength\":20,\"QueueName\":\"mail_queue\",\"SetName\":\"\",\"Workers\":1}" in
```
 2020-04-22 13:38:40 +01:00
zeripath e83daf77ad Avoid logging []byte in queue failures - convert to string first (#10865) 
Signed-off-by: Andrew Thornton <[email protected]>

Co-authored-by: guillep2k <[email protected]>
 2020-03-29 15:12:15 +08:00
Lunny Xiao cf7ece6245 Fix queue log param (#10733) 2020-03-16 16:59:21 +08:00
zeripath 88986746d5 Fix Workerpool deadlock (#10283) 
* Prevent deadlock on boost

* Force a boost in testchannelqueue
 2020-02-15 18:44:58 +00:00
Lunny Xiao 3d69bbd58f Fix queue pop error and stat empty repository error (#10248) 
* Fix queue pop error and stat empty repository error

* Fix error
 2020-02-12 18:12:27 +08:00
zeripath 2c903383b5 Add Unique Queue infrastructure and move TestPullRequests to this (#9856) 
* Upgrade levelqueue to version 0.2.0

This adds functionality for Unique Queues

* Add UniqueQueue interface and functions to create them

* Add UniqueQueue implementations

* Move TestPullRequests over to use UniqueQueue

* Reduce code duplication

* Add bytefifos

* Ensure invalid types are logged

* Fix close race in PersistableChannelQueue Shutdown
 2020-02-02 23:19:58 +00:00
zeripath 9b9dd19d7d Fix broken FlushAll (#10101) 
* go function contexting is not what you expect

* Apply suggestions from code review

Co-Authored-By: Lauris BH <[email protected]>

Co-authored-by: Lauris BH <[email protected]>
 2020-02-01 23:43:50 +00:00
Lunny Xiao eac5142ac7 Fix leveldb test race (#10054) 
Co-authored-by: Lauris BH <[email protected]>
Co-authored-by: techknowlogick <[email protected]>
 2020-01-30 11:09:39 -05:00
zeripath c01221e70f Queue: Make WorkerPools and Queues flushable (#10001) 
* Make WorkerPools and Queues flushable

Adds Flush methods to Queues and the WorkerPool
Further abstracts the WorkerPool
Adds a final step to Flush the queues in the defer from PrintCurrentTest
Fixes an issue with Settings inheritance in queues

Signed-off-by: Andrew Thornton <[email protected]>

* Change to for loop

* Add IsEmpty and begin just making the queues composed WorkerPools

* subsume workerpool into the queues and create a flushable interface

* Add manager command

* Move flushall to queue.Manager and add to testlogger

* As per @guillep2k

* as per @guillep2k

* Just make queues all implement flushable and clean up the wrapped queue flushes

* cope with no timeout

Co-authored-by: Lauris BH <[email protected]>
 2020-01-28 20:01:06 -05:00
zeripath 8cb9b2f016 Update queue_redis.go (#9910) 
Co-authored-by: Lunny Xiao <[email protected]>
 2020-01-21 12:50:27 +00:00
zeripath 960ac36099 Remove unused lock (#9709) 2020-01-11 19:06:35 +00:00
zeripath edd31770aa Missed q.lock.Unlock() will cause panic (#9705) 2020-01-11 17:05:07 +00:00
zeripath 62eb1b0f25 Graceful Queues: Issue Indexing and Tasks (#9363) 
* Queue: Add generic graceful queues with settings

* Queue & Setting: Add worker pool implementation

* Queue: Add worker settings

* Queue: Make resizing worker pools

* Queue: Add name variable to queues

* Queue: Add monitoring

* Queue: Improve logging

* Issues: Gracefulise the issues indexer

Remove the old now unused specific queues

* Task: Move to generic queue and gracefulise

* Issues: Standardise the issues indexer queue settings

* Fix test

* Queue: Allow Redis to connect to unix

* Prevent deadlock during early shutdown of issue indexer

* Add MaxWorker settings to queues

* Merge branch 'master' into graceful-queues

* Update modules/indexer/issues/indexer.go

Co-Authored-By: guillep2k <[email protected]>

* Update modules/indexer/issues/indexer.go

Co-Authored-By: guillep2k <[email protected]>

* Update modules/queue/queue_channel.go

Co-Authored-By: guillep2k <[email protected]>

* Update modules/queue/queue_disk.go

* Update modules/queue/queue_disk_channel.go

Co-Authored-By: guillep2k <[email protected]>

* Rename queue.Description to queue.ManagedQueue as per @guillep2k

* Cancel pool workers when removed

* Remove dependency on queue from setting

* Update modules/queue/queue_redis.go

Co-Authored-By: guillep2k <[email protected]>

* As per @guillep2k add mutex locks on shutdown/terminate

* move unlocking out of setInternal

* Add warning if number of workers < 0

* Small changes as per @guillep2k

* No redis host specified not found

* Clean up documentation for queues

* Update docs/content/doc/advanced/config-cheat-sheet.en-us.md

* Update modules/indexer/issues/indexer_test.go

* Ensure that persistable channel queue is added to manager

* Rename QUEUE_NAME REDIS_QUEUE_NAME

* Revert "Rename QUEUE_NAME REDIS_QUEUE_NAME"

This reverts commit 1f83b4fc9b.

Co-authored-by: guillep2k <[email protected]>
Co-authored-by: Lauris BH <[email protected]>
Co-authored-by: techknowlogick <[email protected]>
Co-authored-by: Lunny Xiao <[email protected]>
 2020-01-07 12:23:09 +01:00