This repository was archived by the owner on Apr 29, 2020. It is now read-only.
Add log lines when daemon set handling goroutine exits#1053
Open
mpuncel wants to merge 2 commits intosquare:masterfrom
Open
Add log lines when daemon set handling goroutine exits#1053mpuncel wants to merge 2 commits intosquare:masterfrom
mpuncel wants to merge 2 commits intosquare:masterfrom
Conversation
Occasionally we see problems in the DS farm where the main control loop gets blocked sending updates to daemon set handlers via channels. This is likely because the handler goroutines are exiting, but logs don't currently reveal the cause. This commit adds logs to all of the places where these goroutines might exit which should shed light on the issue
spenceral
reviewed
May 31, 2018
pkg/ds/daemon_set.go
Outdated
| // so that the timer would be stopped after | ||
| err = nil | ||
| case <-ctx.Done(): | ||
| ds.logger.Warnln("goroutine exiting because ") |
Collaborator
There was a problem hiding this comment.
Can you elaborate on this message
spenceral
approved these changes
May 31, 2018
Occasionally the daemon set farm locks up with the farm goroutine blocking forever attempting to send an update to a daemon set worker goroutine. This can happen due to a race where the worker thread might exit for a number of reasons after the farm goroutine checks the child map to determine a worker already exists but before sending an update. This commit sidesteps the problem by buffering the per-daemon set update channel so that the farm goroutine will never block sending to a worker. If a worker dies, an existing routine grabs a mutex protecting the child map and clears out the child entry and drains the buffered channel. The next time an update is seen for the daemon set, the farm loop should know that it needs to spawn another worker.
mpuncel
force-pushed
the
mpuncel/log-ds-returns
branch
from
0918e7e to
782d378
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Occasionally we see problems in the DS farm where the main control loop
gets blocked sending updates to daemon set handlers via channels. This
is likely because the handler goroutines are exiting, but logs don't
currently reveal the cause. This commit adds logs to all of the places
where these goroutines might exit which should shed light on the issue