Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performant Chunk Processing in Map URL Acquisition #71

Open
Kaszanas opened this issue Nov 15, 2024 · 0 comments
Open

Performant Chunk Processing in Map URL Acquisition #71

Kaszanas opened this issue Nov 15, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@Kaszanas
Copy link
Owner

Kaszanas commented Nov 15, 2024

Is your proposal related to a problem?

As of now it seems that the pre-processing steps related to the acquisition of the map URLs from the replay files is not performant.

Describe the solution you'd like

I guess that the mutexed sync.Map use within the function below is causing this issue.
It is adding each of the processed files right after it is read.
Instead of that I'd propose to add the entries to the mutexed maps per chunk or all of the maps from each of the chunks should be returned to the main thread and synchronized there.

...

for _, replayFullFilepath := range channelContents.ChunkOfFiles {

	// Do not use the sync map here per file:
	processFileExtractMap(
		progressBar,
		replayFullFilepath,
		urls,
		processedReplaysSyncMap,
	)

}

...

func createMapExtractingGoroutines(
	channel chan ReplayMapProcessingChannelContents,
	progressBar *progressbar.ProgressBar,
	urls *sync.Map,
	processedReplaysSyncMap *sync.Map,
	wg *sync.WaitGroup,
)

...

urls.Store(mapURL, mapHashAndExtension)
processedReplaysSyncMap.Store(
replayFilename,
persistent_data.FileInformationToCheck{
	LastModified: fileInfo.ModTime().Unix(),
	Size:         fileInfo.Size(),
},
)

Additional context

image

@Kaszanas Kaszanas added the enhancement New feature or request label Nov 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

When branches are created from issues, their pull requests are automatically linked.

1 participant